Training is very slow

[Venky0892]

I have a problem set to train 10k images with 20 classes. I'm using Tesla K80 4 GPU machine from azure ml studio. I used all the default parameters with the pre-trained weights. It took me around 13.45hrs to complete 300 epochs with a very minimal mAp value. I know yolov5 is one of the fastest algorithms. I could see my GPU's utilization running over 90% all the time but not sure why the training is slow. I have also installed all the necessary packages. Can someone help why this occurs and what i'm doing wrong ?

[glenn-jocher]

@Venky0892 👋 Hello! Thanks for asking about training speed issues. YOLOv5 🚀 can be trained on CPU (slowest), single-GPU, or multi-GPU (fastest). K80 GPUs are one of the slowest possible GPUs. If you would like to increase your training speed some options are:

• Increase --batch-size
• Reduce --img-size
• Reduce model size, i.e. from YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s
• Train with multi-GPU DDP at larger --batch-size
• Train with a cached dataset: python train.py --cache
• Train on faster GPUs, i.e.: P100 -> V100 -> A100
• Train on free GPU backends with up to 16GB of CUDA memory:

Good luck 🍀 and let us know if you have any other questions!

[Venky0892]

Hey @glenn-jocher thanks for the response. This might help a lot:)