Multi GPU Training

[thetznecker]

Is there a real time benefit when training with multiple GPUS or is it just about adding up VRAM for bigger batch sizes?

Or would it be better to train 1 model on device 0 and 1 model on device 1?

Also, has someone got the commands to start the multi-gpu training correctly?
We're getting tensor.detach() errors and can't find any related information about it.

[michnovka]

Have you figured this out? I am unable to train on multiple GPUs and cannot find any relevant information

[thetznecker]

Have you figured this out? I am unable to train on multiple GPUs and cannot find any relevant information

Sadly not, we're also not training at the moment so I can't tell you anything new.
We just went on with single GPU training and made it fit as much as possible and trained one model per GPU.