Multi gpus training doesn't work

[cahya-wirawan]

Hi @prompteus

I tried multi gpus training with torchrun, but it doesn't work. The training just stuck. Do you have idea what could be the issue here? I tried multi gpus training with other whisper training script, it just work by using torchrun. Thanks again for your help.

[prompteus]

Hi,
does single-gpu training work on your hardware?

[cahya-wirawan]

Single gpu training works very well. I trained whisper tiny with 300k audio samples on my single 3090 for 20 hours, and the result is good.
However when I trained it on 2x3090 gpus, the training stuck at the beginning.
I tried multi gpus training with another repo https://github.com/vasistalodagala/whisper-finetune.git on the same hardware, it works properly

[prompteus]

We never made any special adjustments to enable multi-gpu training, so if there are any prerequisites for using e.g. torchrun, our code does not have them. Without further details, it's hard for me to guess what the underlying problem is.

We use Huggingface seq2seq trainer, and torchdata library (by torch team) for data preprocessing, so it isn't apparent to me why the training script couldn't be compatible with torchrun.

I would recommend inspecting the cause with some profiling/debugging tools.

[cahya-wirawan]

I have replaced the torchdata with huggingface datasets for the dataset handling to make multi gpus training working properly.