[DeepSpeed] Multi-GPU can't converge

[thomas-happify]

seed_everything(43, worker=True)

Trainer(gpus=1, accumulate_grad_batches=16, accelerator='ddp', plugin='deepspeed',  max_epoch=20)
Trainer(gpus=2, accumulate_grad_batches=8, accelerator='ddp', plugin='deepspeed', max_epoch=20)

shouldn't these two args have similar training results?
when I use gpus=2 and accumulate_grad_batches=8, the model can't converge.

[ethanwharris]

@thomas-happify Thanks for your question 😃 Are you able to provide a reproducible example showing this in practice? Without that it may be difficult to say why the performances are different.

[SeanNaren]

Hey @thomas-happify I've updaetd the issue tracker here: #8058 (comment)

seems it's working as intended, but let me know if there is anything missing from the reproduce.