How to balance the GPU load

[YanhaoWu]

When I tried to use 8 3090 cards for training, I found that the first card used more memory. Just like this

Due to the limitation of card 1, I can't transfer my batch_size increases. Is there any way to balance the load on multiple cards?

I used the following code to start the training

model_sem_kitti = SemanticKITTIContrastiveTrainer(model, criterion, train_loader, args)
trainer = Trainer(gpus=-1, accelerator='ddp', max_epochs=args.epochs, accumulate_grad_batches=args.accum_steps)
trainer.fit(model_sem_kitti)

I am using pytorch-lightning==1.1.8

[akihironitta]

@WYHbxer We have a similar issue reported in #12651 where they reported that the memory usage on the 0th GPU was significantly higher than the other GPUs. Eventually, it turned out that their script had some manual calls of .to("cuda") or something similar.

Do you have any call of .to(...) in your script?

---

Also, in general, I'd strongly recommend using one of more recent releases of PyTorch Lightning since there're heaps of bugfixes/improvements since the version (1.1.8) you're using.

---

Just realised that you cross-posted this question in an issue #13135. (When you do this, please paste its link here as well to avoid scattering information.)