WandB Logging DDP

[benjaminrwilson]

What is the proper way to prevent repeated WandB experiments when using DDP with multiple GPUs?

The WandB discussion discusses it here: https://docs.wandb.ai/guides/track/advanced/distributed-training (Method 1). Has anyone had any success preventing each GPU spawning an experiment?

Thanks in advance!

[awaelchli]

You can use our wrapper for wandb.

It logs only on the first process, so only one experiment gets created.

from pytorch_lightning.loggers import WandbLogger
logger = WandbLogger(project="the coconut", name="is not a nut")
trainer = Trainer(logger=logger)

[benjaminrwilson]

Thanks for the response, @awaelchli. I am using the built in wrapper for Wandb. For some more context, my jobs are being submitted to a SLURM cluster via the submitit plugin using the --multirun option.

For example, here is a 2-GPU job I submitted:

which causes each GPU to be logged. I'm using groups to at least keep the individual runs together. Maybe I have another setting set incorrectly?

[awaelchli]

It could be that you are using a lightning version where we had bugs surrounding the trainer not setting the rank information correctly when using SLURM. Make sure to test the latest Lightning release.

Are both experiments logging values? I suspect that one of them is empty.

[benjaminrwilson]

I'm using a fork I pulled from the main branch yesterday. Yeah, only the rank zero experiment is logging. There are no functional issues per se, but the two downsides are:

1. Cluttered dashboard.
2. Only the rank zero experiment shows finished after completing the job. All of the other experiments (from other gpus) show crashed.