Sync output dir between DDP processes

[rubencart]

I'm looking for a good way to sync my output dir name (which contains a timestamp etc) between DDP processes. For now, I'm doing something like this:

    local_rank = os.environ.get('LOCAL_RANK', 0)

    if local_rank == 0:
        now = datetime.now(dateutil.tz.tzlocal())
        timestamp = now.strftime('%Y_%m_%d_%H_%M_%S')
        run_output_dir = os.path.join(cfg.output_dir,
                                      '%s_%s_%s_%s' % (cfg.dataset, cfg.cfg_name, timestamp, cfg.seed))
        os.environ['RUN_OUTPUT_DIR'] = run_output_dir
    else:
        run_output_dir = os.environ['RUN_OUTPUT_DIR']

Is this OK or does someone have a better solution?

I've tried to use torch.distributed.send and torch.distributed.recv, but these only work for tensors.

I'm also using the WandBLogger, so I have considered having all processes save output to wandb_logger.experiment.dir, but that doesn't work because the logger returns a dummy experiment in all but the main process (link).

[justusschock]

The accelerators also have a broadcast method, which we use to send around stuff like that by byte-encoding it and then use torch.distributed.broadcast

[rubencart]

Okay sounds good, where do I find these methods? Are you talking about DDPPlugin.broadcast? Would be nice if this was documented somewhere

[rubencart]

Any update?

[justusschock]

Yes that's what I was talking about. We are still in the process of documenting all these changes after the big refactor we did.

[littlespice]

Hi, @justusschock
Is there an ETA on DDPPlugin.broadcast?
I'm using torch.distributed.all_gather now, and it would be much more concise and useful when such update is made from PL.