ddp help

[sneiman]

Training a simple autoencoder as part of a larger project, and using it to get back up to speed on Lightning. Pytorch Lightning v 1.6.2, AMD Ryzen, 2 A6000s, Ubuntu 21.10, using DDP. I have 2 questions:

1. During script startup, I warn the user if they are about to overwrite existing log files. When using DDP and both devices, pl asks this question twice, of course, once on each process. Where can I put this interaction to avoid doing it twice? I suppose I could put it in prepare_data() - but am hoping for something that feels more appropriate.

2. Sometimes - but not always - calling trainer.test() immediately following training fails to find the checkpoint file. This happens with unchanged code and model - sometimes it works and sometimes it doesn't. Interestingly, when it fails, a message is displayed saying it is going to to use a file which does exists and is the best checkpoint, but it does not load it. Instead it fails twice, each time trying to load a different file name which does not exist. Guessing that each process is trying to run its own version of best checkpoint ... Is there something I need to do/call before running test() to make sure this has all been resolved back to one process? Or have I found a bug ...

Any help appreciated.

seth

[sneiman]

I resolved the first question. For those with a similar problem:
There is a set of rank_zero decorators which can be imported from utilities. The one I used is imported like so:
from pytorch_lightning.utilities.rank_zero import rank_zero_only

Put user interaction into a function like so:

@rank_zero_only
def prelim(args):
    # do as you please here - it will only run once from rank 0

Call it before the trainer.fit(), and it will only run once.

See utilities docs for details ...