Missing logger folder warning appearing always

[daniellepintz]

Hi, I have noticed this Missing logger folder warning appearing pretty much anytime I instantiate a TB logger without setting the version:
https://github.com/PyTorchLightning/pytorch-lightning/blob/d31126c3310af62d76ffb87368ae1e4cc5bbac93/pytorch_lightning/loggers/tensorboard.py#L292-L299

Here is an example: https://colab.research.google.com/drive/13lWUvQbQioQAcAFP5gOTNcAyPsZIBmwj?usp=sharing

Is something wrong with line 296? It doesn't seem to be working correctly, or maybe I am missing something. Advice would be appreciated!

[daniellepintz]

Looks like the root_dir is created here:
https://github.com/PyTorchLightning/pytorch-lightning/blob/d31126c3310af62d76ffb87368ae1e4cc5bbac93/pytorch_lightning/loggers/tensorboard.py#L173

Should we move that to the init?
https://github.com/PyTorchLightning/pytorch-lightning/blob/d31126c3310af62d76ffb87368ae1e4cc5bbac93/pytorch_lightning/loggers/tensorboard.py#L88-L111

[daniellepintz]

Any ideas @awaelchli @edward-io @Borda @ananthsub @rohitgr7 ?

[awaelchli]

This warning was introduced originally in #1377 two years ago.

You should see it in two cases:

1. The very first time you run a lightning training in a folder where there is no lightning_logs folder yet. The trainer will create the folder when needed, but it simply means that the version of a previous experiment can't be determined directly by looking at the lightning_logs folder (and hence has to default to 0). Re-running new experiments look at the existing lightning_logs and the version numbers of the folders version_0, version_1, etc. A warning should no longer be printed.

2. When running DDP across multiple nodes thad don't share a filesystem where trainer can write logs to. In this case, if rank 0 is the one writing logs, i.e. lightning_logs/version_0, lightning_logs/version_1, etc. then the rank 0 can determine the next experiment version as described above but all other ranks on node > 0 can't see that. The logdir eventually gets broadcast but I guess the version number will never be correct.

Can you confirm 1 is the behavior you are seeing?
The warning message could be improved but I don't know how smart we can be about 2).