Directory structure / config file location with different loggers and LightningCLI

[cifkao]

I'm observing some inconsistent behavior with different loggers with LightningCLI.

The arguments I'm using are: --trainer.logger=<LoggerClass> --trainer.logger.save_dir=exp/

With TensorBoardLogger, I get a directory like exp/lightning_logs/version_0 containing config.yaml, hparams.yaml, TensorBoard logs and checkpoints.

With CSVLogger, I get a directory like exp/lightning_logs/version_0 containing hparams.yaml, metrics.csv and checkpoints, but not config.yaml

With WandbLogger (which is actually the one I want to use), I get the directories exp/wandb/run-20220411_164045-3ip6qv9j and exp/my-project/3ip6qv9j (where my-project is my W&B project name and 3ip6qv9j is the W&B run ID), with the latter containing the checkpoints but no config files.

In both WandbLogger and CSVLogger, the config.yaml file gets written directly to exp/, which prevents the next run from starting:

RuntimeError: SaveConfigCallback expected exp/config.yaml to NOT exist. Aborting to avoid overwriting results of a previous run. You can delete the previous config file, set `LightningCLI(save_config_callback=None)` to disable config saving, or set `LightningCLI(save_config_overwrite=True)` to overwrite the config file.

So I'm wondering what is actually the expected behavior, whether I'm using the loggers correctly, and what I can do to prevent the config.yaml error from happening.

[Schinkikami]

There are additional problems with the SafeConfigCallback when using multiple loggers.

I have a directory structure like this: save_dir/experiment_name/version_id/{/checkpoints,/logs/, other files}
If I use a single TensorBoardLogger with (save_dir=save_dir, name=experiment_name, subdir="/logs", version=None) the config is saved to save_dir/experiment_name/version_id/logs/config.yaml

However, if multiple loggers are defined (e.g. I additionally define the CSVLogger(save_dir=save_dir, name=experiment_name, version=None)), the config.yaml is safed to save_dir/experiment_name/config.yaml which is in my case the default_root_dir of the trainer WITHOUT the version_id. I think the key problem here is that the SafeConfigCallback uses the trainer.log_dir property to get the safe path. The log_dir is set to trainer.loggers.log_dir if the a single logger is used. However, if multiple loggers are defined it is set to trainer.default_root_dir and in this case multiple experiments are not possible, as it not safed per version.