How to restart the training in the same version directory?

[GuillaumeRochette]

Hi,
Where I am working, there's a job scheduling system, with a wall time of 72 hours after which the job goes back into the queue and restarts when resources are available.
Some of my models take longer than one run to train, and therefore in the main training script, I am globbing on the experiment_dir in order to find the latest checkpoint and resume from it using the resume_from_checkpoint argument from the Trainer.
However, every time it resumes training it increment the version number by one and creates a new directory so that I end up with something of the following:

└── lightning_logs
    ├── version_0
    │   ├── checkpoints
    │   │   ├── epoch=8-step=9215.ckpt
    │   │   └── last.ckpt
    │   ├── events.out.tfevents.xyz123
    │   └── hparams.yaml
    ├── version_1
    │   ├── checkpoints
    │   │   ├── epoch=125-step=129023.ckpt
    │   │   └── last.ckpt
    │   ├── events.out.tfevents.xyz1234
    │   └── hparams.yaml
    └── version_2
        ├── checkpoints
        │   ├── epoch=145-step=149503.ckpt
        │   └── last.ckpt
        ├── events.out.tfevents.xyz1235
        └── hparams.yaml

This makes it unpractical to visualise results in the TensorBoard, since there will be an "experiment" each time the training restarts from a checkpoint.
If it is possible, I would like to have the training to restart in version_0, and the logs (images and loss curves) would be appended to the original events.out.tfevents.xyz123.

Is it possible to do such a thing?

Thanks in advance,
Best regards,

Guillaume

[awaelchli]

Currently not possible afaik. Today, the resumed training will continue in a new directory.
We are working on this, see this issue:
#5342