Same version when resuming training

[shivammehta25]

Hello everyone,

I had a question: Is there any way I can reuse the same version number from which the model checkpoint was saved? Because now when I resume training from a checkpoint the logger starts logging onto a different version and which I don't want.
For example: my checkpoint is saved as v_num 2 and I resume the training, it starts with v_num3

Thank you for the help

[carmocca]

Have you tried creating your own ModelCheckpoint instance with dirpath set?

Trainer(callbacks=[ModelCheckpoint(dirpath="foo")]

[shivammehta25]

Yes my callback looks like

ModelCheckpoint(dirpath='checkpoints/',
                                          filename=f"{hparams.run_name}-" + '{epoch}-{step}-{ver}',
                                          verbose=True)

Because of this, the tensorboard is also generating logs at the new version and this is little inconsistent behaviour, than one can anticipate.