Whether monitor becomes invalid when save_top_k =-1

[ShaneTian]

Whether monitor becomes invalid in following code:

ckpt_callback = pl.callbacks.ModelCheckpoint(
    filename="T5-{epoch}-{step}-{val_loss:.2f}-{val_ppl:.2f}",
    monitor="val_loss",
    save_top_k=-1,
    every_n_train_steps=100
)

[ananthsub]

You can think of monitor as a way to track the best models. Even if you save all of them, they can be ordered according to the monitor + mode. This way, ModelCheckpoint.best_k_models will give you the ordering over all checkpoints saved .

In your example, you're using a model checkpoint with monitor "val_loss" which is expected to be saved "every_n_train_steps". This is not recommended: you're monitoring a metric which is produced during validation, but are telling the model checkpoint to run during the training epoch. This can result in failures because "val_loss" may not be available available when you want to save a checkpoint.

[ShaneTian]

Thanks!

So, what are the best practices when I want to do two things meanwhile:

• save a checkpoint every_n_train_steps,
• save best k checkpoints according to monitor + mode

Maybe I can use two ModelCheckpints? One for every_n_train_steps, one for monitor