About ModelCheckpoint starting point

[ShaneTian]

My ModelCheckpoint callback:

ckpt_callback = pl.callbacks.ModelCheckpoint(
    filename="T5-{epoch}-{step}-{val_loss:.2f}-{val_ppl:.2f}",
    monitor="val_loss",
    save_top_k=-1,
    every_n_train_steps=100
)

Questions:

1. I got the checkpoints starting with 199 step:
   
   Why does NOT pl save the checkpoint in 99 step?
2. Why is step not an integer multiple of 100? Because counting starts at 0? It is kind of weird.

[rohitgr7]

hey @ShaneTian !

1. maybe no validation loop is triggered after 99 steps, possibly because total training steps per epoch > 100?
2. yes, it is zero-indexed.

[ShaneTian]

Hey @rohitgr7 , thank you!

1. Yes, in my experiment, the total training steps per epoch is 120, does this result in no validation loop in epoch 0? Why no validation loop is triggered at step 100?
2. OK.

[rohitgr7]

yes, the validation loop will be triggered after 1st epoch, but since you have set up monitor='val_loss' and every_n_train_steps=100, after 100 steps, when it tries to find the metric to be monitored, it will not find anything since by that time validation loop isn't triggered, so no-metric means nothing is monitored at that point and no checkpoint is being saved.

[ShaneTian]

Thank you!