checkpointing with val_check_interval

[ajtritt]

I would like to run intermediate validation checks (not just at the end of training epochs) and checkpoint to save the best model states according to these validation checks.

I am using val_check_interval to run intermediate validation checks, but as far as I can tell, ModelCheckpoint is agnostic to validation checks. The only way for me to do that would be something like this:

n_train_batches = ...
val_check_interval = 0.2
ckpt_freq = n_train_batches * val_check_interval

Trainer.fit(val_check_interval=val_check_interval, 
            callbacks=[ModelCheckpoint(every_n_train_steps=ckpt_freq, save_top_k=1, ...)], ...)

Will this work? I don't know what the order of operations is and I want to make sure I don't configure things such that checkpointing runs before the validation check runs.