Is there a way to save a checkpoint with a time interval and also at the end of epochs?

[fishbotics]

Looking at the code, it seems like I need to choose whether to checkpoint every so often or after every epoch. My epochs are very long (40 hours), so I need to checkpoint more often. But, I'd like to be able to resume training if a job dies and this seems to only be possible if I use the fault tolerant training or saving after the end of an epoch. I'd like to do the latter. Any ideas?

[semaphore-egg]

Set val_check_interval=0.1 to do validation check 10 times during each epoch.

[fishbotics]

Thanks @semaphore-egg. That increases the rate of the validation check, but doesn't affect the model checkpoint rate, does it?

[semaphore-egg]

I think check point event is triggered at the end of each validation process so the two frequencies are the same (by default)。