Resuming Training

[shtoshni]

Hi,

I have a question regarding what are the best practices to resume training supposing that the training was aborted at some point. Right now I have been setting the resume_from_checkpoint argument in Trainer to the last checkpoint's location. This doesn't work when early stopping led to the training stoppage because the trainer starts a new epoch without bothering to check the early stopping criteria when resuming training. I'm wondering if this is by design or there's a way around this. I'm using a very hacky way of separately loading the checkpoint which has the callback_state for EarlyStopping and checking if the wait number is >= patience. EarlyStopping does have a callback on_load_checkpoint but that function lacks the trainer argument to allow it to stop training.

Thanks,
Shubham

[awaelchli]

Hi, I'm not sure I understand. You want that the Trainer resumes from a checkpoint but then immediately stops again if the early stopping was triggere? If so, early stopping is currently epoch based, so it needs to run until the end of the epoch.

For per step early stopping evaluation, please open a feature request here on github. Happy to take a look.

[shtoshni]

So this feature comes into play when you resume a job, but the reason why the job stopped in the first place was due to early stopping. So just a sanity check that if the job stopped due to early stopping, then don't do any further training when you try to launch training again.

[collinmccarthy]

@shtoshni92 Just FYI I've submitted #8278 with respect to this, in case you want to subscribe to it.