How do you checkpoint retrospectively?

[shehzaidi]

Hi, I have a training script in which the model loss spikes suddenly around 100k steps into training. I would like to create checkpoints to reproduce the ~10-100 steps before this happens to debug it.

What's the easiest way to checkpoint "retrospectively"? I.e. when a certain condition is met, I would like to checkpoint the state in the last e.g. 100 steps of training (or just the checkpoint from 100 steps ago).

Thanks! 🙂

[rohitgr7]

do you mean checkpoint every 100 steps?