How to start training again from last checkpoint

[FatimaArshad-DS]

Hi,

My training got interrupted and Im trying to restart training from last checkpoint. However, training starts from the beginning. How do I make it start from last checkpoint?

[titu1994]

If you're using exp manager, it has two flags for resuming training v

Without it, you can try loading your checkpoint using load_from_checkpoinr() and then calling trainer.train() again but we haven't tested that

[FatimaArshad-DS]

Are you talking about this one? create_checkpoint_callback. It is set to True @titu1994

[itzsimpl]

@FatimaArshad-DS see
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L197-L200
and
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L204-L206

[itzsimpl]

Note that you will need to fix eithr
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L188-L189
or set
https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/utils/exp_manager.py#L194-L195
to point to your specific run, that you wish to resume.

[ericharper]

We also some documentation here: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/core/exp_manager.html