Resuming Training

[ugadiarov-la-phystech-edu]

Hello! Thank you for open-sourcing EfficientZeroV2.
I have implemented saving the ReplayBuffer, optimizer, and scheduler every save_ckpt_interval, along with the model's weights. Now, I need to implement resuming training and would like some guidance on the best approach. Specifically, I’m wondering if there’s any other data I should save to ensure training can be resumed seamlessly. What is the proper way to resume training?
Additionally, could you please share some tips on how to debug EfficientZeroV2? I am using PyCharm as my IDE, and I am encountering an error related to closed sockets during debugging. I also tried initializing Ray with local_mode=True, but that led to another error.
Thanks in advance!

[Shengjiewang-Jason]

Hi, thank you for the comment. I am not sure if I understand your question about resume training. Can you explain more?
For the other question, I suggest you to use ipdb coz it can debug the code in the multiprocessing.

[ugadiarov-la-phystech-edu]

Here is some context for my question: I run my experiments on a cluster where jobs have an execution time limit of 48 hours. For many environments, it takes more than two days to train an RL algorithm. As a result, I typically re-run the job every two days, resuming training by initializing models and optimizers with the states saved from the previous job. In the case of off-policy RL and MBRL, it is also necessary to save the ReplayBuffer in order to resume training.

I found that currently only model weights are saved to disk every save_ckpt_interval steps. My idea is to additionally implement saving the states of the optimizer, scheduler, and attributes of "servers" and "workers" (e.g., counters in GlobalStorage, data in the ReplayBuffer, etc.) every save_ckpt_interval steps and use this data to initialize the EZv2 components when resuming the job.

My main question is: I want to resume training after an interruption as if it never happened. Is my approach correct for achieving this goal? (Let's assume I can restore the simulator state of the environment and the state of the random number generator as well.) Do you see any potential pitfalls with this approach?

[Shengjiewang-Jason]

Oh, I see. I think you consider the storage of all parts, so you can try it.