[Question] Continue training

[AmineAndam04]

❓ Question

I wanted to train a DQN agent for 10M, given the time limit of the cluster I'm using I have to continue training the agent by loading the last saved model.I tried to follow the documentation (see the command bellow) but I seems like the model is not properly loaded: the last model achieves a score of 20 but when I continue training it has a score of -21 and other parameters (e.g exploration rate) don't continue annealing.

I just wanted to make sure that I'm not missing something. Is this the right way to do it?

python train.py --algo dqn --env PongNoFrameskip-v4 -i agents/dqn/PongNoFrameskipv4_1/rl_model_60000_steps.zip -n 1000000

Checklist

• I have checked that there is no similar issue in the repo
• I have read the SB3 documentation
• I have read the RL Zoo documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[araffin]

Hello,

I just wanted to make sure that I'm not missing something. Is this the right way to do it?

You should also save the replay buffer (see doc https://rl-baselines3-zoo.readthedocs.io/en/master/guide/train.html#save-replay-buffer) and update the exploration parameters.

For instance, via the command line, you can do (to start with the final exploration rate)

python train.py --algo dqn --env PongNoFrameskip-v4  -i .../model.zip -param learning_starts:0 exploration_fraction:0.0

For instance, using pre-trained agent from the RL Zoo:

python train.py --algo dqn --env PongNoFrameskip-v4 -i rl-trained-agents/dqn/PongNoFrameskip-v4_1/PongNoFrameskip-v4.zip -param learning_starts:0 exploration_fraction:0.0

Give me a score of ~20 after 4 episodes only:

----------------------------------
| rollout/            |          |
|    ep_len_mean      | 7.37e+03 |
|    ep_rew_mean      | 19.8     |
|    exploration_rate | 0.01     |
| time/               |          |
|    episodes         | 4        |
|    fps              | 277      |
|    time_elapsed     | 26       |
|    total_timesteps  | 7357     |
| train/              |          |
|    learning_rate    | 0.0001   |
|    loss             | 0.00016  |
|    n_updates        | 2476839  |
----------------------------------
----------------------------------
| rollout/            |          |
|    ep_len_mean      | 7.04e+03 |
|    ep_rew_mean      | 20.2     |
|    exploration_rate | 0.01     |
| time/               |          |
|    episodes         | 8        |
|    fps              | 293      |
|    time_elapsed     | 47       |
|    total_timesteps  | 14047    |
| train/              |          |
|    learning_rate    | 0.0001   |
|    loss             | 0.000114 |
|    n_updates        | 2478511  |
----------------------------------