Strange training behavior related to the number of training steps

[mcourtiel]

Hi,

I noticed a really strange behavior during the training of a model.
I trained it on the water monomer, with 160 000 structures of the water monomer extracted from a dynamic at 10K. I use 80% for the training and 20% for the validation and then testing.
I trained two model from this data, but I use exactly the same sets of training and validation and exactly the same input.json, the only parameter I change between this two models is the “numb_steps” in the training part (300 000 for the first one and 600 000 for the second one).

When I train the model on 300 000 steps it looks really good with a RMSE of 3e-05 eV for the energies and 3e-03 eV/A for the forces and when I test the model on the validation and training data I have good results.
But when I train this exactly same model on 600 000 steps something goes wrong, the force does not improve throughout the process (7e-02eV/A) and the RMSE for the energy (1e-03) is worse than the one of the previous model (train only on 300 000 steps). Another really strange behavior is that when I test it, either on a validation or a training set, I obtain the same result for the energies of every structure. And the forces are totally wrong.

Does anyone have any idea why this strange behavior happen ?

300000_input.pdf
600000_input.pdf

[wanghan-iapcm]

Please report the version of deepmd-kit and the hardware you used to train the models. Thank you .

[mcourtiel]

The version I use of deepmd-kit is v2.1.5.

For the hardware of my computer :

Packages:
0: Intel Xeon E31220
Microarchitectures:
4x Sandy Bridge
Cores:
0: 1 processor (0), Intel Sandy Bridge
1: 1 processor (1), Intel Sandy Bridge
2: 1 processor (2), Intel Sandy Bridge
3: 1 processor (3), Intel Sandy Bridge

I don't know if this infos are enought so you can have a look in the following file that contain all the infos of the training :
Training_hardware_info.pdf

Thank you

[wanghan-iapcm]

It seems that your start_lr is too large, please reduce it to e.g. 1e-3.

When using a large lr, the training may become unstable, i.e. the error does not decrease. At the border of stable-unstable lr, uncertain behavior may be observed among tasks, some tasks success by chance and others fail.

[mcourtiel]

Thanks a lot for your advice, it is working way better now.

The training is way better now.