Large RMSE errors for forces

[ghost]

Dear all,

I plan to train the neural network for the MgCl2 (67 Mg+2 and 134 Cl-) molten salt. However, as seen below, while the RMSE for energies is reasonable, the RMSE values for forces are quite large.

For this case, training data was obtained from MD simulations at different temperatures (700 to 2000 K) and pressure conditions (1 to 100 atm), followed by DFT-derived energies and forces obtained for the snapshots using CP2K. This resulted in a total of 120000 (10% used in validation) snapshots. The "input.json" file used is as below:

{
"_comment": " model parameters",
"model": {
"type_map": ["Mg", "Cl"],
"descriptor" :{
"type": "se_e2_a",
"sel": [100, 100],
"rcut_smth": 7.80,
"rcut": 8.00,
"neuron": [25, 50, 100],
"resnet_dt": false,
"axis_neuron": 16,
"seed": 2,
"_comment": " that's all"
},
"fitting_net" : {
"neuron": [240, 240, 240],
"resnet_dt": true,
"seed": 20,
"_comment": " that's all"
},
"_comment": " that's all"
},
"learning_rate" :{
"type": "exp",
"decay_steps": 5000,
"start_lr": 0.001,
"stop_lr": 3.51e-8,
"_comment": "that's all"
},
"loss" :{
"type": "ener",
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0,
"limit_pref_v": 0,
"_comment": " that's all"
},
"training" : {
"training_data": {
"systems": ["./mgcl2/training"],
"batch_size": "auto",
"_comment": "that's all"
},
"validation_data": {
"systems": ["./mgcl2/validation"],
"batch_size": "auto",
"numb_btch": 1,
"_comment": "that's all"
},
"numb_steps": 1000000,
"seed": 20,
"disp_file": "lcurve.out",
"disp_freq": 100,
"save_freq": 100,
"_comment": "that's all"
},
"_comment": "that's all"
}

What I have tried so far but failed to improve force RMSE values:

1. Reduce and increase "sel"
2. Reduce and increase "rcut_smth" and "rcut"
3. Changed fitting neural network size from {240,240,240} to half and two times this value
4. Changed "start_lr" and "stop_lr"
5. Changed weights on the loss function
6. Removed snapshots at high energy and pressures
7. Reduced the total number of snapshots used for training
8. Used atom-specific weight with the "atom_pref.npy" file
9. Increased the number of training steps
10. Increased the "batch_size"
11. Used different starting "seed" values

Surprisingly, using the same inputs and methods for the NNP for LiCl, the RMSE values are much more reasonable:

After checking the RMSE for forces more carefully, I observed that the main source of the error is from the Mg+2 ions. I further tested to see the densities obtained from 2ns trajectories (using the trained NNPs) and the densities are also very overestimated in comparison to experiments, much worse than the values obtained from AIMD (1.6007 g/cm3 for MgCl2 at 1099 K).

I was hoping if anyone could have some suggestions. Is there perhaps an error with the input files that i used?

Thank you for your time

[njzjz]

You may check whether your data is correct. For example, see if your forces are the gradients instead.

Could you provide a correlation figure, where the x-axis represents DFT forces, and the y-axis represents forces from the model?

[njzjz]

In your data, the energy has at least 10 eV difference (-170167.51992343352 and -170177.34785003273), which is 0.05 eV/A for about 200 atoms. However, in your figure, it is confusing to see only less than 0.01 eV/atom difference.

[ghost]

Sorry for the confusion, there was an error with a script, I reconfirmed the data as seen below, where the spread is about 0.25 eV/A.

[njzjz]

Could you check if your energy and forces are consistent? For example, the unit conversion is correct and you do not forget to add a minus sign to forces when extracting from CP2K gradients.

You can train energy or forces alone to see if they are consistent.

[ghost]

As per your suggestions, I confirmed unit conversation. For CP2K, by default, the values are printed in a.u., that is the units are hatree for energies and hatree/bohr for forces. To convert energies and forces from hatree and hatree/bohr to eV and eV/A, respectively, i multiplied the values by 27.2113961 and 51.42210659.

Additionally, previously I did not multiply the forces by -1. Hence I trained the NNP again after multiplying the forces by -1. Following are the results for the last 10000 steps (i stopped the training at step 900000):

step rmse_val rmse_trn rmse_e_val rmse_e_trn rmse_f_val rmse_f_trn lr
890000 6.59e-01 9.69e-01 1.53e-03 5.10e-02 6.26e-01 6.14e-01 1.1e-07
890500 6.48e-01 7.06e-01 2.54e-03 1.32e-02 6.14e-01 6.47e-01 1.1e-07
891000 8.22e-01 7.07e-01 3.67e-02 2.52e-02 6.04e-01 5.80e-01 1.1e-07
891500 6.86e-01 6.24e-01 1.94e-02 1.23e-03 5.97e-01 5.92e-01 1.1e-07
892000 6.82e-01 6.47e-01 1.43e-02 2.27e-03 6.19e-01 6.13e-01 1.1e-07
892500 6.27e-01 6.87e-01 1.78e-03 1.66e-02 5.95e-01 6.13e-01 1.1e-07
893000 1.33e+00 6.32e-01 8.09e-02 3.34e-03 6.31e-01 5.98e-01 1.1e-07
893500 6.73e-01 4.65e-01 1.50e-02 3.34e-03 6.07e-01 4.40e-01 1.1e-07
894000 6.38e-01 7.79e-01 4.60e-03 3.34e-02 6.03e-01 5.87e-01 1.1e-07
894500 1.34e+00 6.68e-01 8.28e-02 9.29e-03 6.14e-01 6.22e-01 1.1e-07
895000 1.46e+00 7.16e-01 9.09e-02 2.63e-02 6.52e-01 5.82e-01 1.0e-07
895500 6.75e-01 6.16e-01 1.85e-02 9.74e-03 5.93e-01 5.72e-01 1.0e-07
896000 8.18e-01 6.63e-01 3.57e-02 6.04e-03 6.12e-01 6.26e-01 1.0e-07
896500 6.91e-01 6.26e-01 9.10e-03 1.50e-04 6.46e-01 5.97e-01 1.0e-07
897000 1.27e+00 6.33e-01 7.73e-02 4.75e-03 6.03e-01 5.99e-01 1.0e-07
897500 8.25e-01 1.19e+00 3.34e-02 6.97e-02 6.43e-01 6.33e-01 1.0e-07
898000 8.96e-01 6.67e-01 4.22e-02 1.23e-02 6.34e-01 6.13e-01 1.0e-07
898500 6.47e-01 8.36e-01 1.94e-03 3.59e-02 6.15e-01 6.31e-01 1.0e-07
899000 8.56e-01 6.75e-01 4.99e-02 1.31e-02 4.58e-01 6.18e-01 1.0e-07
899500 6.95e-01 8.21e-01 1.35e-02 3.57e-02 6.36e-01 6.15e-01 1.0e-07
900000 6.92e-01 6.58e-01 1.77e-02 3.39e-03 6.16e-01 6.27e-01 9.8e-08

Unfortunately, it gave results similar to that before, so I think the forces are negative of the gradients from CP2K.

Similarly, for training with only energies and forces, I trained the NNP with only energies and forces.

Following are the input file setting and the results where I only trained with energies:

"loss" :{
"type": "ener",
"start_pref_e": 1,
"limit_pref_e": 1,
"start_pref_f": 0,
"limit_pref_f": 0,
"start_pref_v": 0,
"limit_pref_v": 0,
"_comment": " that's all"
},

step rmse_val rmse_trn rmse_e_val rmse_e_trn lr
890000 1.88e-02 1.85e-02 1.33e-03 1.30e-03 1.1e-07
890500 1.11e-02 2.73e-02 7.79e-04 1.93e-03 1.1e-07
891000 5.65e-02 9.48e-03 3.98e-03 6.68e-04 1.1e-07
891500 1.63e-02 9.95e-03 1.15e-03 7.02e-04 1.1e-07
892000 3.64e-05 2.02e-02 2.57e-06 1.42e-03 1.1e-07
892500 1.74e-02 2.52e-02 1.23e-03 1.78e-03 1.1e-07
893000 1.32e-02 3.01e-02 9.34e-04 2.13e-03 1.1e-07
893500 1.54e-02 1.78e-02 1.08e-03 1.25e-03 1.1e-07
894000 2.12e-02 1.49e-01 1.49e-03 1.05e-02 1.1e-07
894500 1.80e-02 1.45e-03 1.27e-03 1.02e-04 1.1e-07
895000 4.32e-03 2.66e-02 3.05e-04 1.88e-03 1.0e-07
895500 2.40e-02 2.67e-02 1.69e-03 1.88e-03 1.0e-07
896000 1.26e-03 9.42e-02 8.90e-05 6.65e-03 1.0e-07
896500 4.44e-03 8.91e-03 3.13e-04 6.28e-04 1.0e-07
897000 4.15e-02 3.39e-02 2.93e-03 2.39e-03 1.0e-07
897500 1.16e-02 2.25e-02 8.21e-04 1.58e-03 1.0e-07
898000 3.39e-03 4.03e-03 2.39e-04 2.84e-04 1.0e-07
898500 3.55e-02 8.17e-03 2.50e-03 5.76e-04 1.0e-07
899000 9.79e-03 5.06e-03 6.90e-04 3.57e-04 1.0e-07
899500 5.99e-02 4.03e-02 4.23e-03 2.84e-03 1.0e-07
900000 7.19e-03 3.07e-02 5.07e-04 2.17e-03 9.8e-08

Surprisingly, forces are better predicted, but energies are not.

Following are the input file setting and the results where I only trained with forces:

"loss" :{
"type": "ener",
"start_pref_e": 0,
"limit_pref_e": 0,
"start_pref_f": 1,
"limit_pref_f": 1,
"start_pref_v": 0,
"limit_pref_v": 0,
"_comment": " that's all"
},

step rmse_val rmse_trn rmse_f_val rmse_f_trn lr
890000 6.27e-01 6.29e-01 6.27e-01 6.29e-01 1.1e-07
890500 5.96e-01 6.08e-01 5.96e-01 6.08e-01 1.1e-07
891000 4.13e-01 4.24e-01 4.13e-01 4.24e-01 1.1e-07
891500 5.92e-01 5.98e-01 5.92e-01 5.98e-01 1.1e-07
892000 6.01e-01 6.50e-01 6.01e-01 6.50e-01 1.1e-07
892500 5.98e-01 5.99e-01 5.98e-01 5.99e-01 1.1e-07
893000 6.29e-01 6.04e-01 6.29e-01 6.04e-01 1.1e-07
893500 5.92e-01 6.35e-01 5.92e-01 6.35e-01 1.1e-07
894000 5.97e-01 4.53e-01 5.97e-01 4.53e-01 1.1e-07
894500 6.11e-01 6.13e-01 6.11e-01 6.13e-01 1.1e-07
895000 6.12e-01 6.23e-01 6.12e-01 6.23e-01 1.0e-07
895500 6.01e-01 5.96e-01 6.01e-01 5.96e-01 1.0e-07
896000 5.88e-01 4.14e-01 5.88e-01 4.14e-01 1.0e-07
896500 6.03e-01 5.84e-01 6.03e-01 5.84e-01 1.0e-07
897000 6.11e-01 6.18e-01 6.11e-01 6.18e-01 1.0e-07
897500 6.12e-01 4.51e-01 6.12e-01 4.51e-01 1.0e-07
898000 6.00e-01 6.43e-01 6.00e-01 6.43e-01 1.0e-07
898500 6.25e-01 6.39e-01 6.25e-01 6.39e-01 1.0e-07
899000 6.14e-01 6.00e-01 6.14e-01 6.00e-01 1.0e-07
899500 6.06e-01 4.51e-01 6.06e-01 4.51e-01 1.0e-07
900000 6.04e-01 5.93e-01 6.04e-01 5.93e-01 9.8e-08

In this case, the results are worse than before. From the above data does this imply that there is an issue with the obtained forces?

Thank you for your time

[njzjz]

Are all of your CP2K calculation coverage?

[h840473807]

Hello，I've tried to train a MgCl2 solution recently. The RMSE_f_trn reached only 4e-1, when I set (CUTOFF=600, EPS_SCF=1e-5). However, when I increased CUTOFF to 800 and EPS_SCF to 1e-6, RMSE_f_trn converged to 6e-2. So whether the error of force is caused by the insufficient calculation accuracy of CP2K ?

[rajnichahal]

Hi all, I am wondering if this issue is resolved. I am also seeing this same thing for my MgCl2-KCl system.

Additionally, I noticed that you mentioned about multiplying cp2k forces (written in .for file) by -1. Can you help me understand why is that needed?

Thanks a lot!