How can I solve the problem that the RMSE does not change and the RMSE does not decrease?

[caolz]

After 1,000,000 steps, the RMSE error between my training and testing is very large, and the value of the test part does not change. Is it because the training set is too small? Or is it overfitting caused by the network structure?

997300 2.88e-01 1.75e-01 4.06e-02 1.96e-03 1.28e-01 1.71e-01 3.9e-08
997400 2.88e-01 2.04e-01 4.06e-02 2.70e-02 1.28e-01 1.10e-01 3.9e-08
997500 2.88e-01 1.92e-01 4.06e-02 1.92e-02 1.28e-01 1.45e-01 3.9e-08
997600 2.88e-01 1.17e-01 4.06e-02 4.00e-03 1.28e-01 1.12e-01 3.9e-08
997700 2.88e-01 1.65e-01 4.06e-02 8.76e-04 1.28e-01 1.62e-01 3.9e-08
997800 2.88e-01 9.57e-02 4.06e-02 8.79e-03 1.28e-01 7.64e-02 3.9e-08
997900 2.88e-01 1.63e-01 4.06e-02 3.79e-03 1.28e-01 1.58e-01 3.9e-08
998000 2.88e-01 2.22e-01 4.05e-02 2.55e-02 1.28e-01 1.50e-01 3.9e-08
998100 2.88e-01 1.47e-01 4.06e-02 1.93e-02 1.28e-01 8.04e-02 3.9e-08
998200 2.88e-01 1.12e-01 4.06e-02 4.21e-03 1.28e-01 1.07e-01 3.9e-08
998300 2.88e-01 2.63e-01 4.06e-02 3.72e-02 1.28e-01 1.14e-01 3.9e-08
998400 2.88e-01 1.82e-01 4.07e-02 1.68e-02 1.28e-01 1.45e-01 3.9e-08
998500 2.89e-01 2.17e-01 4.07e-02 1.80e-02 1.28e-01 1.82e-01 3.9e-08
998600 2.89e-01 2.20e-01 4.07e-02 2.82e-02 1.28e-01 1.26e-01 3.9e-08
998700 2.89e-01 1.42e-01 4.08e-02 1.30e-02 1.28e-01 1.13e-01 3.9e-08
998800 2.89e-01 2.33e-01 4.08e-02 2.20e-02 1.28e-01 1.83e-01 3.9e-08
998900 2.89e-01 1.57e-01 4.08e-02 9.40e-03 1.28e-01 1.43e-01 3.9e-08
999000 2.89e-01 4.92e-01 4.08e-02 6.94e-02 1.28e-01 2.18e-01 3.9e-08
999100 2.90e-01 1.06e-01 4.09e-02 3.11e-03 1.28e-01 1.02e-01 3.9e-08
999200 2.90e-01 1.30e-01 4.09e-02 5.21e-03 1.28e-01 1.24e-01 3.9e-08
999300 2.90e-01 2.61e-01 4.10e-02 3.97e-02 1.28e-01 6.86e-02 3.9e-08
999400 2.90e-01 2.20e-01 4.10e-02 3.29e-02 1.28e-01 6.99e-02 3.9e-08
999500 2.90e-01 1.49e-01 4.10e-02 1.06e-03 1.28e-01 1.46e-01 3.9e-08
999600 2.90e-01 2.51e-01 4.10e-02 3.71e-02 1.28e-01 8.59e-02 3.9e-08
999700 2.91e-01 5.13e-01 4.11e-02 7.58e-02 1.28e-01 1.80e-01 3.9e-08
999800 2.91e-01 2.92e-01 4.11e-02 4.17e-02 1.28e-01 1.23e-01 3.9e-08
999900 2.91e-01 1.64e-01 4.11e-02 9.34e-03 1.28e-01 1.50e-01 3.9e-08
1000000 2.91e-01 2.32e-01 4.12e-02 3.21e-02 1.28e-01 1.10e-01 3.5e-08

Here are the settings.
{
"_comment": " model parameters",
"model": {
"type_map": ["Au"],
"descriptor" :{
"type": "se_a",
"sel": [40],
"rcut_smth": 2.20,
"rcut": 6.50,
"neuron": [5, 15, 20],
"resnet_dt": false,
"axis_neuron": 16,
"seed": 1,
"_comment": " that's all"
},
"fitting_net" : {
"neuron": [10, 10, 10],
"resnet_dt": true,
"seed": 1,
"_comment": " that's all"
},
"_comment": " that's all"
},

"learning_rate" :{
    "type":         "exp",
    "decay_steps":  10000,
    "start_lr":     0.001,  
    "stop_lr":      3.51e-8,
    "_comment":     "that's all"
},

"loss" :{
    "start_pref_e": 0.001,
    "limit_pref_e": 1,
    "start_pref_f": 1000,
    "limit_pref_f": 1,
    "start_pref_v": 0,
    "limit_pref_v": 0,
    "_comment":     " that's all"
},

"_comment": " traing controls",
"training" : {
    "systems":      ["../1-data/nopbc"],
    "set_prefix":   "set",    
    "stop_batch":   1000000,
    "batch_size":   1,

    "seed":         1,

    "_comment": " display and restart",
    "_comment": " frequencies counted in batch",
    "disp_file":    "lcurve.out",
    "disp_freq":    100,
    "numb_test":    10,
    "save_freq":    1000,
    "save_ckpt":    "model.ckpt",
    "load_ckpt":    "model.ckpt",
    "disp_training":true,
    "time_training":true,
    "profiling":    false,
    "profiling_file":"timeline.json",
    "_comment":     "that's all"
},

"_comment":         "that's all"

}

[AnguseZhang]

1. What's your reason for choosing "neuron": [5, 15, 20], and "neuron": [10, 10, 10], respectively for embedding net and fitting net, instead of defaulted settings?
2. Is there any warning in training process?
3. Training accuracy is always related to the accuracy of training data, such as k-spacing, energy cut-off etc. You should always be clear about your first-principles settings before you train them

[caolz]

1. In fact, I also encountered the same problem with the default setting, and the RMSE remained unchanged at 10^-2. I think the network may be too big and shrink a lot, and the result is not much different.
2. There is no warning during training.
3. My current training set has 20,000 structures. I think it may be because I have too many training sets and this potential energy surface is not complicated.

[caolz]

The current settings and water_se_a.json are the same except for a few lines of inconsistency.
4c4
<"type_map": ["O", "H"],

"type_map": ["Au"],
7c7
<"sel": [46, 92],

---

"sel": [38],

The following is the output file of lcurve.out, the RMSE of the force test is almost stable at 2.07e-01.

batch l2_tst l2_trn l2_e_tst l2_e_trn l2_f_tst l2_f_trn lr

  0    7.70e+01  7.05e+01    7.80e+00  7.63e+00    2.43e+00  2.22e+00    1.0e-03
100    7.44e+00  5.67e+00    3.92e+00  3.94e+00    2.08e-01  1.41e-01    1.0e-03
200    6.68e+00  6.93e+00    1.36e+00  1.15e+00    2.08e-01  2.17e-01    1.0e-03
300    6.58e+00  8.26e+00    4.16e-01  3.38e-01    2.08e-01  2.61e-01    1.0e-03
400    6.57e+00  8.45e+00    1.67e-01  6.12e-02    2.08e-01  2.67e-01    1.0e-03
500    6.57e+00  5.79e+00    1.66e-01  1.50e-01    2.08e-01  1.83e-01    1.0e-03
600    6.57e+00  8.97e+00    1.66e-01  6.77e-02    2.08e-01  2.84e-01    1.0e-03
700    6.56e+00  7.40e+00    1.67e-01  5.00e-02    2.08e-01  2.34e-01    1.0e-03
800    6.57e+00  7.96e+00    2.49e-01  2.66e-01    2.08e-01  2.52e-01    1.0e-03
900    6.57e+00  6.39e+00    1.71e-01  2.15e-02    2.08e-01  2.02e-01    1.0e-03

1000 6.57e+00 4.57e+00 1.69e-01 4.10e-02 2.08e-01 1.45e-01 1.0e-03
1100 6.57e+00 4.74e+00 1.66e-01 7.03e-02 2.08e-01 1.50e-01 1.0e-03
1200 6.57e+00 7.08e+00 1.67e-01 8.02e-02 2.08e-01 2.24e-01 1.0e-03
1300 6.57e+00 1.59e+01 1.73e-01 1.97e-01 2.08e-01 5.02e-01 1.0e-03
1400 6.57e+00 3.40e+00 1.66e-01 1.83e-01 2.08e-01 1.07e-01 1.0e-03
1500 6.57e+00 3.62e+00 1.66e-01 7.54e-02 2.08e-01 1.15e-01 1.0e-03
1600 6.57e+00 5.57e+00 1.67e-01 6.14e-02 2.08e-01 1.76e-01 1.0e-03
1700 6.56e+00 8.13e+00 1.72e-01 1.95e-02 2.07e-01 2.57e-01 1.0e-03
1800 6.56e+00 5.82e+00 1.67e-01 5.43e-02 2.07e-01 1.84e-01 1.0e-03
1900 6.56e+00 2.97e+00 1.68e-01 6.70e-02 2.07e-01 9.40e-02 1.0e-03
2000 6.56e+00 6.28e+00 1.66e-01 7.44e-02 2.07e-01 1.99e-01 1.0e-03
2100 6.56e+00 6.50e+00 1.72e-01 5.24e-03 2.07e-01 2.06e-01 1.0e-03
2200 6.57e+00 5.90e+00 2.73e-01 3.00e-01 2.08e-01 1.86e-01 1.0e-03
2300 6.57e+00 6.42e+00 1.76e-01 1.30e-01 2.08e-01 2.03e-01 1.0e-03
2400 6.56e+00 5.79e+00 1.75e-01 2.26e-03 2.08e-01 1.83e-01 1.0e-03
2500 6.56e+00 5.12e+00 1.66e-01 2.00e-02 2.08e-01 1.62e-01 1.0e-03
2600 6.56e+00 6.06e+00 1.81e-01 1.31e-01 2.07e-01 1.92e-01 1.0e-03
2700 6.56e+00 8.21e+00 1.87e-01 7.54e-02 2.08e-01 2.60e-01 1.0e-03
2800 6.56e+00 8.91e+00 1.67e-01 1.99e-02 2.08e-01 2.82e-01 1.0e-03
2900 6.56e+00 6.77e+00 1.66e-01 7.02e-02 2.07e-01 2.14e-01 1.0e-03
3000 6.57e+00 1.42e+01 1.70e-01 8.11e-02 2.08e-01 4.49e-01 1.0e-03
3100 6.57e+00 4.98e+00 1.67e-01 5.90e-02 2.08e-01 1.57e-01 1.0e-03
3200 6.56e+00 4.10e+00 1.68e-01 6.95e-02 2.08e-01 1.30e-01 1.0e-03
3300 6.56e+00 1.01e+01 1.67e-01 8.02e-03 2.07e-01 3.19e-01 1.0e-03
3400 6.56e+00 3.85e+00 1.71e-01 9.73e-04 2.07e-01 1.22e-01 1.0e-03
3500 6.56e+00 1.19e+01 1.70e-01 1.20e-01 2.08e-01 3.75e-01 1.0e-03
3600 6.56e+00 6.54e+00 2.56e-01 2.72e-01 2.07e-01 2.07e-01 1.0e-03
3700 6.56e+00 6.95e+00 1.68e-01 1.20e-01 2.07e-01 2.20e-01 1.0e-03
3800 6.56e+00 8.26e+00 1.66e-01 4.75e-02 2.08e-01 2.61e-01 1.0e-03
3900 6.56e+00 6.76e+00 1.69e-01 1.27e-01 2.07e-01 2.14e-01 1.0e-03
4000 6.56e+00 4.75e+00 1.92e-01 1.81e-01 2.07e-01 1.50e-01 1.0e-03
4100 6.56e+00 4.85e+00 1.66e-01 5.52e-02 2.08e-01 1.53e-01 1.0e-03
4200 6.56e+00 9.13e+00 1.65e-01 8.26e-02 2.07e-01 2.89e-01 1.0e-03
4300 6.56e+00 9.11e+00 1.67e-01 1.08e-01 2.07e-01 2.88e-01 1.0e-03
4400 6.56e+00 7.53e+00 1.65e-01 2.51e-02 2.07e-01 2.38e-01 1.0e-03
4500 6.56e+00 4.85e+00 1.67e-01 5.26e-02 2.07e-01 1.54e-01 1.0e-03
4600 6.56e+00 3.71e+00 1.66e-01 5.52e-03 2.07e-01 1.17e-01 1.0e-03
4700 6.56e+00 5.91e+00 1.78e-01 4.89e-02 2.07e-01 1.87e-01 1.0e-03
4800 6.57e+00 4.63e+00 1.68e-01 7.48e-02 2.08e-01 1.46e-01 1.0e-03
4900 6.56e+00 1.10e+01 1.66e-01 3.79e-02 2.08e-01 3.48e-01 1.0e-03
5000 6.40e+00 4.73e+00 1.80e-01 7.35e-02 2.07e-01 1.53e-01 9.5e-04

[caolz]

I am constructing a potential for Au40, and the test set is reduced from 20,000 to about 7,000, but this problem is still not solved.
In the past, with 70,000 data in other software packages, the RMSE of energy can be achieved 2*10^-3. How can I solve this problem?

[AnguseZhang]

1. First, the "sel" means the maximum of possible neighbor atoms of one certain type for each atom, (containing image atoms due to periodic boundary condition). When training a Au potential, I myself choose 500 for this value. You should make sure that 38 is enough for your system. This key is vital for training accuracy.
2. Second, you should carefully check your data preparation, especially the unit conversion, according to the description in manual. You are recommended to use dpdata https://github.com/deepmodeling/dpdata . Otherwise, it's very likely to make mistakes.
3. The training accuracy is less related to your training data size. Don't need to worry about that too much.

[caolz]

Thank you for your reply.

1. Considering the periodic boundary conditions, 38 is indeed not enough, I will re-estimate it.
2. I will confirm my unit problem.