Deepmd 1.3.3 cannot learn force for some atoms

[yzjin]

I was trying to train a deepmd model over my trajectory of organic compound computed by CP in Quantum Espresso. The trajectory is computed under room temperature (298.15K) and normal pressure, and the forces over atom is not very large. After a few iterations of training, the deepmd model can successfully learned the energy prediction, but the loss on the force is unable to decrease after the first few batches. Specifically, the l2_f_trn and l2_f_tst is around 3e-1 for most of the time. Considering the force in the trajectory is around the scale of 5e-1 ev/A for most of the time, such a loss means that the model cannot learn the force at all. I have tried many of the suggestions in the discussions, including increasing rcut, batch sizes and converting everything into the box, but none of them work for me. Here's my training json file and the first few lines of my lcurve.out file:
{
"_comment": "Model Parameters",
"model": {
"type_map": ["C", "H", "N"],
"descriptor" :{
"type": "se_a",
"sel": [25, 30, 4],
"rcut_smth": 0.50,
"rcut": 7.00,
"neuron": [10, 20, 40],
"resnet_dt": false,
"axis_neuron": 16,
"seed": 1,
"_comment": "Initial Setting"
},
"fitting_net" : {
"neuron": [80, 80, 80],
"resnet_dt": true,
"seed": 1,
"_comment": "Initial Setting"
},
"_comment": "Initial Setting"
},

"learning_rate" :{
"type":		"exp",
"decay_steps":	5000,
"start_lr":	0.001,	
"stop_lr":	3.51e-8,
"_comment":	"Copied from H2O example"
},

"loss" :{
"start_pref_e":	0.02,
"limit_pref_e":	1,
"start_pref_f":	10000,
"limit_pref_f":	10,
"start_pref_v":	0,
"limit_pref_v":	0,
"_comment":	"Copied from H2O example"
},

"_comment": " Training controls -- extend batch size and experiment length",
"training" : {
"systems":	["./system/single/"],
"set_prefix":	"set",    
"stop_batch":	1000000,
"batch_size":	44,

"seed":		1,

"_comment": " display and restart",
"_comment": " frequencies counted in batch",
"disp_file":	"lcurve.out",
"disp_freq":	100,
"numb_test":	10,
"save_freq":	1000,
"save_ckpt":	"model.ckpt",
"load_ckpt":	"model.ckpt",
"disp_training": true,
"time_training": true,
"profiling":	false,
"profiling_file": "timeline.json",
"_comment":	"Copied from H2O example"
},

}

batch l2_tst l2_trn l2_e_tst l2_e_trn l2_f_tst l2_f_trn lr
0 1.17e+02 1.05e+02 5.76e+00 5.74e+00 1.17e+00 1.05e+00 1.0e-03
100 1.05e+02 9.57e+01 4.91e+00 4.89e+00 1.05e+00 9.56e-01 1.0e-03
200 9.33e+01 8.29e+01 5.06e+00 5.06e+00 9.32e-01 8.28e-01 1.0e-03
300 8.20e+01 6.60e+01 5.16e+00 5.15e+00 8.19e-01 6.58e-01 1.0e-03
400 6.56e+01 5.84e+01 4.92e+00 4.92e+00 6.54e-01 5.82e-01 1.0e-03
500 6.50e+01 5.58e+01 4.68e+00 4.68e+00 6.49e-01 5.56e-01 1.0e-03
600 6.44e+01 5.54e+01 4.38e+00 4.38e+00 6.43e-01 5.52e-01 1.0e-03
700 6.41e+01 5.47e+01 4.14e+00 4.14e+00 6.40e-01 5.46e-01 1.0e-03
800 6.25e+01 5.41e+01 3.97e+00 3.97e+00 6.24e-01 5.40e-01 1.0e-03
900 6.30e+01 5.35e+01 3.74e+00 3.74e+00 6.29e-01 5.33e-01 1.0e-03
1000 6.50e+01 5.40e+01 3.57e+00 3.57e+00 6.49e-01 5.39e-01 1.0e-03
1100 6.24e+01 5.18e+01 3.34e+00 3.34e+00 6.23e-01 5.17e-01 1.0e-03
1200 6.18e+01 5.08e+01 3.12e+00 3.12e+00 6.17e-01 5.07e-01 1.0e-03
1300 6.11e+01 6.79e+01 2.90e+00 2.90e+00 6.11e-01 6.78e-01 1.0e-03
1400 6.08e+01 6.43e+01 2.68e+00 2.68e+00 6.08e-01 6.42e-01 1.0e-03
1500 6.11e+01 6.40e+01 2.43e+00 2.44e+00 6.11e-01 6.40e-01 1.0e-03
1600 5.97e+01 6.07e+01 2.22e+00 2.22e+00 5.97e-01 6.07e-01 1.0e-03
1700 6.02e+01 6.65e+01 1.96e+00 1.96e+00 6.01e-01 6.65e-01 1.0e-03
1800 5.90e+01 6.45e+01 1.80e+00 1.80e+00 5.90e-01 6.45e-01 1.0e-03
1900 5.92e+01 6.86e+01 1.58e+00 1.59e+00 5.92e-01 6.85e-01 1.0e-03
2000 5.94e+01 6.29e+01 1.45e+00 1.45e+00 5.94e-01 6.28e-01 1.0e-03
2100 5.72e+01 6.00e+01 1.31e+00 1.31e+00 5.72e-01 6.00e-01 1.0e-03
2200 5.76e+01 5.88e+01 1.12e+00 1.12e+00 5.76e-01 5.88e-01 1.0e-03
2300 5.74e+01 5.96e+01 9.34e-01 9.37e-01 5.74e-01 5.96e-01 1.0e-03
2400 5.54e+01 4.88e+01 8.36e-01 8.52e-01 5.54e-01 4.87e-01 1.0e-03
2500 5.43e+01 4.78e+01 7.00e-01 7.14e-01 5.43e-01 4.78e-01 1.0e-03
2600 5.42e+01 4.59e+01 5.96e-01 6.11e-01 5.42e-01 4.59e-01 1.0e-03
2700 5.53e+01 4.73e+01 5.24e-01 5.40e-01 5.53e-01 4.73e-01 1.0e-03
2800 5.44e+01 4.59e+01 4.22e-01 4.35e-01 5.44e-01 4.59e-01 1.0e-03
2900 5.38e+01 4.44e+01 3.22e-01 3.34e-01 5.38e-01 4.44e-01 1.0e-03
3000 5.18e+01 4.48e+01 2.35e-01 2.47e-01 5.18e-01 4.48e-01 1.0e-03
3100 5.16e+01 4.45e+01 1.90e-01 2.02e-01 5.16e-01 4.45e-01 1.0e-03
3200 5.03e+01 4.37e+01 1.16e-01 1.26e-01 5.03e-01 4.37e-01 1.0e-03
3300 5.00e+01 4.23e+01 8.10e-02 9.28e-02 5.00e-01 4.23e-01 1.0e-03
3400 4.98e+01 4.23e+01 4.67e-02 5.91e-02 4.98e-01 4.23e-01 1.0e-03
3500 4.95e+01 4.08e+01 3.71e-02 3.48e-02 4.95e-01 4.08e-01 1.0e-03
3600 4.94e+01 4.04e+01 3.56e-03 7.94e-03 4.94e-01 4.04e-01 1.0e-03
3700 4.95e+01 4.00e+01 1.49e-02 1.83e-02 4.95e-01 4.00e-01 1.0e-03
3800 4.87e+01 4.02e+01 2.62e-02 3.04e-02 4.87e-01 4.02e-01 1.0e-03
3900 4.90e+01 4.08e+01 7.89e-02 8.43e-02 4.90e-01 4.08e-01 1.0e-03
4000 4.87e+01 3.88e+01 3.01e-02 3.54e-02 4.87e-01 3.88e-01 1.0e-03
4100 4.80e+01 3.73e+01 5.40e-02 5.97e-02 4.80e-01 3.73e-01 1.0e-03
4200 4.80e+01 3.83e+01 2.62e-02 3.03e-02 4.80e-01 3.83e-01 1.0e-03
4300 4.79e+01 3.85e+01 5.60e-02 6.05e-02 4.79e-01 3.85e-01 1.0e-03
4400 4.71e+01 3.76e+01 4.76e-02 5.18e-02 4.71e-01 3.76e-01 1.0e-03
4500 4.76e+01 3.59e+01 3.61e-02 4.14e-02 4.76e-01 3.59e-01 1.0e-03
4600 4.55e+01 5.58e+01 5.68e-02 6.67e-02 4.55e-01 5.58e-01 1.0e-03
4700 4.70e+01 5.87e+01 5.30e-02 6.17e-02 4.70e-01 5.87e-01 1.0e-03
4800 4.42e+01 5.81e+01 9.65e-02 1.05e-01 4.42e-01 5.81e-01 1.0e-03
4900 4.36e+01 5.17e+01 7.59e-02 8.41e-02 4.36e-01 5.17e-01 1.0e-03
5000 4.39e+01 5.54e+01 7.68e-02 8.45e-02 4.51e-01 5.68e-01 9.5e-04
5100 4.32e+01 4.93e+01 4.95e-02 5.69e-02 4.43e-01 5.06e-01 9.5e-04
5200 4.24e+01 5.39e+01 7.04e-02 7.76e-02 4.35e-01 5.53e-01 9.5e-04
5300 4.34e+01 5.35e+01 3.49e-02 4.18e-02 4.45e-01 5.49e-01 9.5e-04
5400 4.36e+01 5.77e+01 2.19e-02 2.83e-02 4.47e-01 5.92e-01 9.5e-04
5500 4.20e+01 5.50e+01 1.42e-02 2.00e-02 4.31e-01 5.64e-01 9.5e-04
5600 4.25e+01 4.99e+01 2.59e-02 3.23e-02 4.36e-01 5.12e-01 9.5e-04
5700 4.31e+01 3.79e+01 1.33e-02 4.92e-03 4.42e-01 3.88e-01 9.5e-04
5800 4.46e+01 3.79e+01 3.08e-02 4.64e-02 4.58e-01 3.89e-01 9.5e-04
5900 5.01e+01 3.85e+01 3.63e-02 5.21e-02 5.14e-01 3.94e-01 9.5e-04
6000 4.67e+01 3.70e+01 1.52e-02 3.26e-03 4.79e-01 3.80e-01 9.5e-04
6100 4.79e+01 3.70e+01 1.80e-02 3.38e-02 4.92e-01 3.80e-01 9.5e-04
6200 4.49e+01 3.56e+01 2.05e-02 3.52e-02 4.61e-01 3.65e-01 9.5e-04
6300 4.73e+01 3.61e+01 3.78e-03 1.40e-02 4.86e-01 3.70e-01 9.5e-04
6400 4.39e+01 3.55e+01 1.12e-02 4.97e-03 4.50e-01 3.64e-01 9.5e-04
6500 4.44e+01 3.53e+01 4.12e-02 2.74e-02 4.55e-01 3.62e-01 9.5e-04
6600 4.53e+01 3.34e+01 3.37e-02 2.51e-02 4.65e-01 3.43e-01 9.5e-04
6700 4.52e+01 3.54e+01 5.33e-02 4.42e-02 4.64e-01 3.63e-01 9.5e-04
6800 4.16e+01 3.42e+01 6.08e-02 6.38e-02 4.27e-01 3.51e-01 9.5e-04
6900 4.18e+01 3.41e+01 5.42e-02 5.85e-02 4.29e-01 3.49e-01 9.5e-04
7000 4.09e+01 3.14e+01 4.37e-02 4.57e-02 4.20e-01 3.23e-01 9.5e-04
7100 4.06e+01 3.35e+01 2.83e-02 2.82e-02 4.16e-01 3.44e-01 9.5e-04
7200 4.33e+01 3.48e+01 5.24e-03 3.88e-03 4.45e-01 3.57e-01 9.5e-04
7300 3.96e+01 3.03e+01 1.57e-02 1.58e-02 4.06e-01 3.11e-01 9.5e-04
7400 4.01e+01 3.11e+01 2.54e-03 2.73e-03 4.12e-01 3.19e-01 9.5e-04
7500 3.96e+01 3.03e+01 6.29e-03 5.04e-03 4.06e-01 3.11e-01 9.5e-04
7600 4.28e+01 3.14e+01 1.65e-02 1.92e-02 4.39e-01 3.22e-01 9.5e-04
7700 3.94e+01 3.07e+01 1.06e-02 1.38e-02 4.04e-01 3.14e-01 9.5e-04
7800 4.13e+01 3.03e+01 2.64e-02 3.03e-02 4.24e-01 3.11e-01 9.5e-04
7900 3.96e+01 4.81e+01 1.28e-02 1.28e-02 4.06e-01 4.93e-01 9.5e-04
8000 3.81e+01 5.72e+01 3.20e-02 3.28e-02 3.91e-01 5.87e-01 9.5e-04
8100 3.84e+01 5.55e+01 3.46e-02 3.35e-02 3.94e-01 5.69e-01 9.5e-04
8200 3.85e+01 5.53e+01 5.77e-03 5.73e-03 3.95e-01 5.68e-01 9.5e-04
8300 3.95e+01 4.67e+01 1.50e-02 1.47e-02 4.05e-01 4.79e-01 9.5e-04
8400 3.76e+01 5.23e+01 1.14e-02 1.14e-02 3.86e-01 5.36e-01 9.5e-04
8500 3.75e+01 4.87e+01 4.74e-03 4.83e-03 3.85e-01 4.99e-01 9.5e-04
8600 3.76e+01 5.37e+01 1.41e-02 1.35e-02 3.86e-01 5.51e-01 9.5e-04
8700 3.67e+01 4.73e+01 3.24e-03 4.87e-03 3.77e-01 4.85e-01 9.5e-04
8800 3.61e+01 4.45e+01 7.21e-03 8.78e-03 3.70e-01 4.56e-01 9.5e-04
8900 3.64e+01 4.68e+01 3.91e-03 4.24e-03 3.74e-01 4.80e-01 9.5e-04
9000 4.01e+01 3.39e+01 2.77e-02 3.27e-02 4.12e-01 3.48e-01 9.5e-04
9100 4.01e+01 3.15e+01 9.02e-03 1.37e-02 4.11e-01 3.23e-01 9.5e-04
9200 3.97e+01 3.13e+01 3.43e-03 1.04e-02 4.07e-01 3.21e-01 9.5e-04
9300 4.17e+01 3.05e+01 1.02e-02 2.34e-03 4.28e-01 3.13e-01 9.5e-04
9400 4.05e+01 3.01e+01 1.91e-02 1.18e-02 4.16e-01 3.09e-01 9.5e-04
9500 3.86e+01 2.94e+01 4.77e-03 3.85e-03 3.96e-01 3.01e-01 9.5e-04
9600 4.04e+01 2.96e+01 1.86e-02 1.19e-02 4.14e-01 3.04e-01 9.5e-04
9700 3.88e+01 2.99e+01 1.65e-02 8.03e-03 3.98e-01 3.06e-01 9.5e-04
9800 3.65e+01 3.10e+01 4.68e-02 3.73e-02 3.75e-01 3.18e-01 9.5e-04
9900 3.96e+01 3.03e+01 2.23e-02 1.34e-02 4.06e-01 3.11e-01 9.5e-04
10000 3.80e+01 2.77e+01 2.53e-02 1.81e-02 4.00e-01 2.91e-01 9.0e-04
10100 3.70e+01 3.15e+01 4.78e-02 4.71e-02 3.90e-01 3.31e-01 9.0e-04
10200 3.57e+01 2.90e+01 4.62e-02 4.58e-02 3.76e-01 3.05e-01 9.0e-04
10300 3.78e+01 3.02e+01 5.23e-03 3.43e-03 3.98e-01 3.18e-01 9.0e-04
10400 3.48e+01 3.01e+01 3.58e-02 3.48e-02 3.66e-01 3.17e-01 9.0e-04
10500 3.52e+01 2.72e+01 3.91e-02 3.90e-02 3.71e-01 2.86e-01 9.0e-04
10600 3.72e+01 2.86e+01 5.24e-03 5.03e-03 3.92e-01 3.01e-01 9.0e-04
10700 3.50e+01 2.73e+01 5.60e-03 6.33e-03 3.68e-01 2.88e-01 9.0e-04
10800 3.55e+01 2.75e+01 3.38e-03 2.42e-03 3.74e-01 2.89e-01 9.0e-04
10900 3.47e+01 2.75e+01 3.85e-02 3.72e-02 3.65e-01 2.90e-01 9.0e-04
11000 3.39e+01 2.82e+01 1.10e-02 9.29e-03 3.57e-01 2.97e-01 9.0e-04
11100 3.45e+01 2.59e+01 7.19e-03 8.75e-03 3.63e-01 2.72e-01 9.0e-04
11200 3.40e+01 4.25e+01 1.54e-03 3.04e-03 3.58e-01 4.47e-01 9.0e-04
11300 3.32e+01 4.33e+01 8.05e-03 7.86e-03 3.49e-01 4.56e-01 9.0e-04
11400 3.29e+01 4.21e+01 2.70e-02 2.64e-02 3.46e-01 4.43e-01 9.0e-04
11500 3.43e+01 4.10e+01 1.45e-02 1.36e-02 3.61e-01 4.32e-01 9.0e-04
11600 3.29e+01 4.84e+01 3.43e-02 3.34e-02 3.47e-01 5.09e-01 9.0e-04
11700 3.34e+01 4.52e+01 3.54e-02 3.42e-02 3.51e-01 4.76e-01 9.0e-04
11800 3.47e+01 3.88e+01 3.14e-02 3.02e-02 3.65e-01 4.08e-01 9.0e-04
11900 3.31e+01 4.35e+01 1.66e-02 1.50e-02 3.48e-01 4.57e-01 9.0e-04
12000 3.29e+01 4.91e+01 3.26e-02 3.20e-02 3.46e-01 5.17e-01 9.0e-04
12100 3.24e+01 3.86e+01 1.42e-03 2.80e-03 3.41e-01 4.06e-01 9.0e-04
12200 3.29e+01 3.93e+01 9.00e-03 1.15e-02 3.47e-01 4.13e-01 9.0e-04
12300 3.34e+01 2.75e+01 2.39e-03 2.91e-03 3.51e-01 2.89e-01 9.0e-04
12400 3.51e+01 2.72e+01 2.72e-03 2.22e-03 3.69e-01 2.87e-01 9.0e-04
12500 3.75e+01 2.82e+01 1.46e-02 1.82e-02 3.94e-01 2.97e-01 9.0e-04
12600 3.44e+01 2.55e+01 2.46e-03 3.04e-03 3.62e-01 2.68e-01 9.0e-04
12700 3.40e+01 2.65e+01 2.31e-02 1.86e-02 3.58e-01 2.79e-01 9.0e-04
12800 3.34e+01 2.57e+01 2.76e-02 2.35e-02 3.51e-01 2.71e-01 9.0e-04
12900 3.48e+01 2.71e+01 2.94e-02 2.42e-02 3.67e-01 2.85e-01 9.0e-04
13000 3.26e+01 2.57e+01 4.05e-02 3.34e-02 3.44e-01 2.71e-01 9.0e-04
13100 3.33e+01 2.58e+01 1.50e-02 8.38e-03 3.51e-01 2.71e-01 9.0e-04
13200 3.44e+01 2.56e+01 8.77e-03 2.67e-03 3.62e-01 2.70e-01 9.0e-04
13300 3.39e+01 2.57e+01 1.39e-02 8.68e-03 3.57e-01 2.71e-01 9.0e-04
13400 3.25e+01 2.81e+01 3.85e-02 3.73e-02 3.42e-01 2.95e-01 9.0e-04
13500 3.39e+01 2.72e+01 2.18e-02 2.20e-02 3.57e-01 2.86e-01 9.0e-04
13600 3.19e+01 2.72e+01 1.95e-02 1.93e-02 3.36e-01 2.87e-01 9.0e-04
13700 3.24e+01 2.73e+01 2.00e-02 2.00e-02 3.41e-01 2.87e-01 9.0e-04
13800 3.26e+01 2.57e+01 2.10e-02 2.12e-02 3.43e-01 2.70e-01 9.0e-04
13900 3.21e+01 2.50e+01 1.86e-02 1.97e-02 3.38e-01 2.63e-01 9.0e-04
14000 3.30e+01 2.77e+01 1.59e-02 1.53e-02 3.47e-01 2.91e-01 9.0e-04
14100 3.20e+01 2.51e+01 1.57e-02 1.67e-02 3.37e-01 2.64e-01 9.0e-04
14200 3.19e+01 2.48e+01 2.63e-02 2.71e-02 3.36e-01 2.61e-01 9.0e-04
14300 3.15e+01 2.42e+01 9.00e-03 9.80e-03 3.31e-01 2.55e-01 9.0e-04
14400 3.18e+01 2.53e+01 2.80e-02 2.97e-02 3.34e-01 2.66e-01 9.0e-04
14500 3.19e+01 5.00e+01 2.60e-02 2.48e-02 3.36e-01 5.26e-01 9.0e-04
14600 3.31e+01 4.66e+01 4.54e-02 4.28e-02 3.49e-01 4.91e-01 9.0e-04
14700 3.21e+01 4.03e+01 4.19e-02 4.00e-02 3.38e-01 4.24e-01 9.0e-04
14800 3.20e+01 4.06e+01 3.18e-02 3.05e-02 3.37e-01 4.27e-01 9.0e-04
14900 3.18e+01 4.48e+01 2.74e-02 2.47e-02 3.35e-01 4.72e-01 9.0e-04
15000 3.01e+01 4.54e+01 2.13e-02 1.84e-02 3.25e-01 4.91e-01 8.6e-04
15100 3.03e+01 3.69e+01 2.23e-02 2.12e-02 3.28e-01 3.99e-01 8.6e-04
15200 3.16e+01 5.38e+01 3.16e-02 3.04e-02 3.42e-01 5.81e-01 8.6e-04
15300 3.02e+01 3.84e+01 1.24e-02 9.23e-03 3.26e-01 4.14e-01 8.6e-04
15400 3.08e+01 5.07e+01 2.22e-02 1.93e-02 3.33e-01 5.47e-01 8.6e-04
15500 3.00e+01 4.54e+01 3.25e-03 6.07e-03 3.24e-01 4.90e-01 8.6e-04
15600 3.12e+01 2.52e+01 1.52e-02 1.57e-02 3.36e-01 2.72e-01 8.6e-04
15700 3.06e+01 2.51e+01 1.80e-03 2.85e-03 3.31e-01 2.71e-01 8.6e-04
15800 3.16e+01 2.46e+01 4.14e-03 2.19e-03 3.42e-01 2.65e-01 8.6e-04
15900 3.14e+01 2.43e+01 5.85e-03 1.92e-03 3.39e-01 2.62e-01 8.6e-04
16000 3.04e+01 2.47e+01 4.30e-02 3.98e-02 3.29e-01 2.66e-01 8.6e-04
16100 3.10e+01 2.41e+01 4.14e-02 3.67e-02 3.34e-01 2.60e-01 8.6e-04
16200 3.06e+01 2.48e+01 3.13e-02 2.61e-02 3.30e-01 2.67e-01 8.6e-04
16300 3.15e+01 2.43e+01 1.97e-02 1.29e-02 3.40e-01 2.63e-01 8.6e-04
16400 3.02e+01 2.47e+01 3.01e-02 2.56e-02 3.26e-01 2.67e-01 8.6e-04
16500 3.02e+01 2.32e+01 1.43e-02 9.41e-03 3.26e-01 2.50e-01 8.6e-04
16600 3.18e+01 2.32e+01 1.72e-02 1.28e-02 3.43e-01 2.51e-01 8.6e-04
16700 3.03e+01 2.50e+01 2.98e-02 3.03e-02 3.28e-01 2.70e-01 8.6e-04
16800 3.07e+01 2.53e+01 3.33e-02 3.48e-02 3.32e-01 2.73e-01 8.6e-04
16900 3.01e+01 2.42e+01 2.16e-02 2.23e-02 3.25e-01 2.61e-01 8.6e-04
17000 2.96e+01 2.36e+01 1.92e-02 2.01e-02 3.20e-01 2.55e-01 8.6e-04
17100 3.00e+01 2.36e+01 4.65e-03 4.31e-03 3.24e-01 2.54e-01 8.6e-04
17200 3.03e+01 2.34e+01 3.15e-03 2.71e-03 3.28e-01 2.53e-01 8.6e-04
17300 2.99e+01 2.32e+01 2.82e-03 3.99e-03 3.22e-01 2.50e-01 8.6e-04
17400 2.96e+01 2.30e+01 9.52e-03 9.24e-03 3.20e-01 2.49e-01 8.6e-04
17500 2.99e+01 2.42e+01 3.14e-02 3.05e-02 3.23e-01 2.62e-01 8.6e-04
17600 2.97e+01 2.35e+01 7.76e-03 8.13e-03 3.20e-01 2.54e-01 8.6e-04
17700 3.10e+01 2.40e+01 1.27e-02 1.38e-02 3.34e-01 2.60e-01 8.6e-04
17800 2.95e+01 4.97e+01 2.35e-02 2.12e-02 3.19e-01 5.37e-01 8.6e-04
17900 3.07e+01 3.73e+01 1.76e-02 1.39e-02 3.32e-01 4.03e-01 8.6e-04
18000 3.04e+01 3.72e+01 4.04e-02 3.69e-02 3.28e-01 4.02e-01 8.6e-04
18100 3.03e+01 4.06e+01 4.65e-03 2.34e-03 3.27e-01 4.38e-01 8.6e-04
18200 2.97e+01 4.44e+01 4.37e-03 3.13e-03 3.21e-01 4.79e-01 8.6e-04
18300 2.99e+01 4.04e+01 1.17e-02 8.69e-03 3.23e-01 4.37e-01 8.6e-04
18400 2.94e+01 4.49e+01 1.95e-02 1.78e-02 3.17e-01 4.85e-01 8.6e-04
18500 3.05e+01 3.66e+01 3.53e-02 3.13e-02 3.30e-01 3.95e-01 8.6e-04
18600 2.97e+01 3.75e+01 1.96e-02 1.66e-02 3.21e-01 4.05e-01 8.6e-04
18700 2.98e+01 4.73e+01 1.50e-02 1.18e-02 3.22e-01 5.11e-01 8.6e-04
18800 3.05e+01 3.78e+01 3.10e-02 2.60e-02 3.29e-01 4.08e-01 8.6e-04
18900 3.28e+01 2.61e+01 8.69e-03 8.56e-03 3.55e-01 2.82e-01 8.6e-04
19000 2.91e+01 2.27e+01 2.59e-03 3.95e-03 3.14e-01 2.46e-01 8.6e-04
19100 3.18e+01 2.38e+01 8.31e-03 1.03e-02 3.43e-01 2.57e-01 8.6e-04
19200 2.98e+01 2.27e+01 1.71e-02 1.69e-02 3.22e-01 2.46e-01 8.6e-04
19300 2.95e+01 2.31e+01 3.12e-02 2.90e-02 3.18e-01 2.49e-01 8.6e-04
19400 3.04e+01 2.37e+01 2.21e-03 2.82e-03 3.29e-01 2.56e-01 8.6e-04
19500 2.97e+01 2.31e+01 2.38e-02 1.92e-02 3.21e-01 2.49e-01 8.6e-04
19600 2.95e+01 2.30e+01 3.01e-02 2.52e-02 3.19e-01 2.49e-01 8.6e-04
19700 2.93e+01 2.31e+01 1.52e-02 1.13e-02 3.17e-01 2.49e-01 8.6e-04
19800 2.98e+01 2.29e+01 3.22e-02 3.06e-02 3.22e-01 2.47e-01 8.6e-04
19900 2.87e+01 2.30e+01 2.38e-02 2.21e-02 3.10e-01 2.49e-01 8.6e-04
20000 2.84e+01 2.31e+01 3.90e-02 3.95e-02 3.14e-01 2.56e-01 8.1e-04
20100 2.83e+01 2.26e+01 2.93e-02 2.88e-02 3.13e-01 2.50e-01 8.1e-04
20200 2.89e+01 2.33e+01 2.11e-02 2.22e-02 3.20e-01 2.58e-01 8.1e-04
20300 2.85e+01 2.23e+01 1.73e-02 1.74e-02 3.16e-01 2.47e-01 8.1e-04
20400 2.89e+01 2.24e+01 2.00e-02 1.95e-02 3.20e-01 2.48e-01 8.1e-04
20500 2.89e+01 2.30e+01 8.29e-03 7.34e-03 3.20e-01 2.55e-01 8.1e-04
20600 2.87e+01 2.21e+01 2.57e-03 2.63e-03 3.18e-01 2.45e-01 8.1e-04
20700 2.91e+01 2.23e+01 2.87e-03 2.76e-03 3.22e-01 2.47e-01 8.1e-04
20800 2.84e+01 2.18e+01 6.45e-03 6.43e-03 3.15e-01 2.41e-01 8.1e-04
20900 2.93e+01 2.27e+01 7.26e-03 7.74e-03 3.24e-01 2.51e-01 8.1e-04
21000 2.84e+01 2.16e+01 1.17e-02 1.22e-02 3.15e-01 2.39e-01 8.1e-04
21100 2.87e+01 3.47e+01 1.11e-02 7.36e-03 3.18e-01 3.85e-01 8.1e-04
21200 3.00e+01 4.15e+01 1.56e-02 1.24e-02 3.32e-01 4.60e-01 8.1e-04
21300 2.95e+01 3.81e+01 3.77e-02 3.35e-02 3.27e-01 4.23e-01 8.1e-04
21400 2.84e+01 4.26e+01 1.46e-02 1.07e-02 3.14e-01 4.72e-01 8.1e-04
21500 2.86e+01 4.73e+01 1.53e-02 1.20e-02 3.17e-01 5.24e-01 8.1e-04
21600 3.19e+01 4.56e+01 1.35e-03 3.00e-03 3.53e-01 5.05e-01 8.1e-04
21700 2.96e+01 4.44e+01 1.69e-02 1.46e-02 3.28e-01 4.92e-01 8.1e-04
21800 2.90e+01 4.65e+01 1.34e-02 1.03e-02 3.21e-01 5.15e-01 8.1e-04
21900 2.83e+01 5.62e+01 1.21e-02 8.87e-03 3.13e-01 6.22e-01 8.1e-04
22000 2.87e+01 5.08e+01 2.37e-02 1.86e-02 3.18e-01 5.63e-01 8.1e-04
22100 2.92e+01 3.74e+01 1.19e-02 8.24e-03 3.23e-01 4.15e-01 8.1e-04
22200 3.03e+01 2.40e+01 1.81e-02 1.89e-02 3.36e-01 2.66e-01 8.1e-04
22300 2.85e+01 2.20e+01 1.68e-03 1.27e-03 3.16e-01 2.44e-01 8.1e-04
22400 3.03e+01 2.28e+01 2.62e-02 2.73e-02 3.36e-01 2.53e-01 8.1e-04
22500 2.98e+01 2.40e+01 1.02e-02 1.13e-02 3.30e-01 2.66e-01 8.1e-04
22600 2.86e+01 2.21e+01 3.23e-03 3.41e-03 3.17e-01 2.44e-01 8.1e-04
22700 2.86e+01 2.30e+01 1.73e-02 1.41e-02 3.17e-01 2.55e-01 8.1e-04
22800 2.97e+01 2.23e+01 1.09e-02 1.22e-02 3.29e-01 2.48e-01 8.1e-04
22900 2.94e+01 2.23e+01 1.59e-02 1.25e-02 3.26e-01 2.47e-01 8.1e-04
23000 2.84e+01 2.20e+01 3.29e-02 3.08e-02 3.15e-01 2.44e-01 8.1e-04
23100 2.94e+01 2.24e+01 2.00e-02 1.83e-02 3.26e-01 2.48e-01 8.1e-04
23200 2.84e+01 2.19e+01 2.57e-02 2.27e-02 3.14e-01 2.43e-01 8.1e-04
23300 2.87e+01 2.29e+01 3.42e-02 3.44e-02 3.18e-01 2.54e-01 8.1e-04
23400 2.89e+01 2.31e+01 1.95e-02 2.12e-02 3.20e-01 2.56e-01 8.1e-04
23500 2.82e+01 2.26e+01 9.49e-03 1.04e-02 3.12e-01 2.50e-01 8.1e-04
23600 2.85e+01 2.30e+01 1.58e-02 1.62e-02 3.16e-01 2.55e-01 8.1e-04
23700 2.85e+01 2.30e+01 1.56e-02 1.58e-02 3.15e-01 2.54e-01 8.1e-04
23800 2.82e+01 2.13e+01 6.15e-03 5.06e-03 3.13e-01 2.36e-01 8.1e-04
23900 2.79e+01 2.20e+01 2.15e-03 1.90e-03 3.09e-01 2.44e-01 8.1e-04
24000 2.83e+01 2.21e+01 4.19e-03 3.61e-03 3.14e-01 2.45e-01 8.1e-04
24100 2.82e+01 2.23e+01 2.01e-02 1.96e-02 3.12e-01 2.47e-01 8.1e-04
24200 2.84e+01 2.18e+01 6.36e-03 6.17e-03 3.15e-01 2.42e-01 8.1e-04
24300 2.91e+01 2.29e+01 5.33e-03 5.98e-03 3.23e-01 2.54e-01 8.1e-04
24400 2.85e+01 3.38e+01 4.06e-03 1.95e-03 3.16e-01 3.75e-01 8.1e-04

[njzjz]

#709 may be related - please first confirm it.

[yzjin]

Thank you for your prompt reply. The CP.x software returns forces directly, and I have double checked the units conversion I'm using, so I guess the issue refered by #709 is not the case here. A short scale experiment conducted using the frozen model shows that the system can be stable for a while and then suddenly "explodes" -- the molecular structure disolves and the atoms are getting a very large force from somewhere.

[njzjz]

Can you plot a figure to compare the reference training data and predicted value?

[yzjin]

Here I plotted the predicted vs ground truth energies and forces for some atoms in different sets in the training data. Seems like the model can only learn forces for some atoms in some directions only. And the energies is not very correct as well, even though the loss is small.

[Sulesk1]

Have you managed to find the reason behind it ? Since I have same issue.

[hyhuang00]

Facing a similar problem here. My molecules fail to keep their bonds once I put the trained model into LAMMPS.