compress training error curves are not same as the example

[TongTongMaster]

Version：deepmd-2.1.1
Install: Install with conda
input file：data1(train set)，data2(validation set)，input_1.json，input_2.json，graph-compress.pb
running commands：

1. original training（200000）：dp train input_1.json -l train.log

dp freeze -o graph.pb

2. compressed training（800000）：dp compress -i graph.pb -o graph-compress.pb

dp train input_2.json --init-frz-model graph-compress.pb -l train_new.log

I trained a SiO system with 1000000 steps that 200000 steps use original training strategy and 800000 steps use compressed training strategy.Then I draw the rmse_e_val curve，rmse_e_trn curve，rmse_f_val curve and rmse_f_trn curve，but I find that the general trend of the curves are not same as the picture 2（[https://zhuanlan.zhihu.com/p/406704835]）.The compressed training curve of the first 200000 steps is basically consistent with the original training curve，but the begining of compressed training curve（the begining of 800000 steps ）suddenly increased. （I uploaded the curve picture named compare）.

whether the checkpoint file generated with 200000 steps original training steps or a file （named model-compression） generated with the compress command（dp compress -i graph.pb -o graph-compress.pb）, I failed to draw a curve like that on the site.

Is there a problem with my training process？How to use compressed training strategy？Does compression training apply to such descriptors？whether the checkpoint file generated with 200000 steps original training steps or a file （named model-compression） generated with the compress command（dp compress -i graph.pb -o graph-compress.pb）, I failed to draw a curve like that on the site.

[TongTongMaster]

the descriptor I use：
"model": {
"type_map": ["O", "Si"],
"descriptor" :{
"type": "hybrid",
"list" : [
{
"type": "se_e2_a",
"sel": [50, 50],
"rcut_smth": 3.50,
"rcut": 4.00,
"neuron": [10, 20, 40],
"resnet_dt": false,
"axis_neuron": 4,
"seed": 1,
"_comment": " that's all"
},
{
"type": "se_e2_r",
"sel": [120, 120],
"rcut_smth": 5.00,
"rcut": 6.00,
"neuron": [5, 10, 20],
"resnet_dt": false,
"seed": 1,
"_comment": " that's all"
}
]
},

[njzjz]

Could you also post the learning rate?

[TongTongMaster]

yes，I set "start_lr" to 0.005 and "stop_lr" to 6.43e-4 in input_1.json and
set "start_lr" to 6.43e-4 and "stop_lr" to 1.75e-7 in input_2.json.

[njzjz]

#1680 should fix this issue.

However, I found another bug. It will be tracked in #1681.

[TongTongMaster]

Thanks for your answer.I have a question. Should I use the checkpoint file generated with 200000 steps original training steps or a file （named model-compression） generated with the compress command（dp compress -i graph.pb -o graph-compress.pb）？Or both of them are useful？

[njzjz]

One is before compression and another one is after compression. They are not the same thing.

[TongTongMaster]

Thanks. Which checkpoint file is used for starting compressed training strategy（80w）？Before I tested with two checkpoint files separately.

[njzjz]

model-compression/model.ckpt should be used.