No test_err in lcurve.out for dpgen-train.

[hiluokun]

Hi All,

There is always no test_err in the lcurve.out of my dpgen-train since using dpgen.
I guess it has tested every disp_freq but no results output in lcurve.out.
So, I wonder if my results are right and how to abtain the test_err in lcurve.out.
as shown below,

(base) [********]$ head -n 2 lcurve.out && tail -n 2 lcurve.out
step rmse_trn rmse_e_trn rmse_f_trn rmse_v_trn lr
0 6.12e+01 1.99e+00 1.93e+00 2.31e-01 1.0e-03
499000 5.46e-01 3.57e-02 3.31e-01 3.61e-02 3.7e-08
500000 4.35e-01 1.45e-02 9.97e-02 2.27e-02 3.5e-08

but, I found a normal lcurve.out recently, https://zhuanlan.zhihu.com/p/555628454 as shown below,

cat iter.000000/00.train/000/
head -n 2 lcurve.out && tail -n 2 lcurve.out

batch l2_tst l2_trn l2_e_tst l2_e_trn l2_f_tst l2_f_trn lr
0 8.14e+00 8.00e+00 1.00e+01 1.00e+01 4.78e-02 3.41e-03 1.0e-03
398000 6.47e-03 7.17e-03 3.47e-06 1.81e-06 6.30e-03 6.98e-03 5.3e-08
400000 6.46e-03 7.74e-03 2.85e-06 1.36e-06 6.30e-03 7.55e-03 5.0e-08
batch 的最终数值 param.json 中 stop_batch 的指定值。

So, I wonder if my results are right. I guess it has tested every disp_freq

DEEPMD INFO initialize model from scratch
DEEPMD INFO start training at lr 1.00e-03 (== 1.00e-03), decay_step 2500, decay_rate 0.950006, final lr will be 3.51e-08
DEEPMD INFO batch 100 training time 9.35 s, testing time 0.09 s
DEEPMD INFO batch 200 training time 9.14 s, testing time 0.07 s
DEEPMD INFO batch 300 training time 9.20 s, testing time 0.10 s
DEEPMD INFO batch 400 training time 8.95 s, testing time 0.06 s
DEEPMD INFO batch 500 training time 8.41 s, testing time 0.03 s

and this is my param.json, I set the "numb_test": 1,

"training": {
"stop_batch": 500000,
"disp_file": "lcurve.out",
"disp_freq": 100,
"numb_test": 1,
"save_freq": 10000,
"save_ckpt": "model.ckpt",
"disp_training": true,
"time_training": true,
"profiling": false,
"profiling_file": "timeline.json",
"_comment": "that's all",

best,
Thank you.

[wanghan-iapcm]

All data generated by dpgen is used for training. One should prepare the test dataset by his/herself.

[hiluokun]

Thanks for your kind reply.
As you know, there are only two data systems "init_data_sys" and "sys_configs" in param.json of dpgen.
How do I set the path of the test-dataset in param.json?

Also see the example below (from https://zhuanlan.zhihu.com/p/555628454 )
It would be better if you could write an hypothetical path for the test-dataset in the following template.

{
"type_map": ["Li", "Si"],
"mass_map": [6.9, 28.1],
"init_data_prefix": "",
"init_data_sys": [
"POSCAR_LiSi.01x01x01/02.md/sys-0016-0016/deepmd"
],
"sys_configs_prefix": "",
"sys_configs": [
["POSCAR_LiSi.01x01x01/02.md/sys-0016-0016/scale-1.000/00000[0-2]/POSCAR"]
],

"_comment": " 00.train ",
"numb_models": 4,
"model_devi_activation_func" :        [["tanh","tanh"],["tanh","gelu"],["gelu","tanh"],["gelu","gelu"]],
  
"default_training_param": {
   "model": {
          "type_map": ["Li", "Si"],
          "descriptor": {
              "type": "se_e2_a",
              "sel": [100, 100],
              "rcut_smth": 2.0,
              "rcut": 6.0,
              "neuron":[25, 50, 100],
              "resnet_dt": false,
              "type_one_side": true,
              "axis_neuron": 12,
              "seed": 1
          },
          "fitting_net": {
              "neuron": [240, 240, 240],
              "resnet_dt": true,
              "seed": 1
          }
      },
      "learning_rate": {
          "type": "exp",
          "start_lr": 0.001,
          "decay_steps": 4000
      },
      "loss": {
          "start_pref_e": 0.02,
          "limit_pref_e": 1,
          "start_pref_f": 1000,
          "limit_pref_f": 1,
          "start_pref_v": 0.0,
          "limit_pref_v": 0.0
      },
      **"training":** {
          "stop_batch": 800000,
          "disp_file": "lcurve.out",
          "disp_freq": 2000,
          **"numb_test": 4,**
          "save_freq": 2000,
          "save_ckpt": "model.ckpt",
          "disp_training": true,
          "time_training": true,
          "profiling": false,
          "profiling_file": "timeline.json",
          "_comment": "that's all"
      }
},
  
"_comment": " 01.model_devi ",
"model_devi_dt": 0.001,
"model_devi_skip": 0,
"model_devi_f_trust_lo": 0.15,
"model_devi_f_trust_hi": 0.35,
"model_devi_clean_traj": false,
"model_devi_jobs": [
   {"sys_idx": [0],"temps": [200, 300, 400, 500],"press": [1.0, 100.0, 1000.0, 10000.0, 50000.0], "trj_freq": 5, "nsteps":   1000, "ensemble": "npt", "_idx":     "00" },
   {"sys_idx": [0],"temps": [200, 300, 400, 500],"press": [1.0, 100.0, 1000.0, 10000.0, 50000.0], "trj_freq": 5, "nsteps":   1000, "ensemble": "npt", "_idx":     "01" },
   {"sys_idx": [0],"temps": [200, 300, 400, 500],"press": [1.0, 100.0, 1000.0, 10000.0, 50000.0], "trj_freq": 10, "nsteps":   3000, "ensemble": "npt", "_idx":     "02" },
   {"sys_idx": [0],"temps": [200, 300, 400, 500],"press": [1.0, 100.0, 1000.0, 10000.0, 50000.0], "trj_freq": 10, "nsteps":   3000, "ensemble": "npt", "_idx":     "03" },
   {"sys_idx": [0],"temps": [200, 300, 400, 500],"press": [1.0, 100.0, 1000.0, 10000.0, 50000.0], "trj_freq": 20, "nsteps":   5000, "ensemble": "npt", "_idx":     "04" },
   {"sys_idx": [0],"temps": [200, 300, 400, 500],"press": [1.0, 100.0, 1000.0, 10000.0, 50000.0], "trj_freq": 20, "nsteps":   8000, "ensemble": "npt", "_idx":     "05" }
],
  
"_comment": " 02.fp ",
"fp_style": "vasp",
"fp_accurate_threshold": 0.999,
"fp_accurate_soft_threshold": 0.997,
"shuffle_poscar": false,
"fp_task_max": 50,
"fp_task_min": 3,
"ratio_failed": 0.05,
"fp_pp_path": "<your-POTCAR-path>",
"fp_pp_files": [
  "POTCAR_Li","POTCAR_Si"
],
"fp_incar": "./INCAR_LiSi"

}

Best and Thanks again

[HuangJiameng]

For the meaning of the parameters, you can refer to https://docs.deepmodeling.com/projects/dpgen/en/devel/run/index.html. Besides, here is a possible structure of a single dataset.

[hiluokun]

Thanks for your kind reply.

I mean how to make the test-data effective in the dpgen-run.
I checked the manual you provided, but there was no related introduction to "numb_test" and the path of the test-dataset.
But the example from https://zhuanlan.zhihu.com/p/555628454 also only set "numb_test": 4,, not the path of the test-dataset.
Why there is test_err in the lcurve.out?

batch l2_tst l2_trn l2_e_tst l2_e_trn l2_f_tst l2_f_trn lr

0 8.14e+00 8.00e+00 1.00e+01 1.00e+01 4.78e-02 3.41e-03 1.0e-03
398000 6.47e-03 7.17e-03 3.47e-06 1.81e-06 6.30e-03 6.98e-03 5.3e-08
400000 6.46e-03 7.74e-03 2.85e-06 1.36e-06 6.30e-03 7.55e-03 5.0e-08
batch 的最终数值 param.json 中 stop_batch 的指定值。

As you know, test_ err is important to detect whether the model is overfitted.
Thanks again for your patience.

Best

[hiluokun]

Hi all,

This issue I've solved. The validation is working well, Details are as follows:
We need to add the path of the test-dataset ("validation_data") in param.json as we did before in DeepMD.
It is recommended to add this part to the dpgen instructions, especially in the example from https://zhuanlan.zhihu.com/p/555628454

Thanks everyone!

Best.