model predicts good on separate datasets but bad on combined datasets

[beidouamg]

Hello DeePMD developers,

Thanks for the free available DeePMD code.
I am trying to train a MLP for water-2D material interfacial system. The training sets include ~100000 structures in total, sampled from (1) pure water, (2) pure 2D material and (3) water-2D material interface. What confuses me is it gives good prediction if training model with each of these 3 datasets separately but gives ordinary prediction if training on the combination of 3 datasets. It would be great if you can help to improve the model performance.

Some details are below:

1. When sampling structures, I trained 3 MLPs separately using each of these training sets. I would say that the trained MLPs predicted quite good results and can capture the property I am interested in.

for (1) pure water:

for (2) pure 2D material

for (3) water-2D material interface

2. Then I combined all the above datasets and hoped to get a model that can describe the whole system, but this trained model predict worse than the model trained with separated dataset.
   You can see the prediction and loss functions along the training process.

I used same parameters for all the trainings as below:

{

"model": {
    "type_map":     ["X", "X", "X"],
    "descriptor" :{
        "type":             "se_e2_a",
        "sel":              "auto",
        "rcut_smth":        0.50,
        "rcut":             8.00,
        "neuron":           [30, 60, 120],
        "resnet_dt":        false,
        "axis_neuron":      16,
        "seed":             11300402      },

   "fitting_net" : {
        "type":             "ener",
        "neuron":           [240, 240, 240, 240],
        "resnet_dt":        true,
        "seed":             11300402      }},

"learning_rate" :{
    "type":         "exp",
    "decay_steps":  15000,
    "start_lr":     0.001,
    "stop_lr":      3.51e-8  },

"loss" :{
    "type"        :"ener",
    "start_pref_e": 0.02,
    "limit_pref_e": 1000,
    "start_pref_f": 1000,
    "limit_pref_f": 1000,
    "start_pref_v": 0,
    "limit_pref_v": 0   },

"training" : {
    "training_data": {
        "systems":     ["./training_data/xxx''],
        "batch_size":  "auto",
        "auto_prob":   "prob_sys_size" },
    "validation_data":{
        "systems":     ["./validation_data/xxx''],
        "batch_size":  "auto",
        "numb_btch":   1     },
    "numb_steps":    3000000,
    "disp_file":    "lcurve.out",
    "disp_freq":     100,
    "save_freq":     1000,
    "seed":          11300402      }

}

It would be great if you can check the training parameters and give some suggestions to improve the model performance. Please let me if you need more info. I appreciate your time on this issue.

[wanghan-iapcm]

I find the combined test error is smaller than that trained on the 2-D material dataset.
The magnitude of the error also depends on how you test your model.

[beidouamg]

Thanks for your comments, Dr. Wang.

I think the smaller test error on combined datasets is due to averaging over all datasets. The sampled frames from (1) pure water, (2) pure 2D material and (3) water-2D material interface are roughly 40000:30000:30000; In the second to the last plot for combined dataset, it shows the test error on all sampled frames, which averages good prediction on (1), (3) and ordinary prediction on (2);

My concern is if I can further reduce the test error on combined datasets to the magnitude on (1), (3), and my naive idea is to train more steps or replace "se_e2_a" with other model descriptor.

I would listen to your better idea, your suggestions are appreciated.

[wanghan-iapcm]

It is fully valid to try models with better representatbility, but you may not always achieve higher accuracy, because the model's accuracy is bounded by the accuracy of the labels. I suspect the 2D-mat dataset has a lower accuracy compared to other datasets.

[beidouamg]

Thanks for your reply, Dr. Wang.

I think they have the same accuracy, since I used same DFT parameters for the calculation of all datasets and converged the total energy of each frames to < 10 meV.
I have tried to train more steps to 9000000 steps, it gets just a little better.

The test error of force in 2D-mat dataset can be halved as I trained the same 2D-mat dataset with NequIP, but the test error of energy cannot be lowered, which i think is due to the large energy range (~120 eV) of 2D-mat dataset.

Based on this, i think the test error will be lowered if i try models with better representability. i wonder if you have any suggestions on the other model descriptors i can try. Thanks a lot for your time.

[wanghan-iapcm]

you may try dpa-1 with attention layers and dpa-2

[beidouamg]

Thanks for your suggestion, Dr. Wang. I will try it.

[damonturney]

beidouamg Did you find success with this? Did you publish the results anywhere?