Does DeePMD support automatic restart of interrupted training?

[wxwth]

Dear DeePMD developers,

I am a user of DeePMD-kit (usually version 2.2.7) and DP-GEN (usually version 0.11.1), and have experience in training Deep Potentials on GPU nodes in our university’s high-performance computing clusters. However, GPU resources are often limited. On the other hand, there are numerous CPU nodes available with a time limit of only 2 days per computing task. Unfortunately, training on CPUs typically exceeds this time limit.

I am wondering if DeePMD-kit or DP-GEN supports automatic restart of interrupted training, allowing me to use these time-limited CPUs for Deep Potential training. If so, would such interruptions influence the precision of Deep Potential?

Thank you for your time and I am looking forward to your reply.

Sincerely,
Xu

[njzjz]

You can use the --restart option to restart from a previous checkpoint.
DP-GEN has used this option.

[chenggoj]

Hi, I have the same problem. But I am really confused about your explanations. I noticed that in the documentation of DP-GEN, it says “If the process of DP-GEN stops for some reason, DP-GEN will automatically recover the main process by record.dpgen. You may also change it manually for your purpose, such as removing the last iterations and recovering from one checkpoint. ”
Could you pls give any specific examples?
In my case, I have to train 4 different models and have finished 2 of them. the Tree structure of the DeePMD-kit like this:
├── 000
│   ├── checkpoint
│   ├── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run/iter.000000/00.train/000/input.json
│   ├── input_v2_compat.json
│   ├── lcurve.out
│   ├── model.ckpt-500000.data-00000-of-00001
│   ├── model.ckpt-500000.index
│   ├── model.ckpt-500000.meta
│   ├── model.ckpt-510000.data-00000-of-00001
│   ├── model.ckpt-510000.index
│   ├── model.ckpt-510000.meta
│   ├── model.ckpt-520000.data-00000-of-00001
│   ├── model.ckpt-520000.index
│   ├── model.ckpt-520000.meta
│   ├── model.ckpt-530000.data-00000-of-00001
│   ├── model.ckpt-530000.index
│   ├── model.ckpt-530000.meta
│   ├── model.ckpt-540000.data-00000-of-00001
│   ├── model.ckpt-540000.index
│   ├── model.ckpt-540000.meta
│   ├── model.ckpt.data-00000-of-00001 -> model.ckpt-540000.data-00000-of-00001
│   ├── model.ckpt.index -> model.ckpt-540000.index
│   ├── model.ckpt.meta -> model.ckpt-540000.meta
│   ├── out.json
│   └── train.log
├── 001
│   ├── checkpoint
│   ├── ee31fb159c50d8763b069d4831ec066ca9d72806_task_tag_finished
│   ├── frozen_model.pb
│   ├── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run/iter.000000/00.train/001/input.json
│   ├── input_v2_compat.json
│   ├── lcurve.out
│   ├── model.ckpt-1000000.data-00000-of-00001
│   ├── model.ckpt-1000000.index
│   ├── model.ckpt-1000000.meta
│   ├── model.ckpt-960000.data-00000-of-00001
│   ├── model.ckpt-960000.index
│   ├── model.ckpt-960000.meta
│   ├── model.ckpt-970000.data-00000-of-00001
│   ├── model.ckpt-970000.index
│   ├── model.ckpt-970000.meta
│   ├── model.ckpt-980000.data-00000-of-00001
│   ├── model.ckpt-980000.index
│   ├── model.ckpt-980000.meta
│   ├── model.ckpt-990000.data-00000-of-00001
│   ├── model.ckpt-990000.index
│   ├── model.ckpt-990000.meta
│   ├── model.ckpt.data-00000-of-00001 -> model.ckpt-1000000.data-00000-of-00001
│   ├── model.ckpt.index -> model.ckpt-1000000.index
│   ├── model.ckpt.meta -> model.ckpt-1000000.meta
│   ├── out.json
│   └── train.log
├── 002
│   ├── ab19c8d51306922480ec00427558ad5e99cbfec3_task_tag_finished
│   ├── checkpoint
│   ├── frozen_model.pb
│   ├── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run/iter.000000/00.train/002/input.json
│   ├── input_v2_compat.json
│   ├── lcurve.out
│   ├── model.ckpt-1000000.data-00000-of-00001
│   ├── model.ckpt-1000000.index
│   ├── model.ckpt-1000000.meta
│   ├── model.ckpt-960000.data-00000-of-00001
│   ├── model.ckpt-960000.index
│   ├── model.ckpt-960000.meta
│   ├── model.ckpt-970000.data-00000-of-00001
│   ├── model.ckpt-970000.index
│   ├── model.ckpt-970000.meta
│   ├── model.ckpt-980000.data-00000-of-00001
│   ├── model.ckpt-980000.index
│   ├── model.ckpt-980000.meta
│   ├── model.ckpt-990000.data-00000-of-00001
│   ├── model.ckpt-990000.index
│   ├── model.ckpt-990000.meta
│   ├── model.ckpt.data-00000-of-00001 -> model.ckpt-1000000.data-00000-of-00001
│   ├── model.ckpt.index -> model.ckpt-1000000.index
│   ├── model.ckpt.meta -> model.ckpt-1000000.meta
│   ├── out.json
│   └── train.log
├── 003
│   ├── checkpoint
│   ├── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run/iter.000000/00.train/003/input.json
│   ├── input_v2_compat.json
│   ├── lcurve.out
│   ├── model.ckpt-400000.data-00000-of-00001
│   ├── model.ckpt-400000.index
│   ├── model.ckpt-400000.meta
│   ├── model.ckpt-410000.data-00000-of-00001
│   ├── model.ckpt-410000.index
│   ├── model.ckpt-410000.meta
│   ├── model.ckpt-420000.data-00000-of-00001
│   ├── model.ckpt-420000.index
│   ├── model.ckpt-420000.meta
│   ├── model.ckpt-430000.data-00000-of-00001
│   ├── model.ckpt-430000.index
│   ├── model.ckpt-430000.meta
│   ├── model.ckpt-440000.data-00000-of-00001
│   ├── model.ckpt-440000.index
│   ├── model.ckpt-440000.meta
│   ├── model.ckpt.data-00000-of-00001 -> model.ckpt-440000.data-00000-of-00001
│   ├── model.ckpt.index -> model.ckpt-440000.index
│   ├── model.ckpt.meta -> model.ckpt-440000.meta
│   ├── out.json
│   └── train.log

If I would like to recover the training process from "000" model "model.ckpt-540000", what should I do?
The command in machine.json is "CUDA_VISIBLE_DEVICES=0,1,2,3 mpirun dp"
and record.dpgen is "0 0"."

Thanks a lot for your help.

[njzjz]

You don't need to do anything except re-run DP-GEN.