How to restart DPA-2 training?

[zhbzhbzhbzhb]

Summary

How to restart PDA-2 training?

DeePMD-kit Version

3.0.0a1.dev82+geed7c8ac

Backend and its version

torch2.0.0 torchvision 0.15.1 torchaudio-2.0.1

Python Version, CUDA Version, GCC Version, LAMMPS Version, etc

No response

Details

Should I use: dp train --restart model.ckpt input.json --finetune OpenLAM_2.1.0_27heads_2024Q1.pt --model-branch Domains_OC2M

[anyangml]

if you want to resume a finetune experiment from the latest checkpoint, I believe you can simple do

dp train input.json --restart latest_checkpoint.ckpt --skip-neighbor-stat

[anyangml]

I am confused. Is model.ckpt-800000.pt the latest ckpt from a crashed finetune experiment or the pretrained model?

[JiangXiaoMingSan]

This is the model generated after performing 80W steps of finetune.

[anyangml]

then I am not sure why it says DEEPMD INFO Start to train 300000 steps. what is the total number of finetune steps in your input_finetune. json

[JiangXiaoMingSan]

This model.ckpt-800000. pt is a model generated in 80w steps, and I want it to continue running in 30w steps, so I gave it 30w steps in the input_finetune.json. So the log file says to start training for 300000 steps, but it doesn't continue training and shows as completed. If I want to train a total of 110w steps, should I also write 110w in the input_finetune.json during restart？

[JiangXiaoMingSan]

Thank you very much, I have resolved this issue.