An error occurred while running task07

[ziqi-hu]

err.txt
machine_LSF.json.txt
param.json.txt

In the process of using dp-gen to iterate a system, I set up 8 MD processes. After submitting the task using dpgen run param.json machine.json, the program will report an error when it goes to the task07 part, sometimes it is in iter0, sometimes it is iter1, sometimes iter2. The corresponding input file and err file are attached. I checked the gasussian part, there is a task that is not finished. But I don't know the reason, I hope you can help me check it.

INFO:dpgen:-------------------------iter.000000 task 07--------------------------
Traceback (most recent call last):
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpdispatcher/submission.py", line 238, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpdispatcher/submission.py", line 589, in handle_unexpected_job_state
raise RuntimeError(f"job:{self.job_hash} {self.job_id} failed {self.fail_count} times.job_detail:{self}")
RuntimeError: job:55d298781953a07a48d69cb721e02ef9a18a8db3 94461 failed 3 times.job_detail:{'55d298781953a07a48d69cb721e02ef9a18a8db3': {'job_task_list': [.....................................

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/localhome2/yxhu/deepmd-kit/bin/dpgen", line 8, in
sys.exit(main())
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpgen/main.py", line 175, in main
args.func(args)
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpgen/generator/run.py", line 3236, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpgen/generator/run.py", line 3222, in run_iter
run_fp (ii, jdata, mdata)
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpgen/generator/run.py", line 2693, in run_fp
run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _gaussian_check_fin, log_file = 'output')
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpgen/generator/run.py", line 2649, in run_fp_inner
submission.run_submission()
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpdispatcher/submission.py", line 205, in run_submission
self.handle_unexpected_submission_state()
File "/localhome2/yxhu/deepmd-kit/lib/python3.9/site-packages/dpdispatcher/submission.py", line 241, in handle_unexpected_submission_state
raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/home/yxhu/ch4n2/GAUSSIAN/d5839b15e89a287a1127f727795b6e41a6e9470d.
Debug information: submission_hash==d5839b15e89a287a1127f727795b6e41a6e9470d.
Please check the dirs and scripts in remote_rootThe job information mentioned above may help

[HuangJiameng]

There's something wrong with dpdispatcher.

[ziqi-hu]

dpgen.log and record.dpgen uploaded
so do i need reinstall dpdispatcher？
My train and QM are not on the same computer, do I need to update dpdispatcher to the same version?
Why does the error sometimes occur at iter0 and sometimes at iter1?

[HuangJiameng]

As far as I'm concerned, dpdispatcher v0.4.7 or v0.4.8. are stable, v0.4.12 is also OK. (However, the machine.json they required are different.)

[HuangJiameng]

Why does the error sometimes occur at iter0 and sometimes at iter1?

You can try to reinstall dpdispatcher first. If the problem still exists, please provide your data for reproducing.

[ziqi-hu]

Thank you for your answer. The version I am currently using is v0.4.8, and the corresponding machine file.

[ziqi-hu]

OK, I'm retrying my task. I will update when I have results.

[ziqi-hu]

Relevant documents such as attachments。
dpgen.log
record.dpgen.txt

[ziqi-hu]

More relevant running results I have uploaded to Baidu network disk.
Link: https://pan.baidu.com/s/1Dk8Z754VnjQH2QB-UIATyQ?pwd=uywr
Extraction code: uywr

[HuangJiameng]

I checked your log file, it shows that in task07, 8 model_devi_jobs are running parallelly, while 2 of them failed and were resubmitted at the same time. Since they were from the same directory, their submission_hash were same. Thus the error was raised. It is a bug in dpdispatcher and the contributors have already noticed that.

[ziqi-hu]

OK, Thanks for your answer. That is to say, as long as I have multiple tasks operating in parallel, there is a certain probability of encountering this problem, and there is no way to avoid it for the time being.do you mean this?

[HuangJiameng]

I'm afraid so.

[ziqi-hu]

This problem seems to be solved. The cause of the problem is because the gaussian calculation fails, dpdispatcher provides a workaround, in this url:https://docs.deepmodeling.com/projects/dpdispatcher/en/latest/examples/g16.html

At present, my task has progressed to iter6, and there should be no problem.

[9Jaden]

Hi, I got into the same problem like yours. Could you explain more about how you solve this problem? I am little bit confused about https://docs.deepmodeling.com/projects/dpdispatcher/en/latest/examples/g16.html. It seems like I should add some new command into machine.json. But where should it be? I don't have "task" class in machine.json. I have tried some ways but it's still not working, so maybe could you share some experience with me. thanks.

[ziqi-hu]

I just found your question, I don't know if your problem has been solved.
The cause of my problem was that part of the gaussian task failed, causing the whole program to terminate.
https://docs.deepmodeling.com/projects/dpdispatcher/en/latest/examples/g16.html。In this webpage, the author mentioned this problem, and modified it under the command in the fp part, it can be that the program ignores the failed Gaussian task.

[9Jaden]

thanks! I have solved my problem~

[chb-deep]

This URL can no longer be opened, do you have another URL to solve this problem

[chb-deep]

https://docs.deepmodeling.com/projects/dpdispatcher/en/latest/examples/g16.html%E3%80%82In this URL