dpgen examples: RunTimeError #1285

PolyuWeldingSpock · 2023-07-24T02:09:14Z

PolyuWeldingSpock
Jul 24, 2023

Hi everyone,

I am new in using dpgen. I tried to run the example of CH4 gas molecule in dpgen documentation. After revising my own machine.json. It showed RunTimeError:

/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
  warnings.warn(
DeepModeling
------------
Version: 0.11.1
Path:    /home/polyucmp/.local/lib/python3.10/site-packages/dpgen

Dependency
------------
     numpy     1.23.5   /home/polyucmp/.local/lib/python3.10/site-packages/numpy
    dpdata     0.2.15   /home/polyucmp/.local/lib/python3.10/site-packages/dpdata
  pymatgen            unknown version or path
     monty   2023.5.8   /home/polyucmp/.local/lib/python3.10/site-packages/monty
       ase     3.22.1   /home/polyucmp/.local/lib/python3.10/site-packages/ase
  paramiko      3.2.0   /home/polyucmp/.local/lib/python3.10/site-packages/paramiko
 custodian   2023.6.5   /home/polyucmp/.local/lib/python3.10/site-packages/custodian

Reference
------------
Please cite:
Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E,
DP-GEN: A concurrent learning platform for the generation of reliable deep learning
based potential energy models, Computer Physics Communications, 2020, 107206.
------------

Description
------------
INFO:dpgen:-------------------------iter.000000 task 01--------------------------
2023-07-23 17:57:50,709 - INFO : info:check_all_finished: False
2023-07-23 17:57:50,716 - INFO : job: f1410b377d77b88b9784773b4113c98758f3a578 submit; job_id is 681954
2023-07-23 18:00:52,229 - INFO : job: f1410b377d77b88b9784773b4113c98758f3a578 681954 finished
INFO:dpgen:-------------------------iter.000000 task 02--------------------------
INFO:dpgen:-------------------------iter.000000 task 03--------------------------
INFO:dpgen:-------------------------iter.000000 task 04--------------------------
2023-07-23 18:00:52,268 - INFO : info:check_all_finished: False
2023-07-23 18:00:52,273 - INFO : job: 4039874d99f5ac7c7dd61c28f788be784105ee92 submit; job_id is 688973
2023-07-23 18:00:52,278 - INFO : job: b68e73518ddc2030704d0f55da9930afa6ba666e submit; job_id is 688975
2023-07-23 18:02:23,761 - INFO : job: 4039874d99f5ac7c7dd61c28f788be784105ee92 688973 finished
2023-07-23 18:02:23,782 - INFO : job: b68e73518ddc2030704d0f55da9930afa6ba666e 688975 finished
INFO:dpgen:-------------------------iter.000000 task 05--------------------------
INFO:dpgen:-------------------------iter.000000 task 06--------------------------
INFO:dpgen:system 000 candidate :    210 in    310  67.74 %
INFO:dpgen:system 000 failed    :     74 in    310  23.87 %
INFO:dpgen:system 000 accurate  :     26 in    310   8.39 %
INFO:dpgen:system 000 accurate_ratio:   0.0839    thresholds: 1.0000 and 1.0000   eff. task min and max   -1   20   number of fp tasks:     20
INFO:dpgen:-------------------------iter.000000 task 07--------------------------
2023-07-23 18:02:23,816 - INFO : info:check_all_finished: False
2023-07-23 18:02:23,825 - INFO : job: 2e12908f3d5efc5f1bddfade7a243755fa8cad99 submit; job_id is 691551
2023-07-23 18:02:23,830 - INFO : job: 6bb0971664435fc33b1ba556a24cfaf1f3dcd77c submit; job_id is 691553
2023-07-23 18:02:23,832 - INFO : job: 27cf635f07b3a6fcbc29ecce4573cade256d405f submit; job_id is 691555
2023-07-23 18:02:23,837 - INFO : job: f1b75888edf2e4b0e817b3f2effdf287806e4dbb submit; job_id is 691558
2023-07-23 18:02:55,262 - INFO : job: 2e12908f3d5efc5f1bddfade7a243755fa8cad99 691551 terminated;fail_cout is 1; resubmitting job
2023-07-23 18:02:55,267 - INFO : job:2e12908f3d5efc5f1bddfade7a243755fa8cad99 re-submit after terminated; new job_id is 691955
2023-07-23 18:02:55,527 - INFO : job:2e12908f3d5efc5f1bddfade7a243755fa8cad99 job_id:691955 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:02:55,528 - INFO : job: 6bb0971664435fc33b1ba556a24cfaf1f3dcd77c 691553 terminated;fail_cout is 1; resubmitting job
2023-07-23 18:02:55,531 - INFO : job:6bb0971664435fc33b1ba556a24cfaf1f3dcd77c re-submit after terminated; new job_id is 691981
2023-07-23 18:02:55,797 - INFO : job:6bb0971664435fc33b1ba556a24cfaf1f3dcd77c job_id:691981 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:02:55,797 - INFO : job: 27cf635f07b3a6fcbc29ecce4573cade256d405f 691555 terminated;fail_cout is 1; resubmitting job
2023-07-23 18:02:55,801 - INFO : job:27cf635f07b3a6fcbc29ecce4573cade256d405f re-submit after terminated; new job_id is 692007
2023-07-23 18:02:56,069 - INFO : job:27cf635f07b3a6fcbc29ecce4573cade256d405f job_id:692007 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:02:56,070 - INFO : job: f1b75888edf2e4b0e817b3f2effdf287806e4dbb 691558 terminated;fail_cout is 1; resubmitting job
2023-07-23 18:02:56,076 - INFO : job:f1b75888edf2e4b0e817b3f2effdf287806e4dbb re-submit after terminated; new job_id is 692033
2023-07-23 18:02:56,342 - INFO : job:f1b75888edf2e4b0e817b3f2effdf287806e4dbb job_id:692033 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:03:26,477 - INFO : job: 2e12908f3d5efc5f1bddfade7a243755fa8cad99 691955 terminated;fail_cout is 2; resubmitting job
2023-07-23 18:03:26,484 - INFO : job:2e12908f3d5efc5f1bddfade7a243755fa8cad99 re-submit after terminated; new job_id is 692361
2023-07-23 18:03:26,743 - INFO : job:2e12908f3d5efc5f1bddfade7a243755fa8cad99 job_id:692361 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:03:26,743 - INFO : job: 6bb0971664435fc33b1ba556a24cfaf1f3dcd77c 691981 terminated;fail_cout is 2; resubmitting job
2023-07-23 18:03:26,749 - INFO : job:6bb0971664435fc33b1ba556a24cfaf1f3dcd77c re-submit after terminated; new job_id is 692387
2023-07-23 18:03:27,013 - INFO : job:6bb0971664435fc33b1ba556a24cfaf1f3dcd77c job_id:692387 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:03:27,013 - INFO : job: 27cf635f07b3a6fcbc29ecce4573cade256d405f 692007 terminated;fail_cout is 2; resubmitting job
2023-07-23 18:03:27,019 - INFO : job:27cf635f07b3a6fcbc29ecce4573cade256d405f re-submit after terminated; new job_id is 692413
2023-07-23 18:03:27,290 - INFO : job:27cf635f07b3a6fcbc29ecce4573cade256d405f job_id:692413 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:03:27,290 - INFO : job: f1b75888edf2e4b0e817b3f2effdf287806e4dbb 692033 terminated;fail_cout is 2; resubmitting job
2023-07-23 18:03:27,297 - INFO : job:f1b75888edf2e4b0e817b3f2effdf287806e4dbb re-submit after terminated; new job_id is 692439
2023-07-23 18:03:27,569 - INFO : job:f1b75888edf2e4b0e817b3f2effdf287806e4dbb job_id:692439 after re-submitting; the state now is <JobStatus.running: 3>
2023-07-23 18:03:57,704 - INFO : job: 2e12908f3d5efc5f1bddfade7a243755fa8cad99 692361 terminated;fail_cout is 3; resubmitting job
Traceback (most recent call last):
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 352, in handle_unexpected_submission_state
    job.handle_unexpected_job_state()
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 843, in handle_unexpected_job_state
    raise RuntimeError(
RuntimeError: job:2e12908f3d5efc5f1bddfade7a243755fa8cad99 692361 failed 3 times.job_detail:{'2e12908f3d5efc5f1bddfade7a243755fa8cad99': {'job_task_list': [{'command': 'mpirun -np 8 vasp_std', 'task_work_path': 'task.000.000019', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 8 vasp_std', 'task_work_path': 'task.000.000005', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 8 vasp_std', 'task_work_path': 'task.000.000014', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 8 vasp_std', 'task_work_path': 'task.000.000004', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 8 vasp_std', 'task_work_path': 'task.000.000009', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}], 'resources': {'number_node': 1, 'cpu_per_node': 48, 'gpu_per_node': 0, 'queue_name': '', 'group_size': 5, 'custom_flags': [], 'strategy': {'if_cuda_multi_devices': False, 'ratio_unfinished': 0.0}, 'para_deg': 1, 'module_purge': False, 'module_unload_list': [], 'module_list': [], 'source_list': [], 'envs': {}, 'prepend_script': [], 'append_script': [], 'wait_time': 0, 'kwargs': {}}, 'job_state': <JobStatus.terminated: 4>, 'job_id': 692361, 'fail_count': 3}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/polyucmp/.local/bin/dpgen", line 8, in <module>
    sys.exit(main())
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpgen/main.py", line 233, in main
    args.func(args)
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 5109, in gen_run
    run_iter(args.PARAM, args.MACHINE)
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 4461, in run_iter
    run_fp(ii, jdata, mdata)
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 3723, in run_fp
    run_fp_inner(
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 3702, in run_fp_inner
    submission.run_submission()
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 257, in run_submission
    self.handle_unexpected_submission_state()
  File "/home/polyucmp/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 355, in handle_unexpected_submission_state
    raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/media/polyucmp/DATA/BIAN_Tieyuan/MLFF/dpgen/test/dpgen_example/remote/ddf65b655407aa0b0ec509501f7c7b9cb09a48c0.
Debug information: submission_hash==ddf65b655407aa0b0ec509501f7c7b9cb09a48c0.
Please check the dirs and scripts in remote_root. The job information mentioned above may help.

Here is my param.json and machine.json. I only modified the machine.json for my own local workstation.

{
	"api_version": "1.0",
	"deepmd_version": "2.0.1",
	"train" :[
		{
			"command": "dp",
			"machine": {
				"batch_type": "Shell",
				"context_type": "local",
				"local_root" : "./",
				"remote_root": "/media/polyucmp/DATA/BIAN_Tieyuan/MLFF/dpgen/test/dpgen_example/remote"
			},
			"resources": {
				"number_node": 1,
				"cpu_per_node": 40,
				"gpu_per_node": 1,
				"group_size": 4
			}
		}
	],
	"model_devi":[
		{
			"command": "mpirun -n 4 lmp -i input.lammps",
			"machine": {
				"batch_type": "Shell",
				"context_type": "local",
				"local_root" : "./",
				"remote_root": "/media/polyucmp/DATA/BIAN_Tieyuan/MLFF/dpgen/test/dpgen_example/remote"
			},
			"resources": {
				"number_node": 1,
				"cpu_per_node": 4,
				"gpu_per_node": 1,
				"group_size": 5
			}
		}
	],
	"fp":[
		{
			"command": "mpirun -np 8 vasp_std",
			"machine": {
				"batch_type": "Shell",
				"context_type": "local",
				"local_root" : "./",
				"remote_root": "/media/polyucmp/DATA/BIAN_Tieyuan/MLFF/dpgen/test/dpgen_example/remote"
			},
			"resources": {
				"number_node": 1,
				"cpu_per_node": 48,
				"gpu_per_node": 0,
				"group_size": 5
			}
		}
	]
}

Hope someoen could help me solve the problem. Thanks!

Kristen-Liu152 · 2023-07-26T15:56:33Z

Kristen-Liu152
Jul 26, 2023

The same problem occurred when I started my dptrain job!!!

1 reply

PolyuWeldingSpock Jul 27, 2023
Author

Hi,

You can go to the error directory and manually check the error like @wanghan-iapcm said.

wanghan-iapcm · 2023-07-27T01:47:35Z

wanghan-iapcm
Jul 27, 2023
Maintainer

go to /media/polyucmp/DATA/BIAN_Tieyuan/MLFF/dpgen/test/dpgen_example/remote/ddf65b655407aa0b0ec509501f7c7b9cb09a48c0
and check if you can manually run the job there

0 replies

PolyuWeldingSpock · 2023-07-27T01:52:01Z

PolyuWeldingSpock
Jul 27, 2023
Author

go to /media/polyucmp/DATA/BIAN_Tieyuan/MLFF/dpgen/test/dpgen_example/remote/ddf65b655407aa0b0ec509501f7c7b9cb09a48c0 and check if you can manually run the job there

Hi,

Thanks for your reply. I have solved the problem. My VASP executable file have some problems. Thanks!

4 replies

chb-deep Jul 28, 2023

I also encountered the problem in task07, how exactly did you solve it

PolyuWeldingSpock Sep 9, 2023
Author

@chb-deep Please go to the working directories of fp step. You can check what exactly happened in fp step.

chb-deep Sep 14, 2023

Thank you for your help, I have fixed this issue

xuejunju Sep 25, 2023

Sorry to bother you, I would like to ask how you solved it. I have been trying for a long time but I still can’t find a solution.

PolyuWeldingSpock · 2023-09-26T02:51:59Z

PolyuWeldingSpock
Sep 26, 2023
Author

Sorry to bother you, I would like to ask how you solved it. I have been trying for a long time but I still can’t find a solution.

You can step into fp folder and check error message

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

dpgen examples: RunTimeError #1285

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

dpgen examples: RunTimeError #1285

Uh oh!

PolyuWeldingSpock Jul 24, 2023

Replies: 4 comments · 5 replies

Uh oh!

Kristen-Liu152 Jul 26, 2023

Uh oh!

PolyuWeldingSpock Jul 27, 2023 Author

Uh oh!

wanghan-iapcm Jul 27, 2023 Maintainer

Uh oh!

PolyuWeldingSpock Jul 27, 2023 Author

Uh oh!

chb-deep Jul 28, 2023

Uh oh!

PolyuWeldingSpock Sep 9, 2023 Author

Uh oh!

chb-deep Sep 14, 2023

Uh oh!

xuejunju Sep 25, 2023

Uh oh!

PolyuWeldingSpock Sep 26, 2023 Author

PolyuWeldingSpock
Jul 24, 2023

Replies: 4 comments 5 replies

Kristen-Liu152
Jul 26, 2023

PolyuWeldingSpock Jul 27, 2023
Author

wanghan-iapcm
Jul 27, 2023
Maintainer

PolyuWeldingSpock
Jul 27, 2023
Author

PolyuWeldingSpock Sep 9, 2023
Author

PolyuWeldingSpock
Sep 26, 2023
Author