AttributeError: 'NoneType' object has no attribute 'model_checkpoint_path'

[wangxinxiny]

When i run the dpgen,i meet a error:/home/customer/.local/lib/python3.9/site-packages/gromacs/init.py:286: GromacsImportWarning: Some Gromacs commands were NOT found; maybe source GMXRC first? The following are missing:
['release']

warnings.warn("Some Gromacs commands were NOT found; "
INFO:dpgen:-------------------------iter.000000 task 01--------------------------
/home/customer/.local/lib/python3.9/site-packages/dpgen/generator/run.py:584: UserWarning: the dpdispatcher will be updated to new version.And the interface may be changed. Please check the documents for more details
warnings.warn(f"the dpdispatcher will be updated to new version."
INFO:dpgen:cannot find key "batch" in machine file, try to use deprecated key "machine_type"
/home/customer/.local/lib/python3.9/site-packages/paramiko/client.py:837: UserWarning: Unknown ssh-ed25519 host key for 172.16.10.20: b'bd9729f31ac9072764c4a091c0c5b3e0'
warnings.warn(
INFO:dpgen:new submission of fb8ed06d-10d2-4eb5-b2e5-81afdb672131 for chunk 8aefb06c426e07a0a671a1e2488b4858d694a730
INFO:dpgen:new submission of cd139241-38a9-4a22-9fa1-d999333ecbb8 for chunk e193a01ecf8d30ad0affefd332ce934e32ffce72
INFO:dpgen:new submission of a94eca34-18d9-433d-872c-e24e72e387b6 for chunk 6fc978af728d43c59faa400d5f6e0471ac850d4c
INFO:dpgen:new submission of 79a2c2c8-38e3-4d2d-a530-5eb2191dda22 for chunk 221407c03ae5c73109cce71d27e24637824f3333
INFO:dpgen:job fb8ed06d-10d2-4eb5-b2e5-81afdb672131 finished
WARNING:dpgen:Get error code 2 in calling tar czfh fb8ed06d-10d2-4eb5-b2e5-81afdb672131.tar.gz 000/frozen_model.pb 000/lcurve.out 000/train.log 000/model.ckpt.meta 000/model.ckpt.index 000/model.ckpt.data-00000-of-00001 000/checkpoint through ssh with job: fb8ed06d-10d2-4eb5-b2e5-81afdb672131 . message: tar: 000/frozen_model.pb：无法 stat: 没有那个文件或目录
tar: 000/lcurve.out：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.meta：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.index：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.data-00000-of-00001：无法 stat: 没有那个文件或目录
tar: 000/checkpoint：无法 stat: 没有那个文件或目录
tar: 由于前次错误，将以上次的错误状态退出

WARNING:dpgen:Sleep 60 s and retry the command...
WARNING:dpgen:Get error code 2 in calling tar czfh fb8ed06d-10d2-4eb5-b2e5-81afdb672131.tar.gz 000/frozen_model.pb 000/lcurve.out 000/train.log 000/model.ckpt.meta 000/model.ckpt.index 000/model.ckpt.data-00000-of-00001 000/checkpoint through ssh with job: fb8ed06d-10d2-4eb5-b2e5-81afdb672131 . message: tar: 000/frozen_model.pb：无法 stat: 没有那个文件或目录
tar: 000/lcurve.out：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.meta：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.index：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.data-00000-of-00001：无法 stat: 没有那个文件或目录
tar: 000/checkpoint：无法 stat: 没有那个文件或目录
tar: 由于前次错误，将以上次的错误状态退出

WARNING:dpgen:Sleep 60 s and retry the command...
WARNING:dpgen:Get error code 2 in calling tar czfh fb8ed06d-10d2-4eb5-b2e5-81afdb672131.tar.gz 000/frozen_model.pb 000/lcurve.out 000/train.log 000/model.ckpt.meta 000/model.ckpt.index 000/model.ckpt.data-00000-of-00001 000/checkpoint through ssh with job: fb8ed06d-10d2-4eb5-b2e5-81afdb672131 . message: tar: 000/frozen_model.pb：无法 stat: 没有那个文件或目录
tar: 000/lcurve.out：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.meta：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.index：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.data-00000-of-00001：无法 stat: 没有那个文件或目录
tar: 000/checkpoint：无法 stat: 没有那个文件或目录
tar: 由于前次错误，将以上次的错误状态退出

WARNING:dpgen:Sleep 60 s and retry the command...
Traceback (most recent call last):
File "/home/customer/.local/bin/dpgen", line 8, in
sys.exit(main())
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/main.py", line 175, in main
args.func(args)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 2997, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 2962, in run_iter
run_train (ii, jdata, mdata)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 587, in run_train
dispatcher.run_jobs(mdata['train_resources'],
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/Dispatcher.py", line 95, in run_jobs
while not self.all_finished(job_handler, mark_failure) :
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/Dispatcher.py", line 230, in all_finished
rjob['context'].download(task_chunks[idx], backward_task_files)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/SSHContext.py", line 185, in download
self._get_files(file_list)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/SSHContext.py", line 310, in _get_files
self.block_checkcall('tar czfh %s %s' % (of, " ".join(files)))
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/SSHContext.py", line 201, in block_checkcall
return self.block_checkcall(cmd, retry=retry+1)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/SSHContext.py", line 201, in block_checkcall
return self.block_checkcall(cmd, retry=retry+1)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/SSHContext.py", line 201, in block_checkcall
return self.block_checkcall(cmd, retry=retry+1)
File "/home/customer/.local/lib/python3.9/site-packages/dpgen/dispatcher/SSHContext.py", line 202, in block_checkcall
raise RuntimeError("Get error code %d in calling %s through ssh with job: %s . message: %s" %
RuntimeError: Get error code 2 in calling tar czfh fb8ed06d-10d2-4eb5-b2e5-81afdb672131.tar.gz 000/frozen_model.pb 000/lcurve.out 000/train.log 000/model.ckpt.meta 000/model.ckpt.index 000/model.ckpt.data-00000-of-00001 000/checkpoint through ssh with job: fb8ed06d-10d2-4eb5-b2e5-81afdb672131 . message: tar: 000/frozen_model.pb：无法 stat: 没有那个文件或目录
tar: 000/lcurve.out：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.meta：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.index：无法 stat: 没有那个文件或目录
tar: 000/model.ckpt.data-00000-of-00001：无法 stat: 没有那个文件或目录
tar: 000/checkpoint：无法 stat: 没有那个文件或目录
tar: 由于前次错误，将以上次的错误状态退出

the train.log:
WARNING:tensorflow:From /home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/importlib/init.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
_bootstrap.exec(spec, module)
/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/compat.py:316: UserWarning: It seems that you are using a deepmd-kit input of version 1.x.x, which is deprecated. we have converted the input to >2.0.0 compatible, and output it to file input_v2_compat.json
warnings.warn(msg)
Traceback (most recent call last):
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/bin/dp", line 10, in
sys.exit(main())
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
train_dp(**dict_args)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 90, in train
jdata = update_sel(jdata)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 329, in update_sel
descrpt_data = update_one_sel(jdata, descrpt_data)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 305, in update_one_sel
tmp_sel = get_sel(jdata, rcut)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 268, in get_sel
, max_nbor_size = get_nbor_stat(jdata, rcut)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 253, in get_nbor_stat
train_data = get_data(jdata["training"]["training_data"], max_rcut, type_map, None)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 197, in get_data
data = DeepmdDataSystem(
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data_system.py", line 79, in init
DeepmdData(
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data.py", line 60, in init
atom_type = [type_map.index(self.type_map[ii]) for ii in self.atom_type]
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data.py", line 60, in
atom_type = [type_map.index(self.type_map[ii]) for ii in self.atom_type]
ValueError: 'Re' is not in list
WARNING:tensorflow:From /home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/importlib/init.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
_bootstrap._exec(spec, module)
Traceback (most recent call last):
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/bin/dp", line 10, in
sys.exit(main())
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 439, in main
freeze(**dict_args)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/freeze.py", line 141, in freeze
input_checkpoint = checkpoint.model_checkpoint_path
AttributeError: 'NoneType' object has no attribute 'model_checkpoint_path'

How can i solve it?

[AnguseZhang]

1. What's your running command for DP-GEN?
2. Go to work_dir you configured. Check the running script (xxx.sub) and error log.

[AnguseZhang]

1. There is no need to run DP-GEN by submitting a PBS script, since DP-GEN's main process should be maintained always. Otherwise, there will be a PBS job always running. Whenever this PBS job for DP-GEN's main process is killed, DP-GEN's main process will be killed, too. Under most circumstances you can run dpgen run param.json and machine.json on the master mode, and configure the parameters for PBS in machine.json.
2. Please check the DeePMD-kit's training log files, not DP-GEN's error log. They're in work_path which you configured in machine.json. Please go to that directory, and you will see some folders with long strings which refer to a unique UUID. Enter them and find what happens. If you fail, you can ls the work_path, and let me see.

[wangxinxiny]

Thank you, this is my work_path:

the train log is :
WARNING:tensorflow:From /home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/importlib/init.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
_bootstrap.exec(spec, module)
/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/compat.py:316: UserWarning: It seems that you are using a deepmd-kit input of version 1.x.x, which is deprecated. we have converted the input to >2.0.0 compatible, and output it to file input_v2_compat.json
warnings.warn(msg)
Traceback (most recent call last):
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/bin/dp", line 10, in
sys.exit(main())
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
train_dp(**dict_args)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 90, in train
jdata = update_sel(jdata)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 329, in update_sel
descrpt_data = update_one_sel(jdata, descrpt_data)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 305, in update_one_sel
tmp_sel = get_sel(jdata, rcut)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 268, in get_sel
, max_nbor_size = get_nbor_stat(jdata, rcut)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 253, in get_nbor_stat
train_data = get_data(jdata["training"]["training_data"], max_rcut, type_map, None)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 197, in get_data
data = DeepmdDataSystem(
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data_system.py", line 79, in init
DeepmdData(
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data.py", line 60, in init
atom_type = [type_map.index(self.type_map[ii]) for ii in self.atom_type]
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/utils/data.py", line 60, in
atom_type = [type_map.index(self.type_map[ii]) for ii in self.atom_type]
ValueError: 'Re' is not in list
WARNING:tensorflow:From /home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/importlib/init.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
_bootstrap._exec(spec, module)
Traceback (most recent call last):
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/bin/dp", line 10, in
sys.exit(main())
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 439, in main
freeze(**dict_args)
File "/home/customer/Software/anaconda/anaconda3/envs/deepmd/lib/python3.9/site-packages/deepmd/entrypoints/freeze.py", line 141, in freeze
input_checkpoint = checkpoint.model_checkpoint_path
AttributeError: 'NoneType' object has no attribute 'model_checkpoint_path'

[AnguseZhang]

“atom_type = [type_map.index(self.type_map[ii]) for ii in self.atom_type]
ValueError: 'Re' is not in list”
This is the error log for DeePMD-kit training. Then you should check the parameters in input.json in this DeePMD-kit path and locate the problem. Then you should finally change the parameters in param.json for DP-GEN, and restart the DP-GEN process.

[wangxinxiny]

Thank you for your help. I checked my file and submitted it again. It has started to run. Thanks again!

[wangxinxiny]

Hello, according to the last method, my dpgen can work normally, but when I replace the initial data with a data set lacking virial data, the same problem occurs again. I don't know how to solve it. I look forward to your reply.