WARNING: Radial neighbor list length of type 0 is not enough #602

ChengQIAN93 · 2021-05-10T07:41:17Z

ChengQIAN93
May 10, 2021

Hello everyone.

I am trying to train a dp model with se_a type. I try to change the sel number and make it larger, but it still not enough, . When I continuously increasing the sel value, dp train stops working. By the way, the data set is downloaded from published work and converted through dpdata

Does anyone know how to solve this problem? Thank you

This is part of the input file

   "descriptor" :{
       "type":             "se_a",
       "sel":              [150],
       "rcut_smth":        4.50,
       "rcut":             5.50,
       "neuron":           [25, 50, 100],
       "resnet_dt":        false,
       "axis_neuron":      16,
       "seed":             1,
       "_comment":         " that's all"

njzjz · 2021-05-10T08:03:02Z

njzjz
May 10, 2021
Maintainer

dp train stops working

Could you explain how it stops working?

2 replies

ChengQIAN93 May 10, 2021
Author

I'm not sure if there is not enough memory

This is the output information

2021-05-10 17:20:09.677139: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size:
2021-05-10 17:20:09.677149: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 70 Chunks of size 256 totalling 17.5KiB
2021-05-10 17:20:09.677157: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 512 totalling 2.0KiB
2021-05-10 17:20:09.677166: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 28 Chunks of size 1024 totalling 28.0KiB
2021-05-10 17:20:09.677173: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 1280 totalling 2.5KiB
2021-05-10 17:20:09.677181: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 4864 totalling 19.0KiB
2021-05-10 17:20:09.677189: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 10240 totalling 50.0KiB
2021-05-10 17:20:09.677211: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 40192 totalling 157.0KiB
2021-05-10 17:20:09.677219: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 8 Chunks of size 115200 totalling 900.0KiB
2021-05-10 17:20:09.677241: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 225792 totalling 220.5KiB
2021-05-10 17:20:09.677248: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 676864 totalling 1.29MiB
2021-05-10 17:20:09.677255: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 1536000 totalling 5.86MiB
2021-05-10 17:20:09.677263: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 135360000 totalling 129.09MiB
2021-05-10 17:20:09.677270: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 846000128 totalling 806.81MiB
2021-05-10 17:20:09.677278: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 1692000000 totalling 3.15GiB
2021-05-10 17:20:09.677285: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3384000000 totalling 3.15GiB
2021-05-10 17:20:09.677292: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 7.22GiB
2021-05-10 17:20:09.677299: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 11218736896 memory_limit_: 11218736896 available bytes: 0 curr_region_allocation_bytes_: 22437473792
2021-05-10 17:20:09.677310: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats:
Limit: 11218736896
InUse: 7758287872
MaxInUse: 8164367872
NumAllocs: 121344
MaxAllocSize: 3384000000

2021-05-10 17:20:09.677329: W tensorflow/core/common_runtime/bfc_allocator.cc:439] __________*******************************************************************___________________
2021-05-10 17:20:09.678044: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at concat_op.cc:161 : Resource exhausted: OOM when allocating tensor with shape[4230000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[4230000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node filter_type_0/concat_1}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[4230000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node filter_type_0/concat_1}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[l2_virial_test/_67]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cqian/anaconda3/envs/deepmd/bin/dp", line 10, in
sys.exit(main())
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/deepmd/main.py", line 73, in main
train(args)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/deepmd/train.py", line 87, in train
_do_work(jdata, run_opt)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/deepmd/train.py", line 143, in _do_work
model.train (data)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/deepmd/Trainer.py", line 431, in train
self.test_on_the_fly(fp, data, feed_dict_batch)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/deepmd/Trainer.py", line 495, in test_on_the_fly
feed_dict_batch)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/deepmd/Loss.py", line 151, in print_on_training
feed_dict=feed_dict_test)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/cqian/anaconda3/envs/deepmd/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[4230000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node filter_type_0/concat_1 (defined at /lib/python3.6/site-packages/deepmd/DescrptSeA.py:386) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[4230000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node filter_type_0/concat_1 (defined at /lib/python3.6/site-packages/deepmd/DescrptSeA.py:386) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[l2_virial_test/_67]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Original stack trace for 'filter_type_0/concat_1':
File "/bin/dp", line 10, in
sys.exit(main())
File "/lib/python3.6/site-packages/deepmd/main.py", line 73, in main
train(args)
File "/lib/python3.6/site-packages/deepmd/train.py", line 87, in train
_do_work(jdata, run_opt)
File "/lib/python3.6/site-packages/deepmd/train.py", line 140, in _do_work
model.build (data, stop_batch)
File "/lib/python3.6/site-packages/deepmd/Trainer.py", line 236, in build
self._build_network(data)
File "/lib/python3.6/site-packages/deepmd/Trainer.py", line 270, in _build_network
reuse = False)
File "/lib/python3.6/site-packages/deepmd/Model.py", line 172, in build
reuse = reuse)
File "/lib/python3.6/site-packages/deepmd/DescrptSeA.py", line 211, in build
self.dout, self.qmat = self._pass_filter(self.descrpt_reshape, natoms, suffix = suffix, reuse = reuse, trainable = self.trainable)
File "/lib/python3.6/site-packages/deepmd/DescrptSeA.py", line 257, in _pass_filter
layer, qmat = self.filter(tf.cast(inputs_i, self.filter_precision), type_i, name='filter_type'+str(type_i)+suffix, natoms=natoms, reuse=reuse, seed = self.seed, trainable = trainable, activation_fn = self.filter_activation_fn)
File "/lib/python3.6/site-packages/deepmd/DescrptSeA.py", line 386, in _filter
xyz_scatter = tf.concat([xyz_scatter,xyz_scatter], 1) + hidden
File "/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1606, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1189, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3327, in _create_op_internal
op_def=op_def)
File "/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1791, in init
self._traceback = tf_stack.extract_stack()

njzjz May 10, 2021
Maintainer

Yes, OOM=out of memory. If you’ve already set batch size to 1, you may need devices with larger memory.

AnguseZhang · 2021-05-10T08:26:50Z

AnguseZhang
May 10, 2021
Collaborator

Two problems should be separated.

Sel not enough: DeePMD-kit code finds there is overflow of neighbors. If you think it's wrong, you could write a script to check the local environments and report bugs. If not, you should change the settings, i.e. increasing "sel_a".
Figure out why "continuously increasing the sel value, dp train stops working".

1 reply

ChengQIAN93 May 20, 2021
Author

At the beginning, I didn't realize that the published data had some unreasonable structures. After I deleted these structures, there was no more warning.

Thanks for your reply.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WARNING: Radial neighbor list length of type 0 is not enough #602

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

WARNING: Radial neighbor list length of type 0 is not enough #602

Uh oh!

ChengQIAN93 May 10, 2021

This is part of the input file

Replies: 2 comments · 3 replies

Uh oh!

Uh oh!

njzjz May 10, 2021 Maintainer

Uh oh!

ChengQIAN93 May 10, 2021 Author

This is the output information

Uh oh!

njzjz May 10, 2021 Maintainer

Uh oh!

AnguseZhang May 10, 2021 Collaborator

Uh oh!

ChengQIAN93 May 20, 2021 Author

ChengQIAN93
May 10, 2021

Replies: 2 comments 3 replies

njzjz
May 10, 2021
Maintainer

ChengQIAN93 May 10, 2021
Author

njzjz May 10, 2021
Maintainer

AnguseZhang
May 10, 2021
Collaborator

ChengQIAN93 May 20, 2021
Author