gpu related error
#674
Replies: 1 comment 1 reply
-
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks, that may be the problem, I have solved this by changing "numb_test": from "10%" to 10. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I added new dataset to the input.json, the gpu does not work, but cpu trainning works in a very slow way. Is that my gpu ( Tesla V100-SXM2-16GB ) problem?
The below is my error log:
WARNING:tensorflow:From /home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:65: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
2021-05-29 10:07:53.940950: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2021-05-29 10:07:53.992841: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400175000 Hz
2021-05-29 10:07:53.993601: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556dc4f52f50 executing computations on platform Host. Devices:
2021-05-29 10:07:53.993997: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2021-05-29 10:07:54.004900: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-05-29 10:07:54.053892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:85:00.0
2021-05-29 10:07:54.067576: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-05-29 10:07:54.222701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-05-29 10:07:54.348545: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-05-29 10:07:54.381819: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-05-29 10:07:54.594219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-05-29 10:07:54.729971: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-05-29 10:07:55.061861: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-05-29 10:07:55.064595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-05-29 10:07:55.064664: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-05-29 10:07:55.171141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-29 10:07:55.171212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-05-29 10:07:55.171236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-05-29 10:07:55.175568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15051 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0)
2021-05-29 10:07:55.182390: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556dc5c02630 executing computations on platform CUDA. Devices:
2021-05-29 10:07:55.182427: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 11
OMP: Info #214: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #156: KMP_AFFINITY: 1 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #285: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #285: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #285: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "socket".
OMP: Info #285: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "socket".
OMP: Info #285: KMP_AFFINITY: topology layer "core" is equivalent to "socket".
OMP: Info #285: KMP_AFFINITY: topology layer "thread" is equivalent to "socket".
OMP: Info #191: KMP_AFFINITY: 1 socket x 1 core/socket x 1 thread/core (1 total cores)
OMP: Info #216: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to socket 1
OMP: Info #252: KMP_AFFINITY: pid 19678 tid 19755 thread 0 bound to OS proc set 11
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: ghost idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of lower bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of lower bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
warning: loc idx out of upper bound (ignored if warned for more than 10 times)
OMP: Info #252: KMP_AFFINITY: pid 19678 tid 19678 thread 1 bound to OS proc set 11
2021-05-29 10:08:03.447255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:85:00.0
2021-05-29 10:08:03.447329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-05-29 10:08:03.447360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-05-29 10:08:03.447412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-05-29 10:08:03.447440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-05-29 10:08:03.447465: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-05-29 10:08:03.447493: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-05-29 10:08:03.447522: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-05-29 10:08:03.449574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-05-29 10:08:03.449630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-29 10:08:03.449649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-05-29 10:08:03.449687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-05-29 10:08:03.451828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15051 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0)
2021-05-29 10:08:11.561697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-05-29 10:08:22.295815: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.96GiB (rounded to 2100800000). Current allocation summary follows.
2021-05-29 10:08:22.295975: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256): Total Chunks: 38, Chunks in use: 36. 9.5KiB allocated for chunks. 9.0KiB in use in bin. 244B client-requested in use in bin.
2021-05-29 10:08:22.296039: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512): Total Chunks: 8, Chunks in use: 8. 4.0KiB allocated for chunks. 4.0KiB in use in bin. 3.1KiB client-requested in use in bin.
2021-05-29 10:08:22.296085: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024): Total Chunks: 33, Chunks in use: 33. 36.2KiB allocated for chunks. 36.2KiB in use in bin. 32.9KiB client-requested in use in bin.
2021-05-29 10:08:22.296123: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296160: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296200: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192): Total Chunks: 5, Chunks in use: 5. 77.2KiB allocated for chunks. 77.2KiB in use in bin. 76.7KiB client-requested in use in bin.
2021-05-29 10:08:22.296237: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296279: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768): Total Chunks: 4, Chunks in use: 4. 157.0KiB allocated for chunks. 157.0KiB in use in bin. 156.2KiB client-requested in use in bin.
2021-05-29 10:08:22.296319: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536): Total Chunks: 10, Chunks in use: 10. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2021-05-29 10:08:22.296361: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072): Total Chunks: 5, Chunks in use: 4. 805.8KiB allocated for chunks. 625.0KiB in use in bin. 625.0KiB client-requested in use in bin.
2021-05-29 10:08:22.296427: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296467: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288): Total Chunks: 4, Chunks in use: 4. 2.93MiB allocated for chunks. 2.93MiB in use in bin. 2.93MiB client-requested in use in bin.
2021-05-29 10:08:22.296504: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296540: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296576: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296611: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296674: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216): Total Chunks: 1, Chunks in use: 0. 20.03MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296716: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-05-29 10:08:22.296758: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864): Total Chunks: 3, Chunks in use: 1. 256.45MiB allocated for chunks. 80.14MiB in use in bin. 80.14MiB client-requested in use in bin.
2021-05-29 10:08:22.296800: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728): Total Chunks: 1, Chunks in use: 1. 240.42MiB allocated for chunks. 240.42MiB in use in bin. 240.42MiB client-requested in use in bin.
2021-05-29 10:08:22.296841: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456): Total Chunks: 6, Chunks in use: 5. 14.19GiB allocated for chunks. 12.72GiB in use in bin. 12.72GiB client-requested in use in bin.
2021-05-29 10:08:22.296880: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 1.96GiB was 256.00MiB, Chunk State:
2021-05-29 10:08:22.296941: I tensorflow/core/common_runtime/bfc_allocator.cc:891] Size: 1.47GiB | Requested Size: 32.06MiB | in_use: 0 | bin_num: 20, prev: Size: 80.14MiB | Requested Size: 80.14MiB | in_use: 1 | bin_num: -1
2021-05-29 10:08:22.296980: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 15782644736
2021-05-29 10:08:22.297018: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114000000 next 1 of size 256
2021-05-29 10:08:22.297052: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114000100 next 2 of size 1792
2021-05-29 10:08:22.297086: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114000800 next 3 of size 1024
2021-05-29 10:08:22.297120: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114000c00 next 4 of size 256
2021-05-29 10:08:22.297152: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114000d00 next 5 of size 256
2021-05-29 10:08:22.297186: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114000e00 next 6 of size 16128
2021-05-29 10:08:22.297219: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114004d00 next 7 of size 16128
2021-05-29 10:08:22.297252: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114008c00 next 8 of size 1024
2021-05-29 10:08:22.297285: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009000 next 9 of size 256
2021-05-29 10:08:22.297318: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009100 next 10 of size 256
2021-05-29 10:08:22.297351: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009200 next 11 of size 256
2021-05-29 10:08:22.297396: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009300 next 12 of size 1024
2021-05-29 10:08:22.297432: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009700 next 13 of size 256
2021-05-29 10:08:22.297465: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009800 next 14 of size 256
2021-05-29 10:08:22.297497: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009900 next 15 of size 256
2021-05-29 10:08:22.297530: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009a00 next 16 of size 256
2021-05-29 10:08:22.297563: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009b00 next 17 of size 256
2021-05-29 10:08:22.297596: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009c00 next 18 of size 256
2021-05-29 10:08:22.297629: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009d00 next 19 of size 256
2021-05-29 10:08:22.297678: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009e00 next 20 of size 256
2021-05-29 10:08:22.297716: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114009f00 next 21 of size 1024
2021-05-29 10:08:22.297749: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400a300 next 22 of size 1024
2021-05-29 10:08:22.297782: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400a700 next 23 of size 256
2021-05-29 10:08:22.297815: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400a800 next 24 of size 256
2021-05-29 10:08:22.297847: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400a900 next 25 of size 1024
2021-05-29 10:08:22.297880: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400ad00 next 26 of size 512
2021-05-29 10:08:22.297913: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400af00 next 27 of size 512
2021-05-29 10:08:22.297946: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400b100 next 28 of size 1024
2021-05-29 10:08:22.297979: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111400b500 next 29 of size 160000
2021-05-29 10:08:22.298012: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114032600 next 30 of size 768000
2021-05-29 10:08:22.298046: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11140ede00 next 31 of size 115200
2021-05-29 10:08:22.298079: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111410a000 next 32 of size 115200
2021-05-29 10:08:22.298112: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114126200 next 33 of size 40192
2021-05-29 10:08:22.298145: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111412ff00 next 34 of size 1280
2021-05-29 10:08:22.298178: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114130400 next 35 of size 512
2021-05-29 10:08:22.298211: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114130600 next 36 of size 1024
2021-05-29 10:08:22.298244: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114130a00 next 37 of size 512
2021-05-29 10:08:22.298276: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114130c00 next 38 of size 256
2021-05-29 10:08:22.298309: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114130d00 next 39 of size 1024
2021-05-29 10:08:22.298342: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114131100 next 40 of size 1024
2021-05-29 10:08:22.298384: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114131500 next 41 of size 1024
2021-05-29 10:08:22.298420: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114131900 next 42 of size 1024
2021-05-29 10:08:22.298453: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114131d00 next 43 of size 1792
2021-05-29 10:08:22.298485: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114132400 next 44 of size 256
2021-05-29 10:08:22.298518: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114132500 next 45 of size 512
2021-05-29 10:08:22.298551: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114132700 next 46 of size 160000
2021-05-29 10:08:22.298584: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114159800 next 47 of size 1024
2021-05-29 10:08:22.298616: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114159c00 next 48 of size 1024
2021-05-29 10:08:22.298649: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111415a000 next 49 of size 256
2021-05-29 10:08:22.298681: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111415a100 next 50 of size 512
2021-05-29 10:08:22.298714: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111415a300 next 51 of size 115200
2021-05-29 10:08:22.298762: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114176500 next 52 of size 1024
2021-05-29 10:08:22.298799: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114176900 next 53 of size 1024
2021-05-29 10:08:22.298833: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114176d00 next 54 of size 512
2021-05-29 10:08:22.298866: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114176f00 next 55 of size 1024
2021-05-29 10:08:22.298898: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114177300 next 56 of size 115200
2021-05-29 10:08:22.298931: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114193500 next 57 of size 512
2021-05-29 10:08:22.298964: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114193700 next 58 of size 16128
2021-05-29 10:08:22.298997: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114197600 next 59 of size 16128
2021-05-29 10:08:22.299029: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111419b500 next 60 of size 256
2021-05-29 10:08:22.299062: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111419b600 next 61 of size 256
2021-05-29 10:08:22.299095: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111419b700 next 62 of size 40192
2021-05-29 10:08:22.299127: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11141a5400 next 63 of size 1024
2021-05-29 10:08:22.299160: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11141a5800 next 64 of size 1024
2021-05-29 10:08:22.299193: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11141a5c00 next 65 of size 1024
2021-05-29 10:08:22.299226: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11141a6000 next 66 of size 115200
2021-05-29 10:08:22.299258: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11141c2200 next 67 of size 115200
2021-05-29 10:08:22.299291: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11141de400 next 68 of size 160000
2021-05-29 10:08:22.299324: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114205500 next 69 of size 768000
2021-05-29 10:08:22.299357: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142c0d00 next 70 of size 1024
2021-05-29 10:08:22.299402: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142c1100 next 71 of size 1024
2021-05-29 10:08:22.299436: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142c1500 next 72 of size 40192
2021-05-29 10:08:22.299469: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142cb200 next 73 of size 40192
2021-05-29 10:08:22.299502: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142d4f00 next 74 of size 1792
2021-05-29 10:08:22.299535: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142d5600 next 75 of size 1024
2021-05-29 10:08:22.299567: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142d5a00 next 76 of size 1024
2021-05-29 10:08:22.299600: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142d5e00 next 77 of size 1024
2021-05-29 10:08:22.299633: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142d6200 next 78 of size 115200
2021-05-29 10:08:22.299665: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142f2400 next 79 of size 1024
2021-05-29 10:08:22.299698: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11142f2800 next 80 of size 160000
2021-05-29 10:08:22.299731: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1114319900 next 81 of size 1792
2021-05-29 10:08:22.299763: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111431a000 next 82 of size 768000
2021-05-29 10:08:22.299796: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11143d5800 next 83 of size 1024
2021-05-29 10:08:22.299843: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11143d5c00 next 84 of size 256
2021-05-29 10:08:22.299881: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11143d5d00 next 85 of size 115200
2021-05-29 10:08:22.299914: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11143f1f00 next 86 of size 1024
2021-05-29 10:08:22.299947: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11143f2300 next 87 of size 768000
2021-05-29 10:08:22.299979: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144adb00 next 88 of size 256
2021-05-29 10:08:22.300012: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144adc00 next 89 of size 256
2021-05-29 10:08:22.300045: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144add00 next 90 of size 256
2021-05-29 10:08:22.300077: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ade00 next 91 of size 256
2021-05-29 10:08:22.300110: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144adf00 next 92 of size 256
2021-05-29 10:08:22.300143: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ae000 next 93 of size 256
2021-05-29 10:08:22.300175: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ae100 next 94 of size 256
2021-05-29 10:08:22.300208: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ae200 next 95 of size 256
2021-05-29 10:08:22.300240: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ae300 next 96 of size 256
2021-05-29 10:08:22.300273: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ae400 next 97 of size 256
2021-05-29 10:08:22.300305: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ae500 next 98 of size 256
2021-05-29 10:08:22.300338: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x2b11144ae600 next 99 of size 256
2021-05-29 10:08:22.300380: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144ae700 next 100 of size 126208
2021-05-29 10:08:22.300417: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144cd400 next 101 of size 256
2021-05-29 10:08:22.300450: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144cd500 next 102 of size 256
2021-05-29 10:08:22.300482: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x2b11144cd600 next 107 of size 185088
2021-05-29 10:08:22.300516: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144fa900 next 108 of size 14592
2021-05-29 10:08:22.300548: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x2b11144fe200 next 109 of size 256
2021-05-29 10:08:22.300581: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11144fe300 next 110 of size 126208
2021-05-29 10:08:22.300614: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b111451d000 next 111 of size 256
2021-05-29 10:08:22.300646: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x2b111451d100 next 105 of size 84032000
2021-05-29 10:08:22.300679: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1119540b00 next 104 of size 252096000
2021-05-29 10:08:22.300712: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x2b11285ab900 next 112 of size 21008128
2021-05-29 10:08:22.300745: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11299b4800 next 113 of size 1050400000
2021-05-29 10:08:22.300778: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b1168371d00 next 114 of size 2100800000
2021-05-29 10:08:22.300811: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b11e56ec700 next 115 of size 4201600000
2021-05-29 10:08:22.300844: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b12dfde1b00 next 116 of size 2100800000
2021-05-29 10:08:22.300877: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b135d15c500 next 117 of size 4201600000
2021-05-29 10:08:22.300925: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x2b1457851900 next 122 of size 100838400
2021-05-29 10:08:22.300963: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x2b145d87c500 next 118 of size 84032000
2021-05-29 10:08:22.300997: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x2b146289ff00 next 18446744073709551615 of size 1580076288
2021-05-29 10:08:22.301029: I tensorflow/core/common_runtime/bfc_allocator.cc:914] Summary of in-use Chunks by size:
2021-05-29 10:08:22.301067: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 36 Chunks of size 256 totalling 9.0KiB
2021-05-29 10:08:22.301104: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 8 Chunks of size 512 totalling 4.0KiB
2021-05-29 10:08:22.301141: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 28 Chunks of size 1024 totalling 28.0KiB
2021-05-29 10:08:22.301178: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1280 totalling 1.2KiB
2021-05-29 10:08:22.301213: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 1792 totalling 7.0KiB
2021-05-29 10:08:22.301250: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 14592 totalling 14.2KiB
2021-05-29 10:08:22.301287: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 16128 totalling 63.0KiB
2021-05-29 10:08:22.301324: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 40192 totalling 157.0KiB
2021-05-29 10:08:22.301361: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 8 Chunks of size 115200 totalling 900.0KiB
2021-05-29 10:08:22.301410: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 126208 totalling 246.5KiB
2021-05-29 10:08:22.301448: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 160000 totalling 625.0KiB
2021-05-29 10:08:22.301484: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 768000 totalling 2.93MiB
2021-05-29 10:08:22.301521: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 84032000 totalling 80.14MiB
2021-05-29 10:08:22.301558: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 252096000 totalling 240.42MiB
2021-05-29 10:08:22.301595: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1050400000 totalling 1001.74MiB
2021-05-29 10:08:22.301632: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 2100800000 totalling 3.91GiB
2021-05-29 10:08:22.301668: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 4201600000 totalling 7.83GiB
2021-05-29 10:08:22.301704: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 13.04GiB
2021-05-29 10:08:22.301738: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 15782644736 memory_limit_: 15782644941 available bytes: 205 curr_region_allocation_bytes_: 31565289984
2021-05-29 10:08:22.301778: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats:
Limit: 15782644941
InUse: 13996504320
MaxInUse: 13996548352
NumAllocs: 168
MaxAllocSize: 4201600000
2021-05-29 10:08:22.301827: W tensorflow/core/common_runtime/bfc_allocator.cc:424] ******************************************************************************************__________
2021-05-29 10:08:22.301893: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at slice_op.cc:154 : Resource exhausted: OOM when allocating tensor with shape[2626000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[2626000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node gradients/filter_type_0/concat_1_grad/Slice_1}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[2626000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node gradients/filter_type_0/concat_1_grad/Slice_1}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/bin/dp", line 10, in
sys.exit(main())
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/deepmd/main.py", line 73, in main
train(args)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/deepmd/train.py", line 87, in train
_do_work(jdata, run_opt)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/deepmd/train.py", line 143, in _do_work
model.train (data)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/deepmd/Trainer.py", line 420, in train
self.test_on_the_fly(fp, data, feed_dict_batch)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/deepmd/Trainer.py", line 495, in test_on_the_fly
feed_dict_batch)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/deepmd/Loss.py", line 151, in print_on_training
feed_dict=feed_dict_test)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/XXX/scratch/Anaconda3/envs/deepgpu/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[2626000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node gradients/filter_type_0/concat_1_grad/Slice_1 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[2626000,100] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node gradients/filter_type_0/concat_1_grad/Slice_1 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
Original stack trace for 'gradients/filter_type_0/concat_1_grad/Slice_1':
File "/bin/dp", line 10, in
sys.exit(main())
File "/lib/python3.7/site-packages/deepmd/main.py", line 73, in main
train(args)
File "/lib/python3.7/site-packages/deepmd/train.py", line 87, in train
_do_work(jdata, run_opt)
File "/lib/python3.7/site-packages/deepmd/train.py", line 140, in _do_work
model.build (data, stop_batch)
File "/lib/python3.7/site-packages/deepmd/Trainer.py", line 236, in build
self._build_network(data)
File "/lib/python3.7/site-packages/deepmd/Trainer.py", line 270, in _build_network
reuse = False)
File "/lib/python3.7/site-packages/deepmd/Model.py", line 222, in build
= self.descrpt.prod_force_virial (atom_ener, natoms)
File "/lib/python3.7/site-packages/deepmd/DescrptSeA.py", line 221, in prod_force_virial
[net_deriv] = tf.gradients (atom_ener, self.descrpt_reshape)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
unconnected_gradients)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 350, in _MaybeCompile
return grad_fn() # Exit early
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in
lambda: grad_fn(op, *out_grads))
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/array_grad.py", line 227, in _ConcatGradV2
op, grad, start_value_index=0, end_value_index=-1, dim_index=-1)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/array_grad.py", line 155, in _ConcatGradHelper
out_grads.append(array_ops.slice(grad, begin, size))
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 866, in slice
return gen_array_ops.slice(input, begin, size, name=name)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 9224, in _slice
"Slice", input=input, begin=begin, size=size, name=name)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 793, in _apply_op_helper
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3360, in create_op
attrs, op_def, compute_device)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3429, in _create_op_internal
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1751, in init
self._traceback = tf_stack.extract_stack()
...which was originally created as op 'filter_type_0/concat_1', defined at:
File "/bin/dp", line 10, in
sys.exit(main())
[elided 4 identical lines from previous traceback]
File "/lib/python3.7/site-packages/deepmd/Trainer.py", line 270, in _build_network
reuse = False)
File "/lib/python3.7/site-packages/deepmd/Model.py", line 172, in build
reuse = reuse)
File "/lib/python3.7/site-packages/deepmd/DescrptSeA.py", line 211, in build
self.dout, self.qmat = self._pass_filter(self.descrpt_reshape, natoms, suffix = suffix, reuse = reuse, trainable = self.trainable)
File "/lib/python3.7/site-packages/deepmd/DescrptSeA.py", line 257, in _pass_filter
layer, qmat = self.filter(tf.cast(inputs_i, self.filter_precision), type_i, name='filter_type'+str(type_i)+suffix, natoms=natoms, reuse=reuse, seed = self.seed, trainable = trainable, activation_fn = self.filter_activation_fn)
File "/lib/python3.7/site-packages/deepmd/DescrptSeA.py", line 386, in _filter
xyz_scatter = tf.concat([xyz_scatter,xyz_scatter], 1) + hidden
File "/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 1431, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 1257, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 793, in _apply_op_helper
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3360, in create_op
attrs, op_def, compute_device)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3429, in _create_op_internal
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1751, in init
self._traceback = tf_stack.extract_stack()
Beta Was this translation helpful? Give feedback.
All reactions