Problem with running on GPU

[dgg95223]

Hi,
I am trying to run DeePMD-kit on a PC with GTX 1080ti but somehow it cannot evoke GPU to run the calculation. It gives out the following information:

WARNING:tensorflow:From /home/jingheng/venv/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, TF_INTRA_OP_PARALLELISM_THREADS, and TF_INTER_OP_PARALLELISM_THREADS.
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
WARNING:deepmd.train.run_options:Switch to serial execution due to lack of horovod module.
DEEPMD INFO    Calculate neighbor statistics... (add --skip-neighbor-stat to skip this step)
/home/jingheng/venv/lib/python3.9/site-packages/deepmd/utils/data_system.py:158: UserWarning: system ./raw/ required batch size is larger than the size of the dataset raw/set.000 (6 > 1)
  warnings.warn("system %s required batch size is larger than the size of the dataset %s (%d > %d)" % \
OMP: Info #155: KMP_AFFINITY: Initial OS proc set respected: 0-11
OMP: Info #216: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #157: KMP_AFFINITY: 12 available OS procs
OMP: Info #158: KMP_AFFINITY: Uniform topology
OMP: Info #287: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "core".
OMP: Info #287: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #192: KMP_AFFINITY: 1 socket x 6 cores/socket x 2 threads/core (6 total cores)
OMP: Info #218: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #172: KMP_AFFINITY: OS proc 0 maps to socket 0 core 0 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 1 maps to socket 0 core 0 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 2 maps to socket 0 core 1 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 3 maps to socket 0 core 1 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 4 maps to socket 0 core 2 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 5 maps to socket 0 core 2 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 6 maps to socket 0 core 3 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 7 maps to socket 0 core 3 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 8 maps to socket 0 core 4 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 9 maps to socket 0 core 4 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 10 maps to socket 0 core 5 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 11 maps to socket 0 core 5 thread 1
OMP: Info #254: KMP_AFFINITY: pid 422 tid 486 thread 0 bound to OS proc set 0
OMP: Info #254: KMP_AFFINITY: pid 422 tid 488 thread 1 bound to OS proc set 2
OMP: Info #254: KMP_AFFINITY: pid 422 tid 489 thread 2 bound to OS proc set 4
OMP: Info #254: KMP_AFFINITY: pid 422 tid 490 thread 3 bound to OS proc set 6
OMP: Info #254: KMP_AFFINITY: pid 422 tid 491 thread 4 bound to OS proc set 8
OMP: Info #254: KMP_AFFINITY: pid 422 tid 492 thread 5 bound to OS proc set 10
OMP: Info #254: KMP_AFFINITY: pid 422 tid 493 thread 6 bound to OS proc set 1
OMP: Info #254: KMP_AFFINITY: pid 422 tid 494 thread 7 bound to OS proc set 3
OMP: Info #254: KMP_AFFINITY: pid 422 tid 495 thread 8 bound to OS proc set 5
OMP: Info #254: KMP_AFFINITY: pid 422 tid 496 thread 9 bound to OS proc set 7
OMP: Info #254: KMP_AFFINITY: pid 422 tid 497 thread 10 bound to OS proc set 9
OMP: Info #254: KMP_AFFINITY: pid 422 tid 498 thread 11 bound to OS proc set 11
DEEPMD INFO    training data with min nbor dist: 1.0065080572951215
DEEPMD INFO    training data with max nbor size: [2, 1]
DEEPMD INFO     _____               _____   __  __  _____           _     _  _
DEEPMD INFO    |  __ \             |  __ \ |  \/  ||  __ \         | |   (_)| |
DEEPMD INFO    | |  | |  ___   ___ | |__) || \  / || |  | | ______ | | __ _ | |_
DEEPMD INFO    | |  | | / _ \ / _ \|  ___/ | |\/| || |  | ||______|| |/ /| || __|
DEEPMD INFO    | |__| ||  __/|  __/| |     | |  | || |__| |        |   < | || |_
DEEPMD INFO    |_____/  \___| \___||_|     |_|  |_||_____/         |_|\_\|_| \__|
DEEPMD INFO    Please read and cite:
DEEPMD INFO    Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018)
DEEPMD INFO    installed to:         /home/jingheng/deepmd-kit/_skbuild/linux-x86_64-3.9/cmake-install
DEEPMD INFO    source :              v2.1.0
DEEPMD INFO    source brach:         master
DEEPMD INFO    source commit:        3e54fea
DEEPMD INFO    source commit at:     2022-03-07 11:41:41 +0800
DEEPMD INFO    build float prec:     double
DEEPMD INFO    build with tf inc:    /home/jingheng/venv/lib/python3.9/site-packages/tensorflow/include
DEEPMD INFO    build with tf lib:
DEEPMD INFO    ---Summary of the training---------------------------------------
DEEPMD INFO    running on:           Deng-PC
DEEPMD INFO    computing device:     gpu:0
DEEPMD INFO    CUDA_VISIBLE_DEVICES: unset
DEEPMD INFO    Count of visible GPU: 1
DEEPMD INFO    num_intra_threads:    0
DEEPMD INFO    num_inter_threads:    0
DEEPMD INFO    -----------------------------------------------------------------
DEEPMD INFO    ---Summary of DataSystem: training     -----------------------------------------------
DEEPMD INFO    found 1 system(s):
DEEPMD INFO                                        system  natoms  bch_sz   n_bch   prob  pbc
DEEPMD INFO                                        ./raw/       6       6       1  1.000    F
DEEPMD INFO    --------------------------------------------------------------------------------------
DEEPMD INFO    training without frame parameter
DEEPMD INFO    data stating... (this step may take long time)
OMP: Info #254: KMP_AFFINITY: pid 422 tid 422 thread 12 bound to OS proc set 0
DEEPMD INFO    built lr
DEEPMD INFO    built network
DEEPMD INFO    built training
DEEPMD WARNING To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, TF_INTRA_OP_PARALLELISM_THREADS, and TF_INTER_OP_PARALLELISM_THREADS.
DEEPMD INFO    initialize model from scratch
DEEPMD INFO    start training at lr 1.00e-03 (== 1.00e-03), decay_step 5000, decay_rate 0.598777, final lr will be 3.51e-08
DEEPMD INFO    batch     100 training time 3.98 s, testing time 0.01 s

I have set 'CUDA_VISIBLE_DEVICES' to 0 but it doesn't work at all. Please let me know how to fix it, thank you very much.

Update: it seems that the job is assigned to 12 CPU threads and 12 GPU threads, I am not sure how that happens. If I use pure CPU version of DeePMD-kit, it will be more efficient, it costs around 1s per 100 steps. Is there any way to improve the efficiency?

[njzjz]

What is your issue? I can't see anything wrong in the log you provided.

[dgg95223]

During the training process, the cpu usage is only about 30% and the gpu usage is about 5% with "DP_VARIANT" setting to "cuda". While "DP_VARIANT" is set to"cpu", the cpu usage can be over 90%. As a result, the gpu mode become much slower than the cpu mode. It is supposed to be much faster with gpu acceleration, right? I don't know that why it is.

[njzjz]

Usually yes, but I cannot expect better performance from 1080ti - What's your CPU?

[dgg95223]

It's Intel i7-8700k, 6 cores and 12 threads.

[likefallwind]

I think you should change some parameters.
See this: #1039

[dgg95223]

Do you have any advice on setting the parameters for a system with very limited amount of frames? If you can give me some general ideas, it will be really helpful. Thank you very much.