CPU-GPU collaboration problems during dp train

[DingChangjie]

Dear all,
I'm trying to train two deep potentials simutaneously (let's say task A and task B) on a 10-cores machine equipped with 4 GPU cards. The deepmd-kit version is 2.1.5. Before training the following environment variables are set:
task A
export OMP_NUM_THREADS=1
export TF_INTRA_OP_PARALLELISM_THREADS=1
export TF_INTER_OP_PARALLELISM_THREADS=1
export CUDA_VISIBLE_DEVICES=3
task B
export OMP_NUM_THREADS=1
export TF_INTRA_OP_PARALLELISM_THREADS=1
export TF_INTER_OP_PARALLELISM_THREADS=1
export CUDA_VISIBLE_DEVICES=2
And when the tasks begin, it can be seen through log files that the two processes (pid) are bounded to different OS proc sets:
task A is submitted first. From the bold text we see task A is bounded to OS proc set 0

DEEPMD INFO Calculate neighbor statistics... (add --skip-neighbor-stat to skip this step)
OMP: Info #155: KMP_AFFINITY: Initial OS proc set respected: 0-19
OMP: Info #216: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #157: KMP_AFFINITY: 20 available OS procs
OMP: Info #158: KMP_AFFINITY: Uniform topology
OMP: Info #287: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "core".
OMP: Info #287: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #192: KMP_AFFINITY: 1 socket x 10 cores/socket x 2 threads/core (10 total cores)
OMP: Info #218: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #172: KMP_AFFINITY: OS proc 0 maps to socket 0 core 0 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 10 maps to socket 0 core 0 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 1 maps to socket 0 core 2 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 11 maps to socket 0 core 2 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 2 maps to socket 0 core 4 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 12 maps to socket 0 core 4 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 3 maps to socket 0 core 5 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 13 maps to socket 0 core 5 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 4 maps to socket 0 core 8 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 14 maps to socket 0 core 8 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 5 maps to socket 0 core 9 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 15 maps to socket 0 core 9 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 6 maps to socket 0 core 10 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 16 maps to socket 0 core 10 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 7 maps to socket 0 core 11 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 17 maps to socket 0 core 11 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 8 maps to socket 0 core 12 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 18 maps to socket 0 core 12 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 9 maps to socket 0 core 13 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 19 maps to socket 0 core 13 thread 1
OMP: Info #254: KMP_AFFINITY: pid 424222 tid 424293 thread 0 bound to OS proc set 0
DEEPMD INFO training data with min nbor dist: 0.8864656515622014
DEEPMD INFO training data with max nbor size: [134 186]
...
INFO:tensorflow:Restoring parameters from /home/gpu/ZrC/Refinement/dp_4_hyb/model.ckpt
DEEPMD INFO Restoring parameters from /home/gpu/ZrC/Refinement/dp_4_hyb/model.ckpt
DEEPMD INFO start training at lr 8.29e-06 (== 8.29e-06), decay_step 5000, decay_rate 0.994605, final lr will be 2.00e-08

task B is submitted next. From the bold text we see task B is first bounded to OS proc set 0, then appears to be automatically changed to proc set 1

DEEPMD INFO Calculate neighbor statistics... (add --skip-neighbor-stat to skip this step)
OMP: Info #155: KMP_AFFINITY: Initial OS proc set respected: 0-19
OMP: Info #216: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #157: KMP_AFFINITY: 20 available OS procs
OMP: Info #158: KMP_AFFINITY: Uniform topology
OMP: Info #287: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "core".
OMP: Info #287: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #192: KMP_AFFINITY: 1 socket x 10 cores/socket x 2 threads/core (10 total cores)
OMP: Info #218: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #172: KMP_AFFINITY: OS proc 0 maps to socket 0 core 0 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 10 maps to socket 0 core 0 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 1 maps to socket 0 core 2 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 11 maps to socket 0 core 2 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 2 maps to socket 0 core 4 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 12 maps to socket 0 core 4 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 3 maps to socket 0 core 5 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 13 maps to socket 0 core 5 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 4 maps to socket 0 core 8 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 14 maps to socket 0 core 8 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 5 maps to socket 0 core 9 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 15 maps to socket 0 core 9 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 6 maps to socket 0 core 10 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 16 maps to socket 0 core 10 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 7 maps to socket 0 core 11 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 17 maps to socket 0 core 11 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 8 maps to socket 0 core 12 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 18 maps to socket 0 core 12 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 9 maps to socket 0 core 13 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 19 maps to socket 0 core 13 thread 1
OMP: Info #254: KMP_AFFINITY: pid 424970 tid 425037 thread 0 bound to OS proc set 0
DEEPMD INFO training data with min nbor dist: 0.8864656515622014
DEEPMD INFO training data with max nbor size: [92 92]
...
OMP: Info #254: KMP_AFFINITY: pid 424970 tid 424970 thread 1 bound to OS proc set 1
DEEPMD INFO built lr
DEEPMD INFO built network
DEEPMD INFO built training
DEEPMD INFO initialize model from scratch
DEEPMD INFO start training at lr 1.00e-03 (== 1.00e-03), decay_step 5000, decay_rate 0.996400, final lr will be 2.00e-08

It is expected that the two processes will not interfere each other, since they use different GPUs and CPUs. However, I found that the speed of task A was dramatically decreased (from 2.5s per 100 steps to 4s per 100 steps) immediately after task B was submitted, so did task B. I've also find a very similar issue (#1818), however there seems to be no conclusive answers. Does anyone have ideas on how to solve it? Thanks!

[njzjz]

You can list the CPU usage by ps -L -eo pcpu,psr,pid,lwp,user,args.

[DingChangjie]

You can list the CPU usage by ps -L -eo pcpu,psr,pid,lwp,user,args.

@njzjz
Thanks ! I suddenly realize that I should enforce my CPU to balance all the threads. Therefore, I manually set an environment variable export KMP_AFFINITY=scatter according to Intel's official manual and this WORKS on my Core i9-10900X machine !!!
Hope this can really help others who also face this problem. BTW, I believe there might be even better settings of KMP_AFFINITY since different machines can have different configurations. Maybe the documentation could be enhanced to inform the users who may not be very familiar with the management of threads?

[njzjz]

Per Intel documentation, one can also explicitly set proclist to control the threads.

[DingChangjie]

Thank you very much! Indeed, the "proclist" can achieve even better performance.