deepmd horovodrun error

[Lewis-YL]

Hi guys,

Do you know how to fix this error?

I'm using deepmd v2.1.1, cuda 11.6.

[0] DEEPMD INFO built lr
[0] DEEPMD INFO built network
[0] DEEPMD INFO Scale learning rate by coef: 2.000000
[0] DEEPMD INFO built training
[0] DEEPMD WARNING To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, TF_INTRA_OP_PARALLELISM_THREADS, and TF_INTER_OP_PARALLELISM_THREADS.
[0] DEEPMD INFO initialize model from scratch
[0] DEEPMD INFO broadcast global variables to other tasks
[0] ptxlab:258238:258290 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
[0] ptxlab:258238:258290 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
[0] ptxlab:258238:258290 [0] NCCL INFO NET/IB : No device found.
[0] ptxlab:258238:258290 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
[0] ptxlab:258238:258290 [0] NCCL INFO Using network Socket
[0] NCCL version 2.12.12+cuda11.6
[0]
[0] ptxlab:258238:258290 [0] init.cc:255 NCCL WARN Cuda failure 'CUDA driver is a stub library'
[0] ptxlab:258238:258290 [0] NCCL INFO init.cc:913 -> 1
[0] ptxlab:258238:258290 [0] NCCL INFO init.cc:950 -> 1
[0] ptxlab:258238:258290 [0] NCCL INFO init.cc:963 -> 1
....

[njzjz]

Duplicate of #1774. Could you provide more information and discuss at #1774?