Error while doing Parallel Training

[AnuragKr]

Hi,

I hope you all are doing well. I am facing an error while doing parallel training.

Installation - From source
Version -- DeePMD-kit v2.2.0b0
NCCL - 2.16.2
MPIRUN - 4.14.2
System -- DGX V100 Server
Command --

 mpirun -np 2 dp train input.json

Error --

Command --

CUDA_VISIBLE_DEVICES=4,5,6,7 horovodrun -np 4  dp train --mpi-log=workers input.json

Error --

[AnuragKr]

@njzjz Could you please help me figure out the above issue mentioned?

[njzjz]

For NCCL errors, you may still set the environment variable NCCL_DEBUG=INFO to debug it.

[AnuragKr]

After setting the environment I got the following output --

[njzjz]

Isn't it the same as #1774?

[AnuragKr]

Yes, kind of it is the same, but I installed NCCL where that bug was already solved. Still, I am getting this error.
Last time also I was able to work around through conda but not via source. This time also I am facing a problem with source installation.
Sorry for creating another discussion.