Help understanding error message

[hahahannes]

Hi,

I am running UCX with the following settings:

-x UCX_IB_GPU_DIRECT_RDMA=yes -x UCX_LOG_LEVEL=debug -x UCX_TLS=rc,gdr_copy

to enforce the usage of GPU RDMA. I am getting the following error:

select.c:649  UCX  ERROR   no copy across memory types transport to <no debug data>: rc_verbs/mlx5_0:1 - Destination is unreachable, rc_mlx5/mlx5_0:1 - Destination is unreachable, rc_verbs/mlx5_1:1 - Destination is unreachable, rc_mlx5/mlx5_1:1 - Destination is unreachable, rc_verbs/mlx5_2:1 - Destination is unreachable, rc_mlx5/mlx5_2

With -x UCX_TLS=rc,gdr_copy,cuda_copy, it works. I assume cuda_copy is used as a fallback and will result in staging through host memory.

I would be very grateful for an explanation of what the possible error sources could be. I am assuming that this means that the GDR setup is not correct but I am not sure.

Thanks a lot !