ucx_perftest: Is a GPU required on the client node even when sending from host memory? (host -> cuda)

[kanlkan]

I would like to test GPU Direct RDMA between two nodes. (host memory -> GPU memory)

• Client node
  • Mellanox ConnectX-5
  • No NVIDIA GPUs
    • But CUDA12.9 and cuda-driver are installed.
  • ucx 1.18.1
    • Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --with-cuda=/usr/local/cuda/ --with-rc --with-ud --with-dc --with-ib-hw-tm --with-dm
• Server node
  • Mellanox ConnectX-5
  • 1 NVIDIA GPU (NVIDIA T4)
  • ucx 1.18.1
    • Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --with-cuda=/usr/local/cuda/ --with-rc --with-ud --with-dc --with-ib-hw-tm --with-dm

I tried to test with ucx_perftest on the above nodes, but failed.

Server node outputs

$ CUDA_VISIBLE_DEVICES=0 \
  UCX_NET_DEVICES=mlx5_0:1 \
  UCX_TLS=rc,cuda \
  ucx_perftest \
  -t ucp_am_bw \
  -s 100000 -n 10000 \
  -m host,cuda \
  -c 0
Waiting for connection...
Accepted connection from 192.168.90.125:44790
+----------------------------------------------------------------------------------------------------------+
| API:          protocol layer                                                                             |
| Test:         am bandwidth / message rate                                                                |
| Data layout:  (automatic)                                                                                |
| Send memory:  host                                                                                       |
| Recv memory:  cuda                                                                                       |
| Message size: 100000                                                                                     |
| Window size:  32                                                                                         |
| AM header size: 0                                                                                        |
+----------------------------------------------------------------------------------------------------------+
[1747016720.473912] [eagle04:320848:0]        perftest.c:133  UCX  ERROR recv() remote closed connection
[1747016720.473933] [eagle04:320848:0]        perftest.c:302  UCX  ERROR sock: rte recv: remote peer failure

Client node outputs

$ UCX_NET_DEVICES=mlx5_0:1 \
  UCX_TLS=rc,cuda \
  ucx_perftest \
  -t ucp_am_bw \
  -s 100000 -n 10000 \
  -m host,cuda \
  -c 0 192.168.90.124
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[1747016720.062075] [eagle05:35244:0]    perftest_run.c:348  UCX  ERROR Failed to run test: No such device

I think the client node does not require a GPU when testing host -> cuda because it uses only host memory.
Is a GPU required on the client node even when sending from host memory (host -> cuda) with ucx_perftest?

[yosefe]

yes, it is required, since some tests are bidirectional. there is no special logic to enabled mixed setup scenario for uni directional tests.

[kanlkan]

Thank you for your reply! I understood it and marked it as an answer.