Why does the TL select SELF and KNEM when communicating between nodes

[w7658610]

Hi,

like this, When I used OSU_BW to test communication between 2 nodes (1 process per node).
Why self and knem(or cma)? Is there an IPC?

[1653018173.525616] [j13r2n01:17472:0] ucp_worker.c:1627 UCX INFO ep_cfg[0]: tag(self/memory knem/memory rc_mlx5/mlx5_0:1 rc_mlx5/mlx5_1:1 rc_mlx5/mlx5_2:1);
[1653018173.577027] [j13r2n01:17472:0] ucp_worker.c:1627 UCX INFO ep_cfg[1]: tag(rc_mlx5/mlx5_0:1 rc_mlx5/mlx5_1:1 rc_mlx5/mlx5_2:1 rc_mlx5/mlx5_3:1);

Thanks.

[brminich]

I suppose this configuration ep_cfg[0] represents a connection to self (the same rank), while ep_cfg[1] corresponds to the other rank connection

[w7658610]

I still have some questions.

1. What does rank mean here? The process?

2. And communication between 2 nodes (1 process per node),why ep needs " a connection to self " .

3. Then I notice that self/memory is usually in ep_cfg[0] and posix/memory is in [1].

• ep_cfg[0, 1, 2, 3]，0 1 2 3 ,Is there a relationship between configuration and numbers?

[hoopoepg]

1. rank is MPI unit, it may be represented as separate process or even thread in process
2. MPI allows to send messages to itself, for this case UCX creates configuration to send messages like loopback
3. there is no strong correlation in configuration ordering

[brminich]

1. You may consider rank as a process (rank is MPI term).
2. In MPI there is a mode when all processes establish connections to all others during MPI_Init (typically this is the case when number of ranks is relatively low)
3. The configuration numbers are just internal UCX indexes (like a configurations array index)

BTW, according to the output you are running all the processes on the same node (node name is the same for all outputs)

[w7658610]

thank you for answering my question!!