[BTL/Vader] Can Vader's Single-copy Mechanism Utilize RMDA Loopback Transport ?

[vitduck]

Background information

I am investigating the performance of various single-copy transport implemented in Vader BTL.
The most relevant information I could find is an official announcement circa 2014:
https://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy

My background is not in networking, so I appreciate if you could give me some pointers for better understanding.

Open MPI version

Open MPI: 4.1.7rc1 (gc2cc293)

Open MPI build

$ ompi_info 
...
Configure command line: 
    '--prefix=/apps/compiler/gcc/12.2.0/cudampi/12.3/openmpi/4.1.x'
    '--enable-mpi-cxx' '--with-slurm=/usr'
    '--with-pmi=/usr' '--with-cma'
    '--with-knem=/opt/knem-1.1.4.90mlnx3'
    '--with-xpmem=/apps/common/xpmem/2.7.4/'
    '--with-hcoll=/opt/mellanox/hcoll'
    '--with-ucx=/apps/common/ucx/1.17.0/cuda/12.3'
    '--with-cuda=/apps/cuda/12.3'

System information

• Operating system/version: 7.9.2009
• Computer hardware: 2x Xeon(R) Gold 6140
• Network type: Mellanox HDR100

Methods and Results

• The intra-node latency and bandwidth was conducted using OSU Benchmark v7.5
• Each mechanism was selected via btl_vader_single_copy_mechanism.
• Vader results are compared with UCX's implementation, which also utilized cma, knem and xpmem
• The expectation is cico < cma < knem < xpmem < ucx

mpirun                                      
  --np 2                                  
  --mca pml ^ucx                          
  --mca pml_ob1_verbose 100               
  --mca btl_base_verbose 100              
  --mca btl self,vader                    
  --mca opal_warn_on_missing_libcuda 0    
  --mca btl_vader_single_copy_mechanism {none|cma|knem|xpmem} 
  $PREFIX/osu_latency H H                 

Due to the clustering to data at small message limit. Please refer to the following table:

message size	none	cma	knem	xpmem	ucx
1	0.28	0.27	0.27	0.28	0.27
2	0.27	0.26	0.27	0.27	0.26
4	0.27	0.26	0.26	0.27	0.26
8	0.26	0.26	0.26	0.26	0.26
16	0.27	0.26	0.27	0.27	0.26
32	0.27	0.26	0.27	0.27	0.28
64	0.30	0.29	0.30	0.30	0.28
128	0.36	0.36	0.36	0.35	0.46
256	0.41	0.42	0.43	0.42	0.46
512	0.53	0.55	0.55	0.50	0.55
1024	0.62	0.66	0.65	0.51	0.60
2048	0.77	0.80	0.83	0.53	0.80
4096	1.39	1.48	1.54	0.58	1.16
8192	1.97	1.65	1.63	0.66	1.88
16384	3.35	2.16	2.02	1.04	2.89
32768	5.47	2.88	2.47	1.63	4.50
65536	7.10	4.31	3.33	2.54	7.58
131072	11.05	7.32	5.04	4.40	13.40
262144	19.25	13.17	8.65	8.12	13.29
524288	49.15	27.98	19.59	39.61	19.63
1048576	101.53	80.90	64.44	138.47	41.95
2097152	196.55	211.33	166.92	313.23	120.63
4194304	409.93	444.25	348.34	681.75	246.02

KNEM vs. XPMEM at large message limit.

Up until 256KB, xpmem offers the best latency (8.12us), following by knem (8.65us), and cma (13.17us), respectively.
But at 2MB and 4MB limit, xpmem (681.75us) is outperformed by knem (348.34us) and even cico (409.93us).

By looking at the UCX's protocol selection, UCX also prefers knem over xpmem, e.g.

[1746173000.065184] [skl02:244412:0]   +--------------------------------+------------------------------------------------------------------------------------------------+
[1746173000.065193] [skl02:244412:0]   | ucp_context_2 intra-node cfg#1 | tagged message by ucp_tag_send*(multi) from host memory                                        |
[1746173000.065198] [skl02:244412:0]   +--------------------------------+-----------------------------------------------+------------------------------------------------+
[1746173000.065203] [skl02:244412:0]   |                          0..92 | eager short                                   | sysv/memory                                    |
[1746173000.065207] [skl02:244412:0]   |                       93..5122 | eager copy-in copy-out                        | sysv/memory                                    |
[1746173000.065212] [skl02:244412:0]   |                    5123..13046 | (?) rendezvous copy from mapped remote memory | xpmem/memory                                   |
[1746173000.065216] [skl02:244412:0]   |                     13047..inf | (?) rendezvous zero-copy read from remote     | 71% on knem/memory and 29% on rc_mlx5/mlx5_0:1 |
[1746173000.065220] [skl02:244412:0]   +--------------------------------+-----------------------------------------------+------------------------------------------------+

Here, UCX splits the large message between knem and rc_mlx5/mlx5_0:1 corresponding to loopback mechanism, if I understand correctly.

So my questions are:

• I know this is on a case-by-case basis. But is there a fundamental reason for the transition from xpmem to knem ?
  The direct memory load/store should offer the best latency, unless there is an actual memcpy involves that is slower than kernel-assisted copy.
  In other words, is this an intrinsic properties of the underlying CPU ? For instance:

  • Intel CPUs: knem > xpmem
  • AMD CPUs: 'knem' < 'xpmem`
    The above could be hypothetically explained through ring-bus interconnect and infinity fabric for Intel, and AMD, respectively.

• Can Vader utilize RMDA through Mellanox's loopback to improve latency of large message ?
  Vader's MCA params offers the following parameter:

MCA btl vader: parameter "btl_vader_cuda_rdma_limit" (current value: "18446744073709551615", data source: default, level: 5 tuner/detail, type: size_t)
MCA btl vader: parameter "btl_vader_rdma_pipeline_send_length" (current value: "32768", data source: default, level: 4 tuner/basic, type: size_t)
MCA btl vader: parameter "btl_vader_rdma_pipeline_frag_size" (current value: "32768", data source: default, level: 4 tuner/basic, type: size_t)
MCA btl vader: parameter "btl_vader_min_rdma_pipeline_size" (current value: "2147483647", data source: default, level: 4 tuner/basic, type: size_t)

The idea of pipelining a fragment of message via RDMA is similar to UCX. Yet, I don't see them being utilized in the verbose message.

[jsquyres]

@bosilca @janjust The user is specifically asking about UCX. Can you answer?

[vitduck]

Hi @jsquyres

I also noticed that there is a shmem framework, i.e.

$ ompi_info | grep shmem 
 
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.1.7)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.1.7)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.1.7)

Which implement posix and sysv shared memory.

Perusing the source codes under: open-mpi/ompi/tree/main/opal/mca/shmem, they seems to be not related to btl/vader.
Could you kindly clarify their roles ?

Regards.

[bosilca]

The [very] short answer is that the shmem components provide the backend for the CICO support for vader/sm, while the smsc components provide the support for Shared Memory Single Copy. The idea is that up to a certain size CICO is faster as it does not involve any kernel operation (that is directly related to the number of pages in the payload), even as 2 memory copies are necessary (to and then from the shared memory region). After that threshold, a single copy will save enough time to compensate for the kernel-bound memory registration.

All the single copy mechanisms you named, knem, cma and xpmem work more of less on the same way, memory pages from one process are made available to another process (that's the linear part on the number of pages) allowing a single memcopy to move the data from the source into the destination. There are differences between them, but I would have not expected them to be as high as you pointed in your analysis. I am doubtful the costs are architecture-related.

For the UCX part, you are correct UCX will move the data with a predefined split across the mlx device as well, which involved 2 transits over the PCI bus. The OB1 PML does not do that by defaults for this case, but can do it with a dynamic split (aka each BTL will take another fragment as soon as it finishes the precedents fragments, up to the predefined pipeline depth).

[vitduck]

@bosilca
Thanks for your clarification regarding the role of smsc.
Now I understood that both vader and ucx implement same mechanisms for shared memory transport.
To reiterate:

• Double-copy: sysv/posix
• Single-copy: cma/knem/xpmem.
  Previously, it is not clear from documentation how double-copy method is implemented in Open MPI.
  The differences are:
• UCX can dynamically choose protocols based on cost model
• UCX can use hardware loopback

In case of sysv/posix, my understanding is that

• The sender can declare its buffer as a shared memory object with shm_open()
• The receiver can map this memory region into its virtual memory wiht mmap(), then perform a regular memcpy().
  This would require only single memcpy() instead of two.
  Could you briefly mention why two-copy modes are adopted ?

Again, thanks very much for your time.

[bosilca]

OMPI can use hardware loopback too, it just chooses not to by default. In general, OMPI stays away from benchmark-type optimizations, and focuses on real scenarios performance. This is one such example, while running a bw test, the entire PCI bus is dedicated to your benchmark, you are not colliding with anything else, and not taking away scarce bandwidth from anything else. In a real app, if you start moving your local data over PCI, you will certainly share the bandwidth-limited PCI bus with other devices (such as GPU) and the overall performance will go down.

The protocol you describe is exactly how the single-copy protocol works. But here you have 3 syscall in your critical path, one for registering the send buffer, one for mapping on the receiver memory the send buffer, and another one (that can be done in parallel on the sender and receiver) to undo all these things. Thus, such protocol only make sense if the cost of memcpy saved is larger than the cost of the 3 syscalls. On a 2 copy protocol, you never register your send buffer nor map it on the receiver memory. Instead, during the setup phase the two processes agree on a memory zone that will be common on the two processes, a machanish similar to what you describe but it is not message dependent. During a send operation, the send buffer is then copied (via a fragmented protocol) into this shared memory area, and then the receiver copies the data out into the receive buffer, hence the two copies.

[vitduck]

Thanks for explanation.
I will certainly heed your advice regarding the interpretation of benchmarks.

My goal is not trying to demonstrate that UCX is better than Open MPI or vice versa, but to understand the underlying mechanisms.
Open MPI is crucial part of our workload, yet documentation and public presentation are either too surface-level or too technical.
Using OSU benchmark is just a way for us to have a controlled experiments and figure out the behavior or intra-node transport.

If you have no further comments, I will close this issue after a short while.

Regards.

[vitduck]

@jsquyres @bosilca

Thanks very much for your help.