From 2e62456b540aac94773fb8bc2dd1ee73be47c30a Mon Sep 17 00:00:00 2001 From: Howard Pritchard Date: Wed, 13 Dec 2023 15:41:55 -0700 Subject: [PATCH 1/2] docs: add some info about HPE CXI ofi provider and also make a statement about the OFI BTL more accurate. Related to #12038 Signed-off-by: Howard Pritchard (cherry picked from commit 2718732fc18af7ce00ebd62c002a85282766b38b) --- docs/tuning-apps/networking/ofi.rst | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/tuning-apps/networking/ofi.rst b/docs/tuning-apps/networking/ofi.rst index 4c9ba56290c..8086e9d7aa3 100644 --- a/docs/tuning-apps/networking/ofi.rst +++ b/docs/tuning-apps/networking/ofi.rst @@ -18,6 +18,7 @@ including (but not limited to): * Cisco usNIC * Cray uGNI * Cornelius Networks Omni-Path +* HPE Slingshot 11 In general, the OFI-based components in Open MPI will auto-select themselves as appropriate at run time. @@ -43,8 +44,9 @@ communications: ``FI_EP_RDM`` endpoints, ``FI_TAGGED`` capabilities, and ``FI_ORDER_SAS`` ordering). -#. ``ofi`` BTL: Available since Open MPI v4.0.0, this component is - used for one-sided MPI communications (e.g., ``MPI_PUT``). The +#. ``ofi`` BTL: Available since Open MPI v4.0.0, this component is primarily + intended for one-sided MPI communications (e.g., ``MPI_PUT``). It + can also support BTL send/recv operations. ``ofi`` BTL requires that the Libfabric provider support reliable datagrams, RMA and atomic operations, and remote atomic completion notifications (specifically: ``FI_EP_RDM`` endpoints, ``FI_RMA`` @@ -76,6 +78,11 @@ list. For example: shell$ ompi_info --param mtl ofi --level 9 +.. important:: When using the HPE CXI provider and ``mpirun`` as the job launcher, + the PRTE ``ras_base_launch_orted_on_hn`` MCA parameter must be set to 1. + This can be done by adding ``--prtemca ras_base_launch_orted_on_hn 1`` to the job launch + command line. + For more information refer to the `Libfabric web site `_. From 212d8ae890fe2fb7b4d83a0dad5f5f6199aa4ec5 Mon Sep 17 00:00:00 2001 From: Howard Pritchard Date: Fri, 15 Dec 2023 08:31:05 -0700 Subject: [PATCH 2/2] docs: refine wording related to HPE CXI provider Signed-off-by: Howard Pritchard (cherry picked from commit e583e4848a8935890ddcbab93d5c4bb784d73ce1) --- docs/tuning-apps/networking/ofi.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/tuning-apps/networking/ofi.rst b/docs/tuning-apps/networking/ofi.rst index 8086e9d7aa3..4e3e1387ce0 100644 --- a/docs/tuning-apps/networking/ofi.rst +++ b/docs/tuning-apps/networking/ofi.rst @@ -79,9 +79,10 @@ list. For example: shell$ ompi_info --param mtl ofi --level 9 .. important:: When using the HPE CXI provider and ``mpirun`` as the job launcher, - the PRTE ``ras_base_launch_orted_on_hn`` MCA parameter must be set to 1. + it is recommended that the PRTE ``ras_base_launch_orted_on_hn`` MCA parameter be set to 1. This can be done by adding ``--prtemca ras_base_launch_orted_on_hn 1`` to the job launch - command line. + command line. This ensures that MPI processes launched on the first node of + an allocation are able to use the CXI provider. For more information refer to the `Libfabric web site `_.