README: Update information about UCX

Signed-off-by: Jeff Squyres <[email protected]>

Author: jsquyres

README

@@ -706,6 +706,32 @@ Network Support
   mechanisms for Open MPI to utilize single-copy semantics for shared
   memory.
 
+- In prior versions of Open MPI, InfiniBand and RoCE support was
+  provided through the openib BTL and ob1 PML plugins.  Starting with
+  Open MPI 4.0.0, InfiniBand support through the openib+ob1 plugins is
+  both deprecated and superseded by the UCX PML component.
+
+  UCX is an open-source optimized communication library which supports
+  multiple networks, including RoCE, InfiniBand, uGNI, TCP, shared
+  memory, and others.
+
+  While the openib BTL depended on libibverbs, the UCX PML depends on
+  the UCX library.  The UCX library can be downloaded from
+  http://www.openucx.org/ or from various Linux distribution
+  repositories (e.g., Fedora/RedHat yum repositories).  The UCX
+  library is also part of Mellanox OFED and Mellanox HPC-X binary
+  distributions.
+
+  Once installed, Open MPI can be built with UCX support by adding
+  --with-ucx to the Open MPI configure command. Once Open MPI is
+  configured to use UCX, the runtime will automatically select the UCX
+  PML if one of the supported networks is detected (e.g., InfiniBand).
+  It's possible to force using UCX in the mpirun or oshrun command
+  lines by specifying any or all of the following mca parameters:
+  "-mca pml ucx" for MPI point-to-point operations, "-mca spml ucx"
+  for OpenSHMEM support, and "-mca osc ucx" for MPI RMA (one-sided)
+  operations.
+
 Open MPI Extensions
 -------------------
 
@@ -1018,6 +1044,19 @@ NETWORKING SUPPORT / OPTIONS
   covers most cases.  This option is only needed for special
   configurations.
 
+--with-ucx=<directory>
+  Specify the directory where the UCX libraries and header files are
+  located.  This option is generally only necessary if the UCX headers
+  and libraries are not in default compiler/linker search paths.
+
+--with-ucx-libdir=<directory>
+  Look in directory for the UCX libraries.  By default, Open MPI will
+  look in <ucx_directory>/lib and <ucx_ directory>/lib64, which covers
+  most cases.  This option is only needed for special configurations.
+
+--with-usnic
+  Abort configure if Cisco usNIC support cannot be built.
+
 --with-verbs=<directory>
   Specify the directory where the verbs (also known as OpenFabrics
   verbs, or Linux verbs, and previously known as OpenIB) libraries and
@@ -1063,8 +1102,6 @@ NETWORKING SUPPORT / OPTIONS
    package, configure will safely abort with a helpful message telling
    you that you should not use --with-verbs-usnic.
 
---with-usnic
-  Abort configure if Cisco usNIC support cannot be built.
 
 RUN-TIME SYSTEM SUPPORT
 
@@ -2032,7 +2069,7 @@ timer       - High-resolution timers
 Each framework typically has one or more components that are used at
 run-time.  For example, the btl framework is used by the MPI layer to
 send bytes across different types underlying networks.  The tcp btl,
-for example, sends messages across TCP-based networks; the openib btl
+for example, sends messages across TCP-based networks; the UCX PML
 sends messages across OpenFabrics-based networks.
 
 Each component typically has some tunable parameters that can be