Resource manager integration check is missing with openmpi v5 at configure step

[puneet336]

Hi Team ,
I am trying to enable the openmpi integration with LSF on rockylinux 9.5.

I see that in the openmpi 4.1.8 , i can easily check the support for LSF in the Resource Managers section of configure script. #10943

Example -

[puneets@lsfmaster1 openmpi-4.1.8]$ ./configure --prefix=$PWD/install --with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib --with-lsf-libdir="${LSF_LIBDIR}" 2>&1|tee cflog1

and here is the output :

..........................<output snipped>........................
Open MPI configuration:
-----------------------
Version: 4.1.8
..........................<output snipped>..........................
Miscellaneous
-----------------------
CUDA support: no
..........................<output snipped>..........................

Transports
-----------------------
Cisco usNIC: no
..........................<output snipped>..........................

Resource Managers
-----------------------
Cray Alps: no
Grid Engine: no
LSF: yes
Moab: no
Slurm: yes
ssh/rsh: yes
Torque: no

OMPIO File Systems
-----------------------
DDN Infinite Memory Engine: no

this is great as i dont have to go through make and make install to see the support status of workload manager with openmpi.

in the version 5 of openmpi , i see the following -

Open MPI configuration:
-----------------------
Version: 5.0.7
MPI Standard Version: 3.1
Build MPI C bindings: yes
Build MPI Fortran bindings: no
Build MPI Java bindings (experimental): no
Build Open SHMEM support: false (no spml)
Debug build: no
Platform file: (none)
 
Miscellaneous
-----------------------
Atomics: GCC built-in style atomics
Fault Tolerance support: mpi
HTML docs and man pages: installing packaged docs
hwloc: internal
libevent: internal
Open UCC: no
pmix: internal
PRRTE: internal
Threading Package: pthreads
 
Transports
-----------------------
Cisco usNIC: no
Cray uGNI (Gemini/Aries): no
Intel Omnipath (PSM2): no (not found)
Open UCX: no
OpenFabrics OFI Libfabric: no (not found)
Portals4: no (not found)
Shared memory/copy in+copy out: yes
Shared memory/Linux CMA: yes
Shared memory/Linux KNEM: no
Shared memory/XPMEM: no
TCP: yes
 
Accelerators
-----------------------
CUDA support: no
ROCm support: no
 
OMPIO File Systems
-----------------------
DDN Infinite Memory Engine: no
Generic Unix FS: yes
IBM Spectrum Scale/GPFS: no (not found)
Lustre: no (not found)
PVFS2/OrangeFS: no

### Q: i am unable to see "Resource Managers" section after configure step with v5. Is there a way to determine whether the Resource Manager/WorkLoadManager (say LSF support) has been enabled from config.log?

please advice

i see following test in the config.log, which is not very helpful in determinig if LSF support is enabled with configure step

puneets@lsfmaster1 openmpi-5.0.7]$ grep lsf config.log |grep -v OPAL_PACKAGE_STRING
  $ ./configure --prefix=/home/puneets/openmpi-5.0.7/install --with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib --with-lsf-libdir=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib
hostname = lsfmaster1.example.com
PATH: /usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/etc/
PATH: /usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/bin/
configure:8329: result: Open MPI puneets@lsfmaster1.example.com Distribution
configure:60696: running /bin/sh ./configure --disable-option-checking '--prefix=/home/puneets/openmpi-5.0.7/install' --disable-dns --disable-http --disable-rpc --disable-openssl --enable-thread-support --disable-evport --disable-gcc-warnings --disable-libevent-regress '--with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' '--with-lsf-libdir=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' --cache-file=/dev/null --srcdir=.
configure:62993: running /bin/sh ./configure --disable-option-checking '--prefix=/home/puneets/openmpi-5.0.7/install' --enable-plugins '--with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' '--with-lsf-libdir=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' --cache-file=/dev/null --srcdir=.
configure:64407: running /bin/sh ./configure --disable-option-checking '--prefix=/home/puneets/openmpi-5.0.7/install' --without-tests-examples --enable-pmix-binaries --disable-pmix-backward-compatibility --disable-visibility --disable-devel-check --with-libevent --disable-libevent-lib-checks --with-libevent-extra-libs="/home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi/libevent_core.la /home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi/libevent_pthreads.la" --disable-hwloc-lib-checks --with-hwloc-extra-libs="/home/puneets/openmpi-5.0.7/3rd-party/hwloc-2.7.1/hwloc/libhwloc.la" --with-wrapper-libs="-levent_core -levent_pthreads " '--with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' '--with-lsf-libdir=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' 'CPPFLAGS=-I/home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi -I/home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi/include -I/home/puneets/openmpi-5.0.7/3rd-party/hwloc-2.7.1/include -I/home/puneets/openmpi-5.0.7/3rd-party/hwloc-2.7.1/include' --cache-file=/dev/null --srcdir=.
configure:67055: running /bin/sh ./configure --disable-option-checking '--prefix=/home/puneets/openmpi-5.0.7/install' --enable-prte-ft --with-proxy-version-string=5.0.7 --with-proxy-package-name="Open MPI" --with-proxy-bugreport="https://www.open-mpi.org/community/help/" --disable-devel-check --enable-prte-prefix-by-default --with-libevent --disable-libevent-lib-checks --with-libevent-extra-libs="/home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi/libevent_core.la /home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi/libevent_pthreads.la" --disable-hwloc-lib-checks --with-hwloc-extra-libs="/home/puneets/openmpi-5.0.7/3rd-party/hwloc-2.7.1/hwloc/libhwloc.la" --disable-pmix-lib-checks --with-pmix-extra-libs="/home/puneets/openmpi-5.0.7/3rd-party/openpmix/src/libpmix.la" '--with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' '--with-lsf-libdir=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' 'CPPFLAGS= -I/home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi -I/home/puneets/openmpi-5.0.7/3rd-party/libevent-2.1.12-stable-ompi/include -I/home/puneets/openmpi-5.0.7/3rd-party/hwloc-2.7.1/include -I/home/puneets/openmpi-5.0.7/3rd-party/hwloc-2.7.1/include -I/home/puneets/openmpi-5.0.7/3rd-party/openpmix/include -I/home/puneets/openmpi-5.0.7/3rd-party/openpmix/include -I/home/puneets/openmpi-5.0.7/3rd-party/openpmix/ -I/home/puneets/openmpi-5.0.7/3rd-party/openpmix/' --cache-file=/dev/null --srcdir=.
configure:67711: result:  '--prefix=/home/puneets/openmpi-5.0.7/install' '--with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib' '--with-lsf-libdir=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib'
on lsfmaster1.example.com
OPAL_CONFIGURE_CLI=' \'\''--prefix=/home/puneets/openmpi-5.0.7/install\'\'' \'\''--with-lsf=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib\'\'' \'\''--with-lsf-libdir=/usr/share/lsf/10.1/linux2.6-glibc2.3-x86_64/lib\'\'''
OPAL_CONFIGURE_HOST='lsfmaster1.example.com'
[puneets@lsfmaster1 openmpi-5.0.7]$ 

[rhc54]

I don't know what the OMPI folks do with it, but the runtime does output it on stderr:

Resource Managers
-----------------------
Grid Engine: no
LSF: no
PBS: yes (scheduler)
Slurm: yes
ssh/rsh: yes
Torque: no (not found)

[ggouaillardet]

@puneet336 I guess you only look at the end of the configure output.

It was there because the runtime of Open MPI 4 was orte that was an Open MPI project.
Now the runtime is provided by prrte that is an external package, with its own configure so this information
is not displayed at the end of (Open MPI's) configure any more.

[puneet336]

Thank you @ggouaillardet for the response.
I have a few follow-up questions based on your latest reply:

1. OpenMPI v5's configure script still includes --with-lsf and --with-lsf-dir flags.
   If PRRTE is now responsible for handling runtime and workload manager (WPM) integration, what role do these LSF-related flags play in OpenMPI’s configuration? Are they still functional, or are they legacy placeholders?

2. Is PRRTE a mandatory component for OpenMPI v5?
   If I build OpenMPI v5 using the standard configure && make && make install flow without PRRTE, will mpirun or similar runtime tools be omitted? I'm trying to understand whether OpenMPI now supports alternative runtimes or if this separation from ORTE to PRRTE was explicitly done to make OpenMPI runtime-agnostic.

3. Is the correct process to build PRRTE first, then pass its installation path to OpenMPI during configuration?
   If so, could you confirm whether OpenMPI dynamically links to PRRTE or relies on its binaries at runtime?

4. Could you recommend a detailed tutorial or reference guide for building OpenMPI v5?
   I’m especially interested in enabling UCX,CUDA support, so guidance on which configuration flags to use both in PRRTE and OpenMPI would be highly valuable.

Appreciate your insights!

[ggouaillardet]

Some configure flags are simply passed from Open MPI configure down to prrte configure.
prrte is now a 3rd-party component (such as openpmix or hwloc) and is automatically built (unless specified otherwise). You can definitely build Open MPI with an external prrte (e.g. configure --with-prrte=... but I am not sure if you can build Open MPI without prrte (internal nor external), for example if you only plan to use srun.

The MPI application (or libmpi.so) does not link against prrte because it does not have to. From a runtime point of view, a MPI application is "only" a PMIx client. Open MPI's mpirun is now (oversimplified) an alias to `prrte)

[rhc54]

1. OpenMPI v5's configure script still includes --with-lsf and --with-lsf-dir flags.
   If PRRTE is now responsible for handling runtime and workload manager (WPM) integration, what role do these LSF-related flags play in OpenMPI’s configuration? Are they still functional, or are they legacy placeholders?

Same as before - they control the building of the WPM integration. They just pass down to the runtime, just like they always did. It's just PRRTE instead of ORTE, and has a separate configure execution (which is why the output is not at the end) since it is an embedded copy of an independent project.

2. Is PRRTE a mandatory component for OpenMPI v5?
   If I build OpenMPI v5 using the standard configure && make && make install flow without PRRTE, will mpirun or similar runtime tools be omitted? I'm trying to understand whether OpenMPI now supports alternative runtimes or if this separation from ORTE to PRRTE was explicitly done to make OpenMPI runtime-agnostic.

If you specify --without-prrte, then no - you will not get an mpirun or any other runtime tool. OMPI doesn't support alternative runtimes any more than it used to do. You can, however, still direct launch an OMPI application (with whatever equivalent to Slurm's srun you use in LSF) without building mpirun.

The decision to use an external project (i.e., PRRTE) for the runtime was strictly based on support as the OMPI community wasn't seeing any internal support for the ORTE runtime.

3. Is the correct process to build PRRTE first, then pass its installation path to OpenMPI during configuration?
   If so, could you confirm whether OpenMPI dynamically links to PRRTE or relies on its binaries at runtime?

You can use an external PRRTE if you like - just set --with-prrte=external or point it to the path where PRRTE was installed. As Gillles said, there is no PRRTE dependency within an OMPI app

[jsquyres]

4. Could you recommend a detailed tutorial or reference guide for building OpenMPI v5?
   I’m especially interested in enabling UCX,CUDA support, so guidance on which configuration flags to use both in PRRTE and OpenMPI would be highly valuable.

Don't forget to check out https://docs.open-mpi.org; the diagram and corresponding text in https://docs.open-mpi.org/en/v5.0.x/installing-open-mpi/required-support-libraries.html might be helpful.

[puneet336]

Thank you for the insights and responses ,
We had easybuild script for PRRTE and openmpi v5.

1. We installed libnsl2 and enabled lsf support with prrte.
2. with external prrte , the openmpiv5 was able to get host list from LSF and successfully perform multinode run.