prterun hangs, Open MPI v5.0.6

[axiom-ctrl]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

• Open MPI v5.0.6

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

• from a source/distribution tarball

Please describe the system on which you are running

• Operating system/version: Red Hat Enterprise Linux 8.8
• Computer hardware: HPE Cray EX
• Network type: HPE Slingshot

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

I have noticed that sometimes, without any specific regularity, after initiating an mpirun command, a prterun hang occurs, meaning the main executable application does not start. The likelihood of this happening increases when launching a larger number of MPI processes across different computational nodes.

Open MPI was compiled with PBS support, as outlined below:

./configure \
    CC="gcc" \
    CXX="g++" \
    FC="gfortran" \
    --prefix=${install_dir} \
    --enable-shared \
    --enable-static \
    --with-pbs \
    --with-libfabric="/opt/cray" \
    --with-libfabric-libdir="/opt/cray/lib64" \
    --with-tm="/opt/pbs" \
    --with-tm-libdir="/opt/pbs/lib"

I run the application as follows:

mpirun -np 32 pw.x -i scf.input

I have observed the same behavior with earlier versions of Open MPI 5 as well. This issue is not exclusively related to the application I used (i.e. Quantum ESPRESSO) and occurs independently of any specific computational node. In other words, simply restarting the job often resolves the problem.

[rhc54]

Usually traced back to a race condition in the TM support integration. The TM library isn't the most robust interface, I'm afraid, as it is rather atomistic - launch daemon, poll until reap, etc. We've seen plenty of "lost" interactions over the years.

You can try using the ssh launcher instead - it is more robust. Just add --prtemca plm ssh to your mpirun cmd line. Long-running debate over which (tm or ssh) is faster at scale, but no question that ssh tends to be more reliable.

[axiom-ctrl]

Thank you for the prompt response. I attempted to run the command as:

mpirun --prtemca plm ssh -np 32 --hostfile $PBS_NODEFILE pw.x -i scf.input

... but it outputs:

/apps/utils/openmpi/5.0.6/gnu/11.2.1/bin/prted: error while loading shared libraries: libpbs.so.0: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
PRTE has lost communication with a remote daemon.

  HNP daemon   : [prterun-x8000c1s0b0n1-2021969@0,0] on node x8000c1s0b0n1
  Remote daemon: [prterun-x8000c1s0b0n1-2021969@0,1] on node x8000c1s0b1n0

This is usually due to either a failure of the TCP network
connection to the node, or possibly an internal failure of
the daemon itself. We cannot recover from this failure, and
therefore will terminate the job.
--------------------------------------------------------------------------

It appears that prted is correctly linked:

ldd /apps/utils/openmpi/5.0.6/gnu/11.2.1/bin/prted

	linux-vdso.so.1 (0x00007ffffefd2000)
	libprrte.so.3 => /apps/utils/openmpi/5.0.6/gnu/11.2.1/lib/libprrte.so.3 (0x000014f103e8e000)
	libpbs.so.0 => /opt/pbs/lib/libpbs.so.0 (0x000014f103afa000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x000014f1038da000)

... and the library is present:

find /opt/pbs/lib/ -iname 'libpbs.so.0'

	/opt/pbs/lib/libpbs.so.0

Additionally, I set PRTE_MCA_prte_fwd_environment to true. The aforementioned error disappears if I remove the flag you suggested.

Thanks again.

[rhc54]

You're missing a library on the backend, but that is pretty typical - they don't put the pbs libraries on the compute nodes. Reconfigure without the --with-tm lines and see if that works. If not, then try removing the --with-pbs as well. You may or may not need to forward the environment, but it won't hurt if you do.

[axiom-ctrl]

I attempted to reconfigure the installation with the following settings:

./configure \
    CC="gcc" \
    CXX="g++" \
    FC="gfortran" \
    --prefix=${install_dir} \
    --enable-shared \
    --enable-static \
    --with-libfabric="/opt/cray" \
    --with-libfabric-libdir="/opt/cray/lib64" \
    --without-pbs \
    --without-tm

This allowed me to move past the previous error.

---

Then I ran three identical jobs, each with 32 MPI processes distributed across 32 separate nodes. (PBS_NODEFILE contains 32 distinct lines, corresponding to 32 different compute nodes.)

mpirun --prtemca plm ssh -np 32 --hostfile $PBS_NODEFILE pw.x -i scf.input

What happens, is that the application reports:

     Parallel version (MPI & OpenMP), running on      32 processor cores
     Number of MPI processes:                32
     Threads/MPI process:                     1

     MPI processes distributed on     2 nodes

Indeed, when I SSH into one of the nodes where processes should be running (checking PBS_NODEFILE), I can observe that the prted processes are hanging and have failed to launch the pw.x executable.

---

If I attempt to omit the -np flag (while keeping --hostfile):

--------------------------------------------------------------------------
Open MPI failed an OFI Libfabric library call (fi_domain).  This is highly
unusual; your job may behave unpredictably (and/or abort) after this.

  Local host: x8000c0s7b0n1
  Location: mtl_ofi_component.c:1007
  Error: Invalid argument (22)
--------------------------------------------------------------------------

---

I am not sure if this is related to the way the mpirun is being invoked?

[hppritcha]

To help diagnose the last error message problem could you rerun with the following open mpi mca params set?

--mca pml ob1 --mca btl self,sm,tcp

Does this system have the HPE PALS package installed? Is there a aprun command in your path?

[rhc54]

Indeed, when I SSH into one of the nodes where processes should be running (checking PBS_NODEFILE), I can observe that the prted processes are hanging and have failed to launch the pw.x executable.

I suspect they aren't hanging, but rather don't think they have anything to do. You could also try adding --prtemca plm_base_verbose 5 to get some more debugging output - might want to add --enable-debug to your configure line first to activate some of the debugging messages.

[axiom-ctrl]

I suspect they aren't hanging, but rather don't think they have anything to do. You could also try adding --prtemca plm_base_verbose 5 to get some more debugging output - might want to add --enable-debug to your configure line first to activate some of the debugging messages.

I reconfigured the application with the --enable-debug option and ran it with additional verbose options. I have attached the output for your reference.

output.txt

---

Does this system have the HPE PALS package installed? Is there a aprun command in your path?

Yes, it does; however, it has not been loaded into the environment—neither during the compilation process nor when running the application.

To help diagnose the last error message problem could you rerun with the following open mpi mca params set?

--mca pml ob1 --mca btl self,sm,tcp

I tried adding the options, but unfortunately, I didn’t notice any difference in the output. As rhc54 mentioned, it seems as though some prted processes have nothing to do.

[rhc54]

Looked at your output - everything ran perfectly until you hit an error in the app:

[x8000c0s3b0n0:3126261] [prterun-x8000c0s3b0n0-3126261@0,0] plm:base:receive got update_proc_state for vpid 2 pid 3187523 state CALLED ABORT exit_code 5
[x8000c0s3b0n0:3126261] [prterun-x8000c0s3b0n0-3126261@0,0] plm:base:receive done processing commands
[x8000c0s3b0n0:3126261] [prterun-x8000c0s3b0n0-3126261@0,0] plm:base:orted_cmd sending kill_local_procs cmds
[x8000c0s5b0n1:00000] *** An error occurred in MPI_Comm_free
[x8000c0s5b0n1:00000] *** reported by process [2430205953,6]
[x8000c0s5b0n1:00000] *** on communicator MPI_COMM_WORLD
[x8000c0s5b0n1:00000] *** MPI_ERR_COMM: invalid communicator
[x8000c0s5b0n1:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[x8000c0s5b0n1:00000] ***    and MPI will try to terminate your MPI job as well)
[x8000c0s3b0n0:3126261] [prterun-x8000c0s3b0n0-3126261@0,0] plm:base:receive processing msg
[x8000c0s3b0n0:3126261] 

You might want to review the output - your app reported a bunch of stuff before it hit the error, but I have no idea what it should look like.

[rhc54]

I wonder if the problem here is that you are expecting a different distribution of processes across the nodes? The default mapping method for mpirun is by-core, meaning we will densely pack each node before moving to the next one. In this case, running 32 processes on nodes with 32 cores, the procs are all being mapped onto the first compute node in your allocation.

So yes - all the other nodes are lying there idle. They don't have any procs mapped to them.

If you want your procs to be distributed differently, you have to use the --map-by xxx option. You can check the help output or the online documentation for the valid options. For example, if you want to evenly distribute processes across the nodes, then you would use --map-by node.

[axiom-ctrl]

I wonder if the problem here is that you are expecting a different distribution of processes across the nodes?

That is very likely the root of the problem.

In our case, $PBS_NODEFILE is a newline-separated list of hostnames, which may or may not be symmetrical (meaning some entries might or might not be duplicated). My aim is to launch one MPI process for each entry in the file.

For instance, if $PBS_NODEFILE contains six entries (as shown below), I want to launch exactly six MPI processes, with one process corresponding to each entry:

node001   
node001  
node002    
node003
node003
node003

[rhc54]

You really should then read the help doc - specifically, look at prterun --help map-by to see the various modes. From what you describe, you probably want --map-by seq:file=$PBS_NODEFILE as per the doc:

* "SEQ" (often accompanied by the file=<path> qualifier) assigns one
  process to each node specified in the file. The sequential file is
  to contain an entry for each desired process, one per line of the
  file.

[axiom-ctrl]

I tried using the --host approach, which, at least for me, best illustrates what I want to achieve.

Running MPI on 4 nodes

---

When I request the following resources from the PBS service (4 distinct nodes; each with 1 MPI process and 1 CPU core):

#PBS -l select=4:mpiprocs=1:ncpus=1
#PBS -l place=scatter

which gives me:

PBS_NODEFILE:
x8000c1s3b0n1
x8000c0s2b1n0
x8000c0s3b1n0
x8000c0s4b1n0

and run the application in the following way:

mpirun --host x8000c1s3b0n1:1,x8000c0s2b1n0:1,x8000c0s3b1n0:1,x8000c0s4b1n0:1 --display map pw.x -i sb.vcr.in

not all 4 MPI processes start.

========================   JOB MAP   ========================
Data for JOB prterun-x8000c1s3b0n1-1250675@1 offset 0 Total slots allocated 4
    Mapping policy: BYCORE:NOOVERSUBSCRIBE  Ranking policy: FILL Binding policy: CORE:IF-SUPPORTED
    Cpu set: N/A  PPR: N/A  Cpus-per-rank: N/A  Cpu Type: CORE


Data for node: x8000c1s3b0n1    Num slots: 1    Max slots: 0    Num procs: 1
        Process jobid: prterun-x8000c1s3b0n1-1250675@1 App: 0 Process rank: 0 Bound: package[0][core:0]

Data for node: x8000c0s2b1n0    Num slots: 1    Max slots: 0    Num procs: 1
        Process jobid: prterun-x8000c1s3b0n1-1250675@1 App: 0 Process rank: 1 Bound: package[0][core:0]

Data for node: x8000c0s3b1n0    Num slots: 1    Max slots: 0    Num procs: 1
        Process jobid: prterun-x8000c1s3b0n1-1250675@1 App: 0 Process rank: 2 Bound: package[0][core:0]

=============================================================

This is further confirmed by the application's own output:

     Parallel version (MPI & OpenMP), running on       3 processor cores
     Number of MPI processes:                 3
     Threads/MPI process:                     1

     MPI processes distributed on     3 nodes
     148175 MiB available memory on the printing compute node when the environment starts

Running MPI on 1 node

---

When I request the following resources from the PBS service (1 node; with 4 MPI processes and 4 CPU cores):

#PBS -l select=4:mpiprocs=1:ncpus=1
#PBS -l place=pack

which gives me:

PBS_NODEFILE:
x8000c1s3b0n1
x8000c1s3b0n1
x8000c1s3b0n1
x8000c1s3b0n1

and run the application in the following way:

mpirun --host x8000c1s3b0n1:4 --display map pw.x -i sb.vcr.in

I immediately encounter the following error:

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 1
slots that were requested by the application:

  pw.x

Either request fewer procs for your application, or make more slots
available for use.

Running hybrid MPI+SMP

---

What I ultimately aim to achieve is running hybrid Open MPI + OpenMP parallel applications. When I request the following resources from the PBS service (4 distinct nodes, each with 1 MPI process and 4 CPU core):

#PBS -l select=4:mpiprocs=1:ncpus=4
#PBS -l place=scatter

which gives me:

PBS_NODEFILE:
x8000c1s3b0n1
x8000c1s2b0n0
x8000c0s2b1n0
x8000c0s3b1n0

and run the application in the following way:

mpirun --host x8000c1s3b0n1:1,x8000c1s2b0n0:1,x8000c0s2b1n0:1,x8000c0s3b1n0:1 --map-by slot:PE=4 --bind-to core --display map pw.x -i sb.vcr.in

I receive the following job map and error:

========================   JOB MAP   ========================
Data for JOB prterun-x8000c1s3b0n1-1251216@1 offset 0 Total slots allocated 4
    Mapping policy: BYSLOT:NOOVERSUBSCRIBE  Ranking policy: SLOT Binding policy: CORE
    Cpu set: N/A  PPR: N/A  Cpus-per-rank: 4  Cpu Type: CORE


Data for node: x8000c1s3b0n1    Num slots: 1    Max slots: 0    Num procs: 0

Data for node: x8000c1s2b0n0    Num slots: 1    Max slots: 0    Num procs: 1
        Process jobid: prterun-x8000c1s3b0n1-1251216@1 App: 0 Process rank: 0 Bound: package[0][core:0-3]

=============================================================
[x8000c1s3b0n1:1251216] PMIX ERROR: PMIX_ERR_BAD_PARAM in file gds_utils.c at line 308
[x8000c1s3b0n1:1251216] PMIX ERROR: PMIX_ERR_BAD_PARAM in file gds_hash.c at line 496
[x8000c1s3b0n1:1251216] PRTE ERROR: Bad parameter in file base/odls_base_default_fns.c at line 768

[rhc54]

I can look at the cmd line parsing issue - it appears that the : qualifiers on --host may be causing difficulties. However, at least from what you have previously explained, the --host approach is not going to achieve what you said you wanted. All that option does is specify the allocation - it has absolutely no impact on per-process placement. The sequential and rankfile mappers are the only way to manually specify exact proc placement.

FWIW: this option (--map-by slot:PE=4) will always force --bind-to core as there is no other way to constrain the process to a specific number of CPUs.

And just to clarify: this was all done with the pbs support configured out of OMPI, yes? The only thing OMPI sees is the nodes (and number of slots on each node) you are passing us - you might as well just ask PBS to assign N nodes and be done with it.

[rhc54]

Starting to work through this with your most basic report (the next to last one in your list) - and cannot reproduce.

mpirun --host rhc-node01:4 --display map hostname

========================   JOB MAP   ========================
Data for JOB prterun-rhc-node01-41187@1 offset 0 Total slots allocated 4
    Mapping policy: BYCORE:NOOVERSUBSCRIBE  Ranking policy: FILL Binding policy: CORE:IF-SUPPORTED
    Cpu set: N/A  PPR: N/A  Cpus-per-rank: N/A  Cpu Type: CORE


Data for node: rhc-node01	Num slots: 4	Max slots: 0	Num procs: 4
        Process jobid: prterun-rhc-node01-41187@1 App: 0 Process rank: 0 Bound: package[0][core:0]
        Process jobid: prterun-rhc-node01-41187@1 App: 0 Process rank: 1 Bound: package[0][core:1]
        Process jobid: prterun-rhc-node01-41187@1 App: 0 Process rank: 2 Bound: package[0][core:2]
        Process jobid: prterun-rhc-node01-41187@1 App: 0 Process rank: 3 Bound: package[0][core:3]

=============================================================
rhc-node01
rhc-node01
rhc-node01
rhc-node01

Of course, I'm working with the head of the PMIx/PRRTE master branches, so that might show a problem that has already been fixed. 🤷‍♂️ Or it could be something in your configuration/environment. I'll give the current head of OMPI's v5.0 branch a peek and see if it also works.