mpirun exiting basic program with Signal 27 (Profiling timer expired)

[dsatagaj]

Please submit all the information below so that we can understand the working environment that is the context for your question.

• If you have a problem building or installing Open MPI, be sure to read this.
• If you have a problem launching MPI or OpenSHMEM applications, be sure to read this.
• If you have a problem running MPI or OpenSHMEM applications (i.e., after launching them), be sure to read this.

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.5 (was previously using v4.1.6 with no issues)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Source/Distribution tarball

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: RHEL8
• Computer hardware: Intel XEON(R) Gold 6254
• Network type: ethernet

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

I am upgrading from openmpi-4.1.6 to openmpi-5.0.5. My compilation is successful and I can run the tools, however all tools throw signal 27 (Profiling timer expired), even when running basic test programs. For example, see the output of a simple mpirun command for hostname:

dsat@dsat.org ~/openmpi-5.0.5]$ mpirun -np 4 hostname
dsat.org
dsat.org
------------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 1202879 on node dsat exited on
signal 27 (Profiling timer expired).
------------------------------------------------------------------------------

mpirun built with openmpi-4.1.6 can successfully run this (and my own) programs.
I was configuring openmpi-4.1.6 with the following parameters:

./configure --disable-shared --enable-static --without-memory-manager --with-hwloc=internal --disable-io-romio --disable-hwloc-pci --without-verbs --enable-mpi-thread-multiple --enable-heterogenous --enable-mpi-cxx --enable-cxx-exceptions --enable-mpi-fortran=none

This is what I am attempting to configure openmpi-5.0.5 with:

./configure --disable-shared --enable-static --without-memory-manager --with-hwloc=internal --with-libevent=internal --with-pmix=internal --with-prrte=internal --disable-io-romio --disable-mpi-fortran

I have also attempted configuration with a few different options in an attempt to solve this problem (with no success):

./configure --disable-shared --without-ft --enable-static --without-memory-manager --with-hwloc=internal --with-libevent=internal --with-pmix=internal --with-prrte=internal --disable-io-romio --disable-mpi-fortran --without-ucx --without-libfabric

I have confirmed that there is no profiling going on in the system that would be causing this SIGPROF to be sent to the child processes. That combined with the test that mpirun compiled with openmpi-4.1.6 runs successfully demonstrates to me that the issue is something internal to openmpi-5.0.5, whether that be a bug or a configuration issue.

In order to debug this problem, I inserted the following patch into odls_default_module.c:

--- /openmpi-5.0.5/3rd-party/prrte/src/mca/odls/default/odls_default_module.c.orig 2025-11-06 09:11:24.00000000 -0500
+++ /openmpi-5.0.5/3rd-party/prrte/src/mca/odls/default/odls_default_module.c 2025-11-06 09:13:28.00000000 -0500

@@ -138,6 +138,15 @@
#include "src/mca/odls/default/odls_default.h"
#include "src/prted/pmix/pmix_server.h"

+#include <stdio.h>
+#include <stdlib.h>
+#include <signal.h>
+#include <sys/time.h>
+#include <unistd.h>
+#ifdef __GLIBC__
+extern char *program_invocation_short_name;
+#endif
+
 /*
  * Module functions (function pointers used in a struct)
  */
@@ -631,6 +640,26 @@
     if (pid == 0) {
         close(p[0]);
+        /* --- DSAT DEBUG/FIX - print and clear inherited ITIMER_PROF --- */
+        const char *dsat_env = getenv("DSAT_ITIMER_PATCH");
+        if (dsat_env != NULL) {
+#ifdef ITIMER_PROF
+           struct itimerval t;
+           getitimer(ITIMER_PROF, &t);
+           struct sigaction sa;
+           sigaction(SIGPROF, NULL, &sa);
+           fprintf(stderr, "PID: %d, PPID: %d, executable: %s, inherited ITIMER_PROF: value [sec=%ld usec=%ld] interval [sec=%ld usec=%ld], SIGPROF handler=%p, flags=%d\n", getpid(), getppid(), program_invocation_short_name, t.it_value.tv_sec, t.it_value.tv_usec, t.it_interval.tv_sec, t.it_interval.tv_usec, sa.sa_handler, sa.sa_flags);
+           /* clear the timer so SIGPROF does not fire */
+           struct itimerval zero = {0};
+           setitimer(ITIMER_PROF, &zero, NULL);+
+
+          /* ignore SIGPROF to be safe */
+          signal(SIGPROF, SIG_IGN);
+#endif
+        }
+        /* END DSAT DEBUG/FIX PATCH --- */
        do_child(cd, p[1]);
         /*Does not return */
     }

With this patch included, the program completes successfully (e.g. ignores the SIGPROF). I get the following output from the same the command 'env DSAT_ITIMER_PATCH=1 mpirun -np 4 hostname':

dsat@dsat.org ~/openmpi-5.0.5]$ env DSAT_ITIMER_PATCH=1 mpirun -np 4 hostname
PID= 1207942: PPID: 1207939: executable: prterun: inherited ITIMER_PROF: value [sec=0 usec=0] interval [sec=0 usec=0], SIGPROF handler=0x6c3d60, flags=335544320
PID= 1207943: PPID: 1207939: executable: prterun: inherited ITIMER_PROF: value [sec=0 usec=0] interval [sec=0 usec=0], SIGPROF handler=0x6c3d60, flags=335544320
PID= 1207944: PPID: 1207939: executable: prterun: inherited ITIMER_PROF: value [sec=0 usec=0] interval [sec=0 usec=0], SIGPROF handler=0x6c3d60, flags=335544320
PID= 1207945: PPID: 1207939: executable: prterun: inherited ITIMER_PROF: value [sec=0 usec=0] interval [sec=0 usec=0], SIGPROF handler=0x6c3d60, flags=335544320
dsat.org
dsat.org
dsat.org
dsat.org

I took it one step farther and ran the program using GDB. I get similar output, and when I inspect the SIGPROF handler value, I see that it comes from the following location:

(gdb) info symbol 0x6c3d60
evsig_handler in section .text of /home/dsat/openmpi-5.0.5/build/openmpi/bin/prte

The summary of my investigation is this: It seems that libevent or prte is using profiling timers internally for timing. Somehow, the handler is not correctly getting cleared for the child processes, and is therefore catching, even during simple programs. Does anyone have insight into what is going on, and if it is a bug, or if there are configuration options that could be used to fix it?

[dsatagaj]

Its worth noting that I am compiling for multiple compilers: gcc 14.3.0 & the Intel Compiler icx 2025.2.1. In both mpirun binaries, I get the same error.

[bosilca]

This is where we trap the all signals and react to them so as far as I can see that part is normal. The real question is why is that signal generated and who asked for it (aka who calls setitimer on behalf of the process) ?

[rhc54]

Correct - mpirun in OMPI v5 automatically traps and forwards all system-defined signals. We didn't do that in v4 except for a couple of them. So something external to mpirun is generating your SIGPROF - we are just forwarding it along.

If you want to turn that forwarding off, you can pass the following MCA param to mpirun:

--prtemca ess_base_forward_signals <value>

where the value is:

Comma-delimited list of additional signals (names or integers) to forward to
application processes [\"none\" => forward nothing]. Signals provided by
default usually include SIGTSTP, SIGUSR1, SIGUSR2, SIGABRT, SIGALRM, and SIGCONT
depending upon system definitions

[dsatagaj]

@bosilca @rhc54 thank you for responding. I tried adding the ess_base_forward_signals parameter as you suggested, but did not see improvement. I tried with 3 different commands (to check for syntax issues). See the commands:

dsat@dsat.org ~/openmpi-5.0.5]$ mpirun --prtemca ess_base_forward_signals 27 -np 4 hostname
dsat@dsat.org ~/openmpi-5.0.5]$ mpirun --prtemca ess_base_forward_signals SIGPROF -np 4 hostname
dsat@dsat.org ~/openmpi-5.0.5]$ mpirun --prtemca ess_base_forward_signals "SIGPROF" -np 4 hostname

All failed with similar output:

dsat@dsat.org ~/openmpi-5.0.5]$ mpirun --prtemca ess_base_forward_signals 27 -np 4 hostname
dsat.org
dsat.org
------------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 1202879 on node dsat exited on
signal 27 (Profiling timer expired).
------------------------------------------------------------------------------

Any thoughts on why it might still fail?

I will continue looking for what might be sending SIGPROFs to all programs. I didn't see any profilers in my system but there are other users, so it is possible there is something I'm missing.

[rhc54]

This command option:

--prtemca ess_base_forward_signals SIGPROF

is telling mpirun to FORWARD the signal - exactly the opposite of what you wanted. Use none for the value, or some subset of signals that you actually want to have forwarded.

[dsatagaj]

@rhc54 thank you for the clarification. I still see the same behavior though, even with "--prtemca ess_base_forward_signals none". I also tried with "none" quoted

I added "--prtemca ess_base_verbose 10" to try and get more info, and didn't see anything relevant in the output.

[rhc54]

No ideas - could be that whatever is generating it is delivering that signal to your application procs directly, regardless of mpirun forwarding or not. Afraid you need to find the root cause of who is generating the signal. Don't really know anything about your system - sometimes you can have the resource manager (if you have one) generate such things via some option in your allocation request. Just a pure WAG.

[ggouaillardet]

FWIW

mpirun -np 4 strace -e 'trace=!all' hostname

Should tell you which pid sent the signal.

[dsatagaj]

Thank you guys for your insights. I ended up figuring out the issue. I was building three types of builds - including a profiling and debug build. In openmpi-4.1.6, I was depending on the "--enable-binaries=no" option for the profile and debug builds in order to not generate competing binaries, however it appears in openmpi-5.0.5, while that option works for the OMPI layer, it does not propagate down to PRRTE and PMIX. The issue was that profile binaries for prterun were overwriting the release ones, causing the profiling issues mentioned above. I didn't notice the issue until recently since ompi_info was the correct binary and didn't report any profiling flags or similar. After changing my personal build scripts to put profile and debug builds in a separate install directory (allowing my main directory to house the release builds for all binaries), I was able to confirm everything works correctly. Thank you for all the support!