MPI Errhandler causes segfault on receipt of signal

[rhc54]

In trying to track down an apparent "hang" on the 1.10.3 branch, I discovered that the job wasn't actually hanging. Instead, the job was taking a long time to complete because nearly every proc was segfaulting and writing out a core dump - and it was taking so long to write out all those core dumps that MTT thought we had hung.

The test is Intel MPI_Errhandler_fatal1_f (a few others show the same symptom - this is just the one I was working with). Everything starts correctly, and then rank=0 calls MPI_Abort. ORTE correctly identifies the situation and issues "kill" signals to the other procs, starting with SIGCONT followed by SIGTERM, and then SIGKILL.

The output results look okay:

$ mpirun -n 16 ./MPI_Errhandler_fatal1_f
 MPITEST_INFO (         0): Starting test MPI_Errhandler_fatal     
 MPITEST_INFO (         0): This test should abort after printing the results
 MPITEST_INFO (         0): message, otherwise a f.a.i.l.u.r.e is noted
MPITEST_results: MPI_Errhandler_fatal all tests PASSED (        16)
[bend001:21445] *** An error occurred in MPI_Send
[bend001:21445] *** reported by process [139981018365953,21474836480]
[bend001:21445] *** on communicator MPI COMMUNICATOR 3 DUP FROM 0
[bend001:21445] *** MPI_ERR_RANK: invalid rank
[bend001:21445] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[bend001:21445] ***    and potentially your MPI job)

However, the writing out of the core files takes a long time, and the shell prompt doesn't return until that dump is completed. Thus, MTT thinks the job timed out and tries to kill it - and fails, because the mpirun is actually already dead.

I have no idea why this test is segfaulting upon receipt of a signal. Can someone please investigate?

@ggouaillardet @bosilca @hjelmn @jsquyres

Not sure which of you might have time and knowledge to dig this one out.

[rhc54]

Should have provided this - here is the MTT complaint:

>> Command: mpirun     --hostfile /home/common/hosts -np 16 --prefix /home/common/openmpi/build/ompi-release/ src/MPI_Errhandler_fatal1_f 
   Elapsed:       00:00:10 0.00u 0.01s
*** WARNING: Test: MPI_Errhandler_fatal1_f, np=16, variant=1: TIMED OUT
    (failed)
*** Past timeout of 2 seconds by 1 seconds
*** Killing mpirun with SIGTERM
** Kill TERM (after 5 seconds) didn't work!
*** Killing mpirun with SIGINT
** Kill INT (after 5 seconds) didn't work!
*** Now I'm mad.  Killing mpirun with SIGKILL
** Kill KILL didn't work!  Sleeping and trying again...
*** Past drain timeout; quitting

[rhc54]

Final tidbit - when I gdb the core dump, all I can see is that the proc was sitting in a signal handler, and that the rest of the stack was corrupted.

[ggouaillardet]

@rhc54 i was able to reproduce it on one system, and the problem went away with

export IPATH_NO_BACKTRACE=1

could you please give it a try ?

libpsm_infinipath.so sets its own signal handler when mca_mtl_psm.so is dlopen'ed, and it overwrites the OpenMPI signal handler.
if you built with --disable-dlopen i am not sure which signal handler is set first, nor whether this is a consistent behavior across all systems.

this issue was already discussed at https://www.open-mpi.org/community/lists/devel/2015/08/17838.php (loooong thread) and at that time, it was noticed only when using Java

[jsquyres]

When I run this, I end up in opal/mca/pmix/pmix120/pmix/src/client/pmix_client.c in PMIx_Abort.

It calls PMIX_ACTIVATE_SEND_RECV on line 549, and then returns. This seems to be a race condition, because the orted will kill the process "right away" -- but perhaps not before the call to PMIx_Abort returns and unwinds the stack into areas that are not expecting a return.

Shouldn't PMIx_Abort guarantee to not return?

[rhc54]

@jsquyres I'm scratching my head over your comment and failing to grok it. One process calls "abort", and we correctly handle that call and it cleanly exits. The other processes (that registered errhandlers) are the ones that are segfaulting. Are you saying that those procs are also calling "abort" upon receiving a signal? If so, then that would indeed be a problem.

[jsquyres]

@rhc54 and I talked about this on the phone:

1. ORTE's abort is catching the return from PMIx_Abort() and is calling exit(). So we're good there.
2. I looped over calling MPI_Errhandler_fatal1_f and am unable to replicate the error. 😦

@rhc54 is going to keep investigating...

[ggouaillardet]

@rhc54 did you confirm this issue is not related to libpsm_infinipath.so that sets its signal handler when dlopen-ed and is then unmapped because there is no PSM hardware ?

[rhc54]

I don't have psm support on my system, nor are any of those components built. I can't swear that isn't the problem - maybe some other component is opening something that can cause the problem. For example, I do have libfabric installed, and the usnic btl is opening it, so perhaps this problem is creeping in that way?

[ggouaillardet]

iirc, i ran the application as a singleton under gdb, set a breakpoint in signal and sigaction in order to find who sets signal handlers.
a simpler solution would have been to call sigaction and then use dladdr after MPI_Init but i could not figure out how to get dladdr work ...

[jsquyres]

usnic doesn't set any signal handlers.

Does your libfabric had libpsm(2) support? You should be able to run fi_info --list to see what providers your libfabric has.

[hiliev]

Same problem here. libpsm_infinipath.so is setting a signal handler for SIGINT while the psm MTL is dlopen()-ed and then not restoring the previous handler on dlclose(). Ctrl-C then results in a call right into the middle of an unmapped memory region. It took me quite some staring at the stack trace before I realised what is going on and my initial assumption was that the stack frame was somehow getting corrupted. We don't have any InfiniPath hardware on our cluster, but the library is installed anyway and the build system picks it up and builds the MTL. Our solution was to disable the psm MTL.

[matcabral]

@rhc54 is saying he's build does not have PSM support. Please confirm.
If PSM is actually in, as @ggouaillardet mentioned, using IPATH_NO_BACKTRACE=1 will disable libpsm_infinipath.so playing with sighandlers. There was an issue related to psm hijacking sighandlers and JVM. This was fixed in PSM_VERNO 0x0110. Please look at:

intel/psm@225bbc9

I've been thinking on adding a sanity check on the psm mtl to warn or fail when psm version is older than 0x0110.

[rhc54]

I can confirm that I do not have the PSM support built on my machine. However, I do have the OFI MTL component built, and have libfabric on the machine. Thus, even though I lack the hardware and have not built the PSM MTL, I fear that the libraries may be getting opened via the libfabric provider.

If that is true, then this problem could hit everyone who installs libfabric, even if they don't have supporting hardware for it, as the OFI MTL will always build if that library is present.