Cisco bad MTT results for v3.1.x

[jsquyres]

Per discussion on 2018-03-20 webex, Cisco is seeing a large number of failures in v3.1.x testing that it is not seeing in v3.0.x testing.

Need to investigate what all these failures are.

@bwbarrett asked me to file this as a blocker.

[ggouaillardet]

@jsquyres note you can (likely) ignore the spawn_argv* failures. Your MTT config should be updated to start on a smaller number of MPI tasks, or use the --oversubscribe option.

[ggouaillardet]

some errors are related to #4788
I could not read a clear reply from the MPI Forum, so how should we move forward ?

[jsquyres]

There are 3 classes of errors in the Cisco MTT results:

1. Some local / config issues. I've fixed most of these. There's still some segv's coming from --disable-mpi-params runs; I might end up disabling them (because some tests intentionally pass bad params to MPI API calls, which, when we have param checking disabled, tend to [rightfully] end up as segvs or other monsters).
2. Some legit errors. These seem to be pretty small in number.
3. Periodically, the error below shows up. Sometimes it only happens a few times in a given MTT run, sometimes it happens a lot. I've seen this on master, v3.0, and v3.1 MTT runs. @rhc54's gut feeling is that it may be a race condition in the OOB. I've cranked up OOB verbosity in my MTT (as of ~4pm US Eastern, Saturday, 7 Apr). Let's see if that turns up anything new/interesting.

--------------------------------------------------------------------------
An ORTE daemon has unexpectedly failed after launch and before
communicating back to mpirun. This could be caused by a number
of factors, including an inability to create a connection back
to mpirun due to a lack of common network interfaces and/or no
route found between them. Please check network connectivity
(including firewalls and network routing requirements).
--------------------------------------------------------------------------

Sometimes I get a few, sometimes I get a lot.

[bwbarrett]

@jsquyres any update?

[jsquyres]

Updates:

1. This is happening on master, v3.0.x, and v3.1.x.
2. This is happening inside a slurm allocation. The head node (where mpirun is running) in my example below is savbu-usnic-a.cisco.com, and the compute nodes are mpi005 and mpi006.
3. I put a strategic sleep(99999) in MTT to not kill mpirun when it times out, and I attached a gdb to mpirun when this happens. I can see that mpirun is just blocking in poll() inside libevent. I.e., it's not even spinning on calling libevent -- it's blocking in poll(). Because nothing is happening on libevent fd's. I.e., it's rightfully blocking. Poking around, I see that mpirun is still running. All MPI processes and orteds are dead. ...but the srun child of mpirun still exists -- I don't know why. 😦 "ps" on the node where mpirun shows these 2 procs:

mpiteam  21545 13786  0 10:49 pts/39   00:00:00 mpirun --mca plm_base_verbose 100 -np 32 --mca orte_startup_timeout 10000 --mca oob tcp --mca btl tcp,self --mca mpi_leave_pinned 1 ./hello_usempi

mpiteam  21579 21545  0 10:49 pts/39   00:00:00 srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --ntasks=2 orted -mca ess "slurm" -mca ess_base_jobid "86966272" -mca ess_base_vpid "1" -mca ess_base_num_procs "3" -mca orte_node_regex "savbu-usnic-a,mpi[3:5-6]@0(3)" -mca orte_hnp_uri "86966272.0;tcp://10.0.8.254,10.193.184.48,10.193.184.49,10.10.0.254,10.2.0.254,10.3.0.254,10.3.0.252,10.50.0.254:55040" --mca plm_base_verbose "100" --mca orte_startup_timeout "10000" --mca oob "tcp" --mca btl "tcp,self" --mca mpi_leave_pinned "1"

1. I don't know why srun is still running. Looking in /proc/PID_OF_MPIRUN/fd, I see 3 sockets open (for fd 0, 1, 2), which I think is expected. But I'm befuddled as to why srun would hang like this. I haven't touched the slurm config in forever.
2. In further explorations, I found:
   1. I can get this to happen even with an "AlreadyInstalled" OMPI installation (i.e., not just an OMPI that was compiled by MTT).
   2. But still, I have to run under MTT: if I run a) with the exact same OMPI install, and b) the exact same MPI app binary, it only seems to hang if I run the app binary under MTT.
   3. I do notice that MTT only hangs if I use --mca plm_base_verbose 100 or --leave-session-attached --bind=none. Ralph points out to me that in both of these cases, we do not redirect stdout/stderr to /dev/null -- that might have something to do with what is going on here.
   4. Ralph also has a dim recollection of us having an mpirun bug that caused hangs eons ago that was a race condition having something to do with fd's that weren't closed properly (or in the right order?). mpirun would end up waiting forever for something that either wouldn't happen or had already happened. We wonder if SLURM 14.03 srun (which is the version of SLURM on my cluster) has a similar race condition.

All this being said, we added --leave-session-attached and --mca plm_base_verbose 100 to my MTT config because we were seeing some hangs on my cluster. Adding either of those options changes the situation from "some jobs hang" to "all jobs hang".

So I think all I can do is remove the --leave-session-attached and --mca plm_base_verbose 100 options from my MTT config, get at least some/most jobs to run successfully, and try to debug the "some processes" hang situation. Note: I never caught a job from the "some processes hang" case in an interactive session (or any other situation where I could attach a debugger, see what was still running, etc.) -- so we don't know if it's the same SLURM issue that we're seeing with --leave-session-attach / --mca plm_base_verbose 100 or not.

[jsquyres]

@bwbarrett @hppritcha I assume you are using versions of SLURM significantly more recent than 14.03. Can you confirm that running MTT with mpirun in a SLURM allocation does not hang if you --leave-session-attached or --mca plm_base_verbose 100?

I ask because I'm becoming convinced that this particular issue (w.r.t. --leave-session-attached and --mca plm_base_verbose 100) is a SLURM bug, and I'm wondering if it's fixed in newer versions of SLURM.

Specifically:

• If you neither --leave-session-attached nor --mca plm_base_verbose 100, the SLURM plm will dup2() an fd for /dev/null into fd's 1 and 2 after the fork() but before the exec() of srun.
  • In this case, everything runs fine.
• If you either --leave-session-attached or --mca plm_base_verbose 100, the SLURM plm will do nothing to fd's 1 and 2.
  • In this case, srun hangs (and therefore mpirun hangs).
• I have found that if I manually comment out the dup2()s into fd's 1 and 2 (which effectively emulates that part of --leave-session-attached or --mca plm_base_verbose 100)
  • In this case, srun hangs (and therefore mpirun hangs).

This all seems to indicate a bug in SLURM v14.03 srun. I'm wondering a) if it is a real bug in srun, and b) if it has been fixed since 14.0.3.

[ggouaillardet]

@jsquyres i was not able to reproduce the issue with slurm 17.02.1

I ran salloc on the head node, and it allocated two distinct compute nodes.
Then I ran MTT with the trivial test suite and the mentionned options, and no hang occured.

BTW, do you run MTT from a terminal or via a cron ?

[jsquyres]

@ggouaillardet I normally run MTT via cron (i.e., every night, with lots of sbatch jobs). However, I am able to replicate this problem even when salloc'ing an allocation and running MTT manually in a terminal.

This is probably the best we can do with the --leave-session-attached and --mca plm_base_verbose 100 options: there's likely some bug in this ancient version of SLURM. So I'll just stay away from those options until I can upgrade my cluster.

In the meantime, let's get back to trying to identify / debug the real issue that this started with: the sometimes job hangs on the Cisco cluster...

[bwbarrett]

Per discussions with @jsquyres , removing this from the blocker list for 3.1.