v1.10.1: mpirun does not warn about oversubscription

[ammore1]

When I mpirun with more processes than cores with Open MPI v1.8.4, I get the following warning:

$ mpirun -V
 mpirun (Open MPI) 1.8.4cisco1.0.1.367.sles12sp0
$ mpirun --host <HOSTNAME1,HOSTNAME2> -np 36 hello_c 
---------------------------------------------------- ----------------------
A request was made to bind to that would result in binding more
processes than cpus on a resource:

   Bind to:     NONE
   Node:        usnic3
   #processes:  2
   #cpus:       1

You can override this protection by adding the "overload-allowed"
option to your binding directive.
--------------------------------------------------------------------------

But when I mpirun with more processes than cores with Open MPI v1.10, I do not get a warning:

$ mpirun -V
mpirun (Open MPI) 1.10.1cisco1.0.0.387.sles12sp0
$ mpirun --host <HOSTNAME1,HOSTNAME2> -np 36 hello_c 
Hello, world, I am 18 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 19 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 20 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 21 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 22 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 23 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 24 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 27 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 25 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 28 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 26 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 31 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 32 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 34 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 30 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 33 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 35 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 29 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 5 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 6 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 7 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 8 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 9 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 10 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 11 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 12 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 14 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 13 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 15 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 16 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 17 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 0 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 1 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 3 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 4 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)
Hello, world, I am 2 of 36, (Open MPI v1.10.1cisco1.0.0.387.sles12sp0, package: Cisco Open MPI, ident: 1.10.1cisco1.0.0.387, repo rev: git_ad777cc, Sat Sep 19 12:05:03 2015 PDT, 148)

Is this intended behaviour, or is it a regression?

This was found during Cisco QA testing.

@jsquyres @rhc54

[rhc54]

I obviously don't know what default MCA parameters may be getting set in the Cisco version of OMPI, so it's a little difficult to comment. I'm a little suspicious because the error you show for 1.8 is about binding, not oversubscription. I also don't know what environment you are executing in - is it a managed environment (e.g., Slurm), or an unmanaged one where you have a hostfile and/or simply specify the hosts on the cmd line?

The behavior of mpirun depends on all those factors. Can you provide some info on them?

[ammore1]

I am not using a SLURM job.I am running a simple C application "hello_C" using mpirun and simply specifying the hosts from commandline.
The command I am using is "mpirun --host <HOSTNAME1,HOSTNAME2> -np 36 hello_c".
The command "mpirun --host <HOSTNAME1,HOSTNAME2> -np 36 hostname" also gives the same behaviour.

I get an error when I run more processes than total number of cpu cores in ompi 1.8.4, but not in 1.10.1.

[rhc54]

@ammore1 Please try the following patch:

open-mpi/ompi-release#664

[ammore1]

@rhc54

I tried this patch. But with this patch I am able to run maximum 2 processes between 2 hosts which does not seem to be correct .It should allow maximum 32 processes to run since there are 32 number of cores on each of my server.

$ mpirun -host <host1,host2> -np  2 hostname
usnic2
usnic3
$ mpirun -host <host1,host2> -np  3 hostname
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 3 slots that were requested by the application:
  hostname

Either request fewer slots for your application, or make more slots available for use.
--------------------------------------------------------------------------

$ nproc
32

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45
Model name:            Intel(R) Xeon(R) CPU E5-2470 0 @ 2.30GHz
Stepping:              7
CPU MHz:               1520.335
CPU max MHz:           3100.0000
CPU min MHz:           1200.0000
BogoMIPS:              4601.53
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0-7,16-23
NUMA node1 CPU(s):     8-15,24-31

[jsquyres]

Per http://www.open-mpi.org/community/lists/devel/2015/11/18354.php, this is not a critical bug fix. Pushing to v1.10.3.

[rhc54]

As commented on open-mpi/ompi-release#664, the identified behavior is indeed the specified one, so this issue has been corrected.