Processing element assignment worked in OpenMPI 4.1.4 but now the same command fails in 5.0.5

[johnmoore4]

We have an application where we launch a single process per node, but want to use all the available threads (processing elements) available on a dual-socket machine in that single process. Our application uses OpenMP within each process for shared memory parallelism, and we want to run OpenMP across the two sockets.

In OpenMPI 4.0.4, we had been using the command and it runs fine:

/openmpi-4.1.4/bin/mpirun -np 1 --report-bindings  --map-by ppr:1:node:PE=112 ls
[b200-003:1981388] MCW rank 0 is not bound (or bound to all available processors)

However, in OpenMPI 5.0.5 (and 5.0.7), we get this behavior:

/openmpi-5.0.5/bin/mpirun -np 1 --report-bindings --map-by ppr:1:node:PE=112 ls
--------------------------------------------------------------------------
Your job failed to map because the resulting process placement
would cause the process to be bound to CPUs in more than one
package:

  Mapping policy:  BYNODE:NOOVERSUBSCRIBE
  Binding policy:  CORE:IF-SUPPORTED
  CPUs/rank:       112

This configuration almost always results in a loss of performance
that can significantly impact applications. Either alter the
mapping, binding, and/or cpus/rank policies so that each process
can fit into a single package, or consider using an alternative
mapper that can handle this configuration (e.g., the rankfile mapper).
--------------------------------------------------------------------------

Has the behavior changed here between MPI 5.x and 4.x? If so, how should I map my threads in OpenMPI 5.0.5 to assign all the threads in the node to a single rank?

[rhc54]

Well, the warning is legit - you will see a performance loss due to running one process across multiple packages (or sockets, if you prefer) on each node as the memory is disjoint. However, you can still do it. Simply tell it:

$ mpirun --map-by ppr:1:node --bind-to none ...

After all, you are not binding the process if you are having it run across all CPUs on the node. OMPI v4.x simply allowed the performance hit without the warning/error, which resulted in complaints due to unexpected performance losses.

[johnmoore4]

@rhc54 Thank you for the assistance. I assumed we still needed to specify ppr:1:node:PE=112, but it was complaining that PE cannot be set when not --bind-to is not set to core.

[rhc54]

Correct - you cannot tell it to bind you to 112 PE's because that doesn't make any sense when there are only 112 PE's on the node. Binding to everything is the same as binding to nothing (which is what OMPI v4 was telling you). In this case, you need to bind to nothing because you are running cross-package, thereby indicating that you understand and accept the performance hit (i.e., it isn't our fault 😄 )