One or two ranks implicitly seems to do -bind-to core

[jeckstei]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

4.1.2

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Installed with apt-get

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: Ubuntu 22.04.5 LTS
• Computer hardware: AMD Ryzen Threadripper PRO 5995WX 64-Cores (SuperMicro motherboard)
• Network type: Ethernet, but these runs were just using internal SMP parallelism

---

Details of the problem

I am prototyping an application combining MPI with C++ native multithreading. Currently I'm doing this on my big workstation, although eventually the application will move to an HPC platform.

When I use mpirun with 1 or 2 MPI ranks, the app behaves as if I specified "-bind-to core" -- that is, each rank runs as if only two of its threads can be running at any given time. When I specify 3 or more MPI ranks, then every rank runs as if I specified "-bind-to none", that is, each rank can spawn as many threads as it wants and they run simultaneously up to the hardware limits of the chip (128 total threads).

I can obtain better performance and the expected behavior for 1 or 2 ranks if I manually specify "-bind-to none", but these seem like strange defaults. Is there some legitimate reason for the behavior I have described?

[rhc54]

It's in the documentation, section 17.1.2.2 (https://docs.open-mpi.org/en/v5.0.x/man-openmpi/man1/mpirun.1.html#quick-summary):

Please note that mpirun automatically binds processes
to hardware resources. Three binding patterns are used
in the absence of any further directives (See
[map/rank/bind defaults](https://docs.open-mpi.org/en/v5.0.x/man-openmpi/man1/mpirun.1.html#man1-mpirun-map-rank-bind-defaults)
for more details):

    Bind to core: when the number of processes is <= 2

    Bind to package: when the number of processes is > 2

    Bind to none: when oversubscribed

If your application uses threads, then you probably
want to ensure that you are either not bound at all
(by specifying --bind-to none), or bound to multiple cores
using an appropriate binding level or specific number of
processing elements per application process.

Justification is simple:

• bind-to core for num procs <= 2 was established to better support the traditional benchmarks (which are single-threaded)
• "Bind-to package" when > 2 is the better option in general, and definitely for threaded apps.
• "Bind-to none" is not a good option in general (including for threaded apps) as the process can then bounce between packages (or have threads spread between packages), which yields poorer performance. However, it tends to help when there are not enough resources to support the number of procs

[jsquyres]

@jeckstei I assume this answers your question. Feel free to re-open / ask further followups if needed.