Ansys fluent 2023.1 failed to run with Open MPI 5.0.x

[wzamazon]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v5.0.x branch

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

built from source

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

 7f6f8db13b42916b27b690b8a3f9e2757ec1417f 3rd-party/openpmix (v4.2.3-8-g7f6f8db1)
 c7b2c715f92495637c298249deb5493e86864ac8 3rd-party/prrte (v3.0.1rc1-36-gc7b2c715f9)
 237ceff1a8ed996d855d69f372be9aaea44919ea config/oac (237ceff)

Please describe the system on which you are running

• Operating system/version: Amazon Linux 2
• Computer hardware: AMD EPYC 7R13
• Network type: EFA

---

Details of the problem

When trying to run ansys fluent 2023.1 using Open MPI 5.0.x I got the following error:

Starting fixfiledes /fsx/wzam/PortaFiducia/build/libraries/openmpi/v5.0.x/install//bin/mpirun --bind-to core --map-by slot:pe=1 --report-bindings --mca mtl ofi --prefix /fsx/wzam/PortaFiducia/build/libraries/openmpi/v5.0.x/install/ -x LD_LIBRARY_PATH -x KMP_AFFINITY=disabled -x FLUENT_ARCH=lnamd64 -x FLUENT_AFFINITY=0 -x FLUENT_PROD_DIR=/fsx/akraman/fluent/v231/ansys_inc/v231/fluent/fluent23.1.0 -x FLUENT_INC=/fsx/akraman/fluent/v231/ansys_inc/v231/fluent -x PYTHONHOME=/fsx/akraman/fluent/v231/ansys_inc/v231/fluent/fluent23.1.0/../../commonfiles/CPython/3_7/linx64/Release/python -np 1152 --hostfile /tmp/fluent-appfile.ec2-user.44716 /fsx/akraman/fluent/v231/ansys_inc/v231/fluent/fluent23.1.0/linmic/3ddp_node/fluent_mpi.23.1.0 node -mpiw openmpi -pic ethernet -mport 10.17.0.201:10.17.0.201:41469:0
--------------------------------------------------------------------------
The PE=<list> mapping directive cannot be combined with a
binding directive as it already mandates that we bind to
the specified cpu(s). The conflicting directives that were
given are:

  map-by:  slot:pe=1
  bind-to: core

Please resolve the conflict and try again.
--------------------------------------------------------------------------

From what I can tell, the error was because ansys fluent add the following two arguments:

--bind-to core --map-by slot:pe=1 

to mpirun command, but Open MPI 5.0.x consider these two options conflict with each other thus failed.

I wonder if this is expected behavior?

One thing worth noting is that Open MPI 4.1.x will accept such arguments

[rhc54]

I wonder if this is expected behavior?

Yes, that is correct PRRTE behavior.

One thing worth noting is that Open MPI 4.1.x will accept such arguments

OMPI is free to implement whatever behavior it wants! However, note that if the pe is a hwthread, then this is once again a conflicting directive.

[rhc54]

Some said that perhaps the mapping you were trying to achieve as to get one proc assigned to each hwthread, and then to bind that proc to the core (so it actually ran across all the hwt's in that core). If that's the case, then what you really want is something like this:

--map-by ppr:2:core:hwtcpus --bind-to core

where you can replace the 2 with however many hwthreads you have per core. This would map two procs to each core (allowed because you specified use of hwthreads as cpus) and then bind each of them to a core. Since you are mapping at the core level, I think you'll be able to bind to the core level (if you map to hwthread, then you can't bind upwards to the core). Haven't tested it.

You might give it a try and see if it helps.

[bwbarrett]

@wzamazon can you clarify; did the -bind-to core --map-by slot:pe=1 come from your test arguments (which appears to be the case, since the path includes /fsx/wzam/PortaFiducia/build/libraries/openmpi/v5.0.x/install//bin, or did that come from Ansys? Trying to understand the impact of this on our general user base.

[wzamazon]

-bind-to core --map-by slot:pe=1 was added by Ansys fluent's script.

It can be overridden by user

[rhc54]

I did a little digging and found the problem, assuming I am using the correct topology from your nodes. According to the file I have, your system equates "core" with "hwthread" - i.e., they have the exact same cpuset. So if we try to do what I believe they are seeking (i.e., to have multiple procs share a core), then that won't work as there aren't enough cpus under a "core" to allow it (your topology shows only one cpu since the core equates to the hwthread).

I'm not sure why hwloc is making that equivalency - it does create some problems. Used to be that this scenario would result in no cores being defined in the system and it would just consist of hwthreads. Made it easier to check and "do the right thing". Guess I'll need to update the check to see if the cpuset of core and hwthread are the same, and then declare "no cores in system" if they are. Sigh.

Your fluent script might work with older OMPI versions as we weren't accounting for cpu usage - we just blindly mapped and bound, limited only by the number of slots. So if you want to use fluent's script, you might just need to limit users to OMPI v4. Otherwise, they can use --map-by slot --bind-to core and get the same thing you'd see with the older OMPI versions as the pe=1 does nothing on your systems.

[rhc54]

Okay, I spent some time working this problem and have fixed it here. Basically, I added detection for cases like your topology where hwloc is reporting core and pu objects, with them having identical cpusets. PRRTE now "does the right thing" and defaults to using hwtcpus, making the mapping system happy again.

I then checked and found that we don't need to error out if both "pe=N" and a binding directive is given so long as the binding directive is to a cpu (either "core" or "hwt"). I adjusted that check accordingly.

Bottom line is that your fluent script will now work as-is. Please try PRRTE master branch to confirm once the PR has been merged.

I'm somewhat leery of bringing this into the PRRTE v3.0.1 release as I'm unsure what unforeseen consequences might emerge. Our joint testing is poor for these kind of options, and we have all learned the hard way about user anger over problems in the map/rank/bind system. Any thoughts?

And oh yes - you owe me a great big one for this PITA!

[wenduwan]

I will sync with @wzamazon on the issue.