What is the golden path to assign cuda devices to ompi processes?

[a-szegel]

Please submit all the information below so that we can understand the working environment that is the context for your question.

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

OpenMPI 5

Details of the problem

What is the recommended way for OMPI5 applications to assign cuda devices to ranks?

From the docs:

https://www.open-mpi.org/faq/?category=runcuda#mpi-cuda-dev-opa

In addition, each process of the application should select a specific GPU card to use before calling MPI_Init(), by using cudaChooseDevice(), cudaSetDevice() and similar.

But an example MPI application cannot get the rank until after MPI_Init() is called:

int main(int argc, char** argv) {
    // Requires GPU to be set here

    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    // Get the number of processes
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // learns rank here
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

I didn't see any flags in mpirun to mask which gpu's are being exposed to which process, although this should be possible to do with a custom launch script.

Given these constraints, what is the golden path for a user developing a cuda aware OMPI application?

[a-szegel]

needed to read further down in the docs, section 11.

[rhc54]

It may be due to MPI not providing an appropriate command, or perhaps someone not realizing it - but OMPI already knows both your local rank within your job (0 to number of procs in your job on the node), and the node rank, which takes into account any other processes (e.g., from a comm_spawn) that are on your node. This information is collected during MPI_Init and actually stored in a global construct.

The little code snippet provided in the docs doesn't access that info, and so you might in fact get the wrong answer and unnecessarily overload a GPU. Perhaps someone here can advise you on how to access that info?

[jsquyres]

We're definitely open to improving the docs -- if there's something that can be communicated better or a way to improve that example, please open a PR. Thanks!

[rhc54]

@jsquyres I cannot open a PR to improve your docs because I don't know how you provide access to the stored information in opal_process_info. This is why I asked if someone could please advise on that point. Motivation is that the provided sample is in fact capable of providing the incorrect answer - and the correct info is already present in OMPI. Just need to know how to access it.

Or maybe the problem is that you haven't provided a way to access it? 🤷‍♂

[bosilca]

Giving access to internal information would provide a non-portable way to access information, contrary to MPI spirit. I don't understand how the provided sample can provide the incorrect answer ?

Anyway, I saw many users use the environment variable OMPI_COMM_WORLD_LOCAL_RANK to bind the process to a specific GPU.

[rhc54]

Perhaps the term "incorrect" isn't entirely accurate - better to perhaps say "leads to undesired behavior"?

What the user is actually attempting to do is distribute the load (aka, ranks) across the available GPUs. The provided method will correctly result in that desire IF there is only one job running on the node - e.g., nobody has called MPI_Comm_spawn. If that isn't the case, then the result of the given operation will cause preferential loading on the lowest numbered GPUs, perhaps leaving the higher numbered GPUs idle.

What you really want is the node rank, which takes all procs on the node into account. This will achieve the desired objective, but the provided code doesn't get you there, and I'm not sure there is an equivalent MPI communicator-based method that can achieve it.

mpirun does indeed provide envars for this purpose. In fact, we do provide OMPI_COMM_WORLD_NODE_RANK, which is the value you want to use. However, using that envar isn't portable either - though it does remove the need to access internal data. Either way will work.

I'm happy to update the doc, if you like. It's a pretty simple change - I'd just add an option to use the envar, and if that isn't available, then fallback to the code snippet. I was only asking if there is some provided way (maybe some MPI Info request?) to get at internal data. If not, then that's okay - can just do it the alternative way.

[bosilca]

Users using MPI_Comm_spawn, quite an advanced feature in MPI, should be well aware of issues related to resource sharing, not only at the GPU level, but any other node-level resources, and handle them carefully.

The code snippet from the online FAQ if applied on the "universe" communicator (aka. the intra-comm created from the intercomm resulting from a single spawn) will get the same sane behavior. But without global knowledge of the entire process placement, there is little that can be done. Even the solution you propose will not work if instead of spawn we use connect/accept (which bridge across multiple jobs).

[rhc54]

Even the solution you propose will not work if instead of spawn we use connect/accept (which bridge across multiple jobs).

Hmmm...actually, I don't think that is true. Connect/accept has nothing to do with node rank as the latter is based solely on process placement. mpirun knows exactly what procs were placed on the node where we are executing, and it computes the node rank for each process on that node. So the node rank will be correct.

A process in job A will not know the node rank of a process in job B, but that isn't required. I just need to know my own node rank.

I agree that you could apply the snippet to the intercomm to get the answer IF only one comm_spawn is performed. However, that isn't a general solution. Using the node rank works regardless of how many comm_spawns are performed, and even for the case where multiple independent jobs are launched from a common DVM. So it is the most general solution.

Given that the envar is available, it seems a trivial change to the docs. If the envar is present, use it - if not, you can fallback to the provided snippet with some given caveats.

Make sense?

[bosilca]

We had this discussion in the past, your view diverges from what the MPI standard defines regarding accept/connect (where it can cross dvm boundaries).

I agree with you regarrding the scenarios with a single DVM: the node rank is correctly computed and can be used to uniformly bind devices. Therefore, adding a blurb about the envar to the documentation would be a nice addition, as it will address most of the traditional use cases.

[rhc54]

your view diverges from what the MPI standard defines regarding accept/connect (where it can cross dvm boundaries).

Perhaps we could connect via Slack and then have a phone conversation about this? I really would like to understand it, and I feel that I am missing something.

If you are talking about DVMs that share nodes, then I readily agree there will be resource contention. The DVMs assign compute resources, so the user can use the cmd line options to separate those between the DVMs. We don't, however, assign GPUs, and so there is no mpirun cmd line option to separate those. If there is something we can/should do to help with that scenario, I'm happy to help make that happen.

adding a blurb about the envar to the documentation would be a nice addition

Thanks - I will put something together and flag it for your review.

[rhc54]

We had this discussion in the past, your view diverges from what the MPI standard defines regarding accept/connect (where it can cross dvm boundaries).

Just putting this back on my radar to write up and looked at this comment again. Just to clarify, I have no "views" about crossing DVM boundaries. PRRTE currently doesn't support connecting DVMs (because it has nothing to do with the original requirements for that package), and I honestly cannot remember anyone ever asking for such support until a couple of months ago. However, that request has nothing to do with MPI - different programming model, no MPI involved.

Still, that request might provide the connection to give OMPI the support it would need for accept/connect across DVMs. So once we satisfy the existing request, someone can see if it also resolves this one - or can do whatever is required to extend the support to make it work.

[bosilca]

MPI has always allowed an MPI process to connect to any other MPI processes independent on how they were created. This means processes from different DVMs can be connected together into a single MPI communicator, and will have to have the full range of MPI behaviors (including error reporting).

I guess what changed is that finally people are starting to try to use this MPI feature, and come to realize it was broken in OMPI for a long time (it did work with ORTE at some point).

[ggouaillardet]

My 0.02$
The initial question was how to get some information ••before** MPI_Init() is invoked.
Checking some environment variables is an option, but not a portable one.

Could the user PMIx_Init() and get the required information? (Assuming of course it is ok to select the GPUs after PMIx_Init() and before MPI_Init())

[rhc54]

Could the user PMIx_Init() and get the required information? (Assuming of course it is ok to select the GPUs after PMIx_Init() and before MPI_Init())

Short answer is "yes" - the info is available and we refcount the calls to PMIx_Init, so all they would need to do is be sure to call PMIx_Finalize to balance things.