mpi4py: intercomm_create_from_groups can hang in certain scenarios

[dalcinl]

The testsuite hangs until timeout. This happens only with main. Runs from yesterday were fine.

[dalcinl]

@jsquyres @rhc54 The regression comes from #11408.

[rhc54]

I'm afraid I don't know anything about your test suite. Precisely what test were you running, how was it executed, etc?

[dalcinl]

@rhc54 I've asked many times for mpi4py to be integrated into Open MPI test infrastructure. I even offered a ready to use workflow working GitHub Actions. Unfortunately, it never happened, I never got a plain answer for yes or no. At this point I think it is some political issue, but no one wants to say that in the open (or even in private). Open MPI issue reports from mpi4py users have made my life miserable over the years, with regressions every other patch release. I'm doing the only thing slightly less miserable for me: run against v5.0.x and main, and submit an issue here every time there is a failure, to prevent mpi4py users a broken Open MPI reaching to their hand. I either do that, or I say all my users: mpi4py no longer supports Open MPI, and offload all my misery to the Open MPI team. Sorry for the rant, you have nothing to do with it, but sometimes I get really frustrated.

Running mpi4py testsuite is pretty easy. Just copy&paste the following hunk. Everything will happen in a Python virtual environment created under /tmp, so no modifications to your system afterwards.

cd /tmp
git clone --depth 1 https://github.com/mpi4py/mpi4py.git
python3 -m venv /tmp/venv
source /tmp/venv/bin/activate
pip install ./mpi4py
export CFLAGS="-O0 -Wp,-U_FORTIFY_SOURCE"
mpiexec -n 2 python3 ./mpi4py/test/main.py -v -k TestCommWorld.testCreateFromGroup

The last -k ... argument is to run a particular test, which is the one that is failing (it hangs forever).

And then, by looking at the failing test, the following Python code is a minimal reproducer:

from mpi4py import MPI
group = MPI.COMM_WORLD.Get_group()
comm = MPI.Intracomm.Create_from_group(group)

Put these three lines in a reproducer.py file, next run:

mpiexec -n 2 python3 reproducer.py 

and it hangs for ever.

MPI_Comm_create_from_groups is typically used with MPI_Groups out of MPI_Sessions, but IIRC there is nothing in the MPI-4.0 standard that prevent us from using the routine with a group that comes from a communicator. My wild guess is that there is some issue/oversight that triggered after the PMIx/PRRTE submodule bump. If that endup being the case, then I'm sorry @rhc54, this would be the second time I ping you for things you are ultimately not responsible for.

@hppritcha Maybe you should take a look into this issue.

[rhc54]

Yeah, ranting at me doesn't accomplish anything. First, I'm not part of the OMPI community - and second, I'm retired. OMPI just uses a couple of my software packages.

I can't take on different test suites as there are several, no clear reason to favor one over the other - and I have limited time. Besides, I support multiple MPI and other programming model libraries, and I'd drown if I tried to get that deeply involved in any one of them. Sorry, I know that isn't an answer one wants to hear, but it's the reality. Which is why I need to know what was being tested.

As for this problem: it probably isn't something to do with PRRTE as mostly PRRTE just starts the procs. It might have something to do with PMIx if MPI is getting some PMIx-provided data and has a problem getting it, or is calling something like PMIx_Group (which could extend up into a call to PRRTE) and hits a problem. I honestly don't know anything about what OMPI is doing in that function, so somebody over there will have to investigate and let me know if something is amiss on my side.

[dalcinl]

Yeah, ranting at me doesn't accomplish anything.

Well, I'm not native English speaker, and I'm not sure the exact meaning of "ranting at me". I said in my comment: you have nothing to do with it. I'm aware you are not part for OMPI. I'm having trouble to manage my frustration. Probably I'm being victim of the usal open-source mainter burnout syndrome, and I resist to acknowledge it and move away for good.

I can't take on different test suites as there are several, no clear reason to favor one over the other.

Well, perhaps you do not ultimately care about MPI testing, but just FYI: mpi4py can build and run its tests in about one minute in bare metal (CFLAGS=-O0, mpiexec -n 2 ...). The testsuite covers almost all of MPI-4.0, uses the new large-count APIs exclusively if available, and all tests run in a single mpiexec execution, so it can potentially catch (and have catched in the past) internal MPI components not playing well each other.

[hppritcha]

@dalcinl i'll take a look and yes it would be good to add the mpi4py testsuite to a reliable MTT platform. I'll add both to my enormous todo list for this week.

[awlauria]

How long does the mpi4py bucket take to run? With how many breakages we have it might be better to put these in a CI test.

From your previous comment it takes about a minute to run - is that right? I think moving this to a CI check is a good idea.

[dalcinl]

@awlauria Don't take my word for granted regarding running times. See yourself the times in GitHub actions, for example this run: https://github.com/mpi4py/mpi4py-testing/actions/runs/4716216855/jobs/8363742095. Note that GitHub Actions runners only have 2 virtual cores, so when running 5 MPI processes with oversubscription it takes close to 7 min. With singleton, np=1, and np=2, thus no oversubscription, it is about one minute a a half each. On a dedicated, a bare metal server, singleton and np=5 would be enough. On the underpowered 6 core box I'm using right now, np=5 takes less than a 1 minute and a half with MPICH. All this is without testing GPU communication.

[dalcinl]

Why don't you guys just start with setting up the thing in GitHub Actions? You just need to grab https://github.com/mpi4py/mpi4py-testing/blob/master/.github/workflows/openmpi.yml and build from there, adding triggers to build on pull requests. If you remove the np=5 case, everything will run in 10 to 15 minutes.

[dalcinl]

@hppritcha Sorry for my insistence, but mpi4py test suite have been failing for about two weeks. I'm just worried that the failure from this know issue may be masking other potential regressions as main keeps accumulating changes.

[ggouaillardet]

I assume you meant the main branch.
The master branch is no more updated and lags behind the main branch by more than one year.

Here is the mpi4py code I mimic'ed in C (I am unable to run the script you provided as-is)

from mpi4py import MPI
group = MPI.COMM_WORLD.Get_group()
comm = MPI.COMM_WORLD.Create_group(group)

FWIW, I was able to run the following C program with 2 MPI tasks on my mac and with the latest main branch

#include <stdio.h>
#include <mpi.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    MPI_Group group;
    MPI_Comm_group(MPI_COMM_WORLD, &group);
    MPI_Comm comm;
    MPI_Comm_create_group(MPI_COMM_WORLD, group, 0, &comm);
    MPI_Finalize();
    return 0;
}

The mpi4py test suite can definitely help notice regressions, but from a very pragmatic point of view,
I am afraid this is only what it does.
The next step (from an Open MPI developer) is to isolate the test case and write a C reproducer.
In this very specifc case it looked like an easy one so I tried it and ... I was unable to reproduce the hang.

[dalcinl]

I assume you meant the main branch.

Yes, of course. I should have said ... as the main branch keeps accumulating changes.

The master branch is no more updated and lags behind the main branch by more than one year.

I did not mention the master branch. I'm aware that master is no longer active.

(I am unable to run the script you provided as-is)

Well, yes, you would have to install mpi4py for that. But I guess people can find that step cumbersome.

FWIW, I was able to run the following C program

Your program does not use the broken MPI routine I'm reporting. You are testing MPI_Comm_create_group, but it should be MPI_Comm_create_from_group. This is the proper reproducer:

#include <mpi.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    MPI_Group group;
    MPI_Comm_group(MPI_COMM_WORLD, &group);
    MPI_Comm comm;
    MPI_Comm_create_from_group(group,
                               "org.example",
                               MPI_INFO_NULL,
                               MPI_ERRORS_ARE_FATAL,
                               &comm);
    MPI_Group_free(&group);
    MPI_Comm_free(&comm);
    MPI_Finalize();
    return 0;
}

The mpi4py test suite can definitely help notice regressions, but from a very pragmatic point of view,
I am afraid this is only what it does.
The next step (from an Open MPI developer) is to isolate the test case and write a C reproducer.

So are you complaining about the inconvenience of writing a C reproducer because you don't want to use Python?
Is an unnoticed regression preferable?

In this very specifc case it looked like an easy one so I tried it and ... I was unable to reproduce the hang.

Well, it was an easy one indeed, you just confused the routine to test because of the very similar names.

[ggouaillardet]

My bad: mympi4py tree was still pointing to bitbucket.org and was out of date, that's why I missed the right subroutine.

My point is python/mpi4py adds an extra layer that generally does not fly well with debuggers, so a C reproducer is way easier for troubleshooting.

FWIW, on my mac,

• the C test passes
• the mpi4py does not hang but fails on self.assertEqual(ccmp, MPI.CONGRUENT)
  in C, comparison returns MPI_IDENT and I guess MPI_CONGRUENT is expected.

[hppritcha]

@dalcinl you are still observing the test timeout problem, correct? would the assert test failing cause this timeout behavior?
I will have some time this week to look at this.