mpi4py: Regression in with OMPI at branch v5.0.x

[dalcinl]

Looks like an intercommunicator create error has slipped into branch v5.0.x
https://github.com/mpi4py/mpi4py-testing/actions/runs/8241699956/job/22548076522
https://github.com/mpi4py/mpi4py-testing/actions/runs/8257555604/job/22588319271

[fv-az1022-842:142460] [[58254,1],1] selected pml ob1, but peer [[58254,1],0] on unknown selected pml 
�
[fv-az1022-842:142460] OPAL ERROR: Unreachable in file communicator/comm.c at line 2385
[fv-az1022-842:142460] 0: Error in ompi_get_rprocs
setUpClass (test_ulfm.TestULFMInter) ... ERROR

======================================================================
ERROR: setUpClass (test_ulfm.TestULFMInter)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/mpi4py-testing/mpi4py-testing/test/test_ulfm.py", line 197, in setUpClass
    INTERCOMM = MPI.Intracomm.Create_intercomm(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/mpi4py/MPI/Comm.pyx", line 2336, in mpi4py.MPI.Intracomm.Create_intercomm
    with nogil: CHKERR( MPI_Intercomm_create(
mpi4py.MPI.Exception: MPI_ERR_INTERN: internal error

cc @wenduwan, I see you opened #12403.

PS: Can someone please backport the mpi4py CI tests to branch v5.0.x?

[wenduwan]

Can someone please backport the mpi4py CI tests to branch v5.0.x?

That's on our plate. Haven't figured out the assignee yet.

[wenduwan]

The failure started on 3/11 after we merged c0f98b8

[dalcinl]

So, the submodule bump probably exposed in v5.0.x similar bugs to the ones we have been seeing on main 🤦‍♂️ .

[wenduwan]

@dalcinl I have problem reproducing the failure. mpi4py points to this setup hook
.

So I wrote a minimal reproducer

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv) {
	int size, rank, COLOR, LOCAL_LEADER, REMOTE_LEADER;
	MPI_Comm intra_comm, inter_comm;
	MPI_Init(NULL, NULL);
        MPI_Comm_size(MPI_COMM_WORLD, &size);
	MPI_Comm_rank(MPI_COMM_WORLD, &rank);
        if (rank < size / 2) {
            COLOR = 0;
            LOCAL_LEADER = 0;
            REMOTE_LEADER = size / 2;
	} else {
            COLOR = 1;
            LOCAL_LEADER = 0;
            REMOTE_LEADER = 0;
	}
	MPI_Comm_split(MPI_COMM_WORLD, COLOR, 0, &intra_comm);
	printf("Before intercomm create\n");
	MPI_Intercomm_create(intra_comm, LOCAL_LEADER, MPI_COMM_WORLD, REMOTE_LEADER, 0, &inter_comm);
	printf("After intercomm create\n");
	MPI_Finalize();
}

Then I build and run it mpiexec -n 2 reproducer. But I don't see any issue with that.

Am I reading it wrong?

[dalcinl]

Your C code looks mostly the same except it is missing MPI_Comm_free(&intra_comm) after MPI_Intercomm_create() and before MPI_Finalize().

MMM... what happen if you execute the code in between Init/Finalize in a loop for at least two iterations?
Did you try putting the Python code with just the intercomm create in a script and running it?
Can you tell me the exact fork/branch/commit you are using? I would like to reproduce on my side and be back to you.

[wenduwan]

@dalcinl Thanks for your response. I tweaked the reproducer as suggested and it does not fail either

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv) {
	int size, rank, COLOR, LOCAL_LEADER, REMOTE_LEADER;
	MPI_Comm intra_comm, inter_comm;
	MPI_Init(NULL, NULL);
        MPI_Comm_size(MPI_COMM_WORLD, &size);
	MPI_Comm_rank(MPI_COMM_WORLD, &rank);
        if (rank < size / 2) {
            COLOR = 0;
            LOCAL_LEADER = 0;
            REMOTE_LEADER = size / 2;
	} else {
            COLOR = 1;
            LOCAL_LEADER = 0;
            REMOTE_LEADER = 0;
	}
	for (int i=0; i < 10; ++i) {
		MPI_Comm_split(MPI_COMM_WORLD, COLOR, 0, &intra_comm);
		MPI_Intercomm_create(intra_comm, LOCAL_LEADER, MPI_COMM_WORLD, REMOTE_LEADER, 0, &inter_comm);
		MPI_Comm_free(&intra_comm);
	}
	MPI_Finalize();
}

I ran the mpi4py test with mpiexec -n 2 python main/test.py but nothing more.

This is on latest v5.0.x.

[dalcinl]

I can reproduce the mpi4py failure locally. However, if I try mpiexec -n {2,3,4,5} python test/test_ulfm.py -v to narrow down to the previously failing test, now everything runs just fine. Therefore, there is something in Open MPI state that is screwed up by the time the ULFM tests run.

[dalcinl]

Next I run the following way to EXCLUDE the ULFM tests

mpiexec -n 2 python test/main.py -v -e ulfm

and then a intercommunicator failed the following different way:

testAllgatherInter (test_util_pkl5.TestMPIWorld.testAllgatherInter) ... [kw61149:407450] [[50638,1],1] selected pml ob1, but peer [[50638,1],0] on unknown selected pml ��
[kw61149:407450] OPAL ERROR: Unreachable in file ../../ompi-v5.0.x/ompi/communicator/comm.c at line 2385

[dalcinl]

This may be the same or similar issue being discussed in #12367. @bosilca nailed down the problem #12367 (comment), but I have no idea if anyone is investigating further or someone is working towards a fix.

[rhc54]

If all those tests are executing within the same mpiexec, then there is going to be a lot of crosstalk and problems. If you really want to test each of those fairly, then you need to execute each of them in their own mpiexec instance.

[dalcinl]

If you really want to test each of those fairly,

I'm not sure what do you mean by "fair". That way of testing is what everyone else does. Such approach goes easy on MPI, short lived tests, a few features at at time. But mess can accumulate out of control and only pop up when users run more complex applications.

then you need to execute each of them in their own mpiexec instance.

mpi4py does almost all of its testing on a single mpiexec instance, and that has been the approach for ages. The hole point is to make sure that the internal state of the MPI library is always consistent, there are no races, etc. If there is crosstalk, then that is a bug that has to be fixed.