Open-Coarrays and gfortran-15 with OpenMPI

[amckinstry]

Hi,

I'm maintaining OpenMPI in Debian, and open-coarrays.

As of gfortran >= 14, we get the following error:

OpenMPI uses addresses for windows instead of identical ids on all images for the same token.
    # Therefore we can't use it (yet; and probably never).

Is this ever likely to change, or is there anything we can do about it? As it stands OpenMPI is the default MPI implementation in Debian/Ubuntu and coarrays in Fortran is an important feature for new high-performance libraries like Caffeine.

Thanks
Alastair

[ggouaillardet]

@amckinstry can you point me to where this message is coming from?
I am not exactly sure what it means, and there could be a workaround for this.

[devreal]

@amckinstry I suspect that it warns about the way Open MPI implements MPI handles: pointers to opaque structs. MPICH has historically used integers as MPI handles, requiring a lookup in a table when accessing the underlying object. It seems open-coarrays can only deal with the latter.

I'm afraid this problem will not go away. In fact, the ABI introduced in MPI 5.0 has standardized the use of pointers. Both MPICH and Open MPI will support that ABI. I suspect that the MPICH legacy ABI will stay around for a long time but Open MPI is not going to adopt it.

The path forward is for open-coarrays to adapt to using pointers as MPI handles. I'm happy to support that effort (albeit with limited resources) if we can identify what the challenges are.

[bosilca]

I agree with @ggouaillardet why would that matter you only care about local addresses and remote displacements when dealing with windows. Why is the naming scheme important ?

[ggouaillardet]

I would suspect co-arrays took some shortcuts and need the opaque handler to be the same on all ranks in order to work. That requirement could be met by MPICH but Open MPI offers no such guarantees and (very likely) never will. The Fortran id might fit the requirement though, that's why I'd like to understand the context better.

I am not sure the MPI.ABI will change anything though. My understanding is the opaque handler size must be the same than a pointer size, but they are generally not pointers that can be dereferenced (constants such as MPI_COMM_WORLD are clearly no valid pointers).

[bosilca]

We're talking about a shortcut that would require one process to know how a different process named it's windows. Basically, a delegation mechanism, where instead of applying some operation directly on the window the operation is delegated to the target to be executed. In that case, they could have created their own naming scheme, with a globally unique identifier if necessary. Creating a globally unique ID for an object is well detailed in the literature, there would be little challenge there.

Is there a guarantee in MPICH that the window ID is consistent across the entire job ?

[devreal]

Not sure there is any such guarantee. Maybe if you don't use non-world communicators. Or it might just be the way they do lookups or handle predefined communicators/windows. This should be escalated to open-coarray to see what the problem is and how it can be addressed.

[ggouaillardet]

Just to be crystal clear, the unique opaque handler thing is a speculation of mine. I will know for sure once I see the code. I googled and did a few greps in open-coarrays and GCC but did not find the error message. FWIW, grok hallucinated a reply (that looked compelling though).

[amckinstry]

@amckinstry can you point me to where this message is coming from? I am not exactly sure what it means, and there could be a workaround for this.

Its from the open-coarrays source:
https://salsa.debian.org/science-team/open-coarrays/-/blob/upstream/latest/src/runtime-libraries/mpi/CMakeLists.txt?ref_type=heads
line 150

[bosilca]

I love the comments in that file # Write out a host file because OMPI's mpiexec is dumb !! Fascinating

[amckinstry]

Just found this discussion:
Homebrew/homebrew-core#230634

Further down, the issue itself:
sourceryinstitute/OpenCoarrays#793

[bosilca]

Exactly what I though, they use a non-MPI feature to implement support for delegation. They identified a few solutions, including the correct solution, aka. compute a unique ID in CAF, but failed to implement any.

If we assume all windows (or CAF images) are created globally (aka. using only MPI_COMM_WORLD as the underlying communicator), then MPI_Win_c2f will provide a globally unique ID for each window in Open MPI. Honestly, I think this is also the case in MPICH, because why would one generate a globally unique ID that is not necessary.

[bosilca]

Here is a reproducer highlighting that CAF assumption that MPICH Window ID are unique is generally incorrect, and it only works if all windows are created globally. In that case, OMPI c2f window id is also globally unique, so they can just use it directly. In fact, using MPI_Win_c2f is working on all cases as long as all windows are globally created (and I guess CAF + MPI applications are excluded).

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv)
{
  int8_t buf[1024];
  MPI_Comm split;
  MPI_Win win_world1, win_world2, win_split = MPI_WIN_NULL;
  int rank, size, ww1, ww2, ws = -1;

  MPI_Init(NULL, NULL);

  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  MPI_Win_create( buf, 1024, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win_world1);
  ww1 = MPI_Win_c2f(win_world1);

  MPI_Comm_split(MPI_COMM_WORLD, rank < (size/2) ? 0 : MPI_UNDEFINED, 0, &split);

  if( MPI_COMM_NULL != split ) {
    MPI_Win_create( buf, 1024, 1, MPI_INFO_NULL, split, &win_split);
    ws = MPI_Win_c2f(win_split);
  }

  MPI_Win_create( buf, 1024, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win_world2);
  ww2 = MPI_Win_c2f(win_world2);

  printf("%d/%d  ww1 = %d ws = %d ww2 = %d\n", rank, size, ww1, ws, ww2);

  // Cleanup
  MPI_Win_free(&win_world2);
  if( MPI_WIN_NULL != win_split ) {
    MPI_Win_free(&win_split);
  }
  MPI_Win_free(&win_world1);
  if( MPI_COMM_NULL != split )
    MPI_Comm_free(&split);
  MPI_Finalize();

  return 0;
}

[jsquyres]

Note that relying on MPI_COMM_WORLD for global unique IDs only works in the world model, and will not work with sessions. So... that solution isn't even a good one, either. They really should compute a global unique ID in CAF; a global unique ID is just not something that MPI guarantees, and any particular implementation that happens to give one (in some very specific circumstances) is a recipe for future bug reports.