Do we want libompitrace in OMPI v5.0?

[jsquyres]

libompitrace is an incomplete library that intercepts user application MPI API calls, print them out, and then invoke the underlying PMPI call. It currently intercepts 20 of the public MPI APIs.

It was first introduced as libtrace in August 2009 in e1662b4, and then renamed to libompitrace in July of 2010 in 1802325 (due to some other pre-existing / non-MPI libtrace project).

Do we still want to carry this contrib code in Open MPI v5.0?

• Per libompitrace does not check use of MPI_DATATYPE_NULL when calling PMPI_Type_get_name and causes MPI_ERR_TYPE  #10028, there's at least one user using it. 😄
• It's only monitoring 20 (out of hundreds) of MPI APIs.
• I noticed today that it reports the MPI_COMM_WORLD rank for every call (even if there is an input communicator).
• No one has tested or maintained this code in forever.
• I can't find any user-level documentation for it.

[awlauria]

If someone wants to do the work to remove it from master and bring it back to v5 we'll take it.

[rhc54]

As I said in Slack - why? The code has been there for a very long time, requires zero maintenance, there is no way on earth for us to know if anyone is using it (or how many there might be). So why would you tear it out? Because someone used it to expose a bug in OMPI???

[bosilca]

This library is such a weird collection of functions: some collectives, alloc_mem, get_address, accumulate and send/recv. It clearly was meant to be some form of proof of concept, that unfortunately remained blocked at that level.

[awlauria]

We (the v5.0.x rm's) discussed it and are indifferent. If someone wants to do the work to remove it, we wouldn't be opposed.

That said this is pretty late in the game, and if someone out there is using this, it is short notice for removal.

[rhc54]

This library is such a weird collection of functions: some collectives, alloc_mem, get_address, accumulate and send/recv. It clearly was meant to be some form of proof of concept, that unfortunately remained blocked at that level.

No, that's completely incorrect. It is a collection of functions that certain researchers were specifically requesting be traced. They didn't want a profiler or anything more complex - they just wanted a listing of what functions were being called. If someone wants to extend it, feel free. My only point was that it shouldn't be ripped out simply because it exposed a bug in OMPI while someone was using it.

[bosilca]

Taking in account that It takes literally 3 lines of code to write a wrapper for an MPI function, it makes perfect sense to let these certain researchers to write their own support for exactly the few functions they need.

[jsquyres]

FWIW, this didn't expose a bug in Open MPI -- it exposed:

1. An ambiguity in the MPI spec (see Allow MPI_*_GET_NAME functions to accept MPI_*_NULL handles mpi-forum/mpi-issues#544)
2. Bugs in libompitrace (i.e., because of the ambiguity, the only safe thing that can be done is to handle MPI_DATATYPE_NULL because at least one user is using it with a popular library, NCCL, where the MPI_DATATYPE_NULL issue comes up -- see libompitrace: handle MPI_DATATYPE_NULL #10035)

[rhc54]

I guess I will no longer attempt to help anyone in this community given the rancor and nastiness of these conversations. Do what you want - you will anyway 🤷‍♂️

[jsquyres]

@rhc54 and I chatted about this verbally on the phone this past Friday. We both agree that although this library was added for a specific purpose a few years ago, there is no longer a maintainer. There's also few -- if any -- users (I checked with the reporter from #10028: she filed that issue not because they are using libompitrace, but because they just happened to notice the bug, and therefore they filed the bug just to let us know/as a Good Samaritan). For those reasons, it seems reasonable to remove libompitrace from the Open MPI repo.

Hence, I just filed #10051 to remove libompitrace from master. When it is merged to master, I'll also cherry-pick it to v5.0.x.