Does Open MPI support asynchronous message progression?

[Yiltan]

Is there a way to configure OMPI to launch a helper process to aid with message progression?

I see this old comment here:

ompi/config/opal_configure_options.m4

Line 513 in f6c84cd

dnl We no longer support the old OPAL_ENABLE_PROGRESS_THREADS. At

But I cant seem to work out what OMPI's new way to do it is. I'm specifically looking to use UCX if possible.

[MichaelLaufer]

I was also researching this for a few hours earlier today, and could not find an answer, and would also be very interested in the current status of this. We have just written a new code that was designed for compute/communication overlap using MPI_Isend/MPI_Irecv + MPI_Waitall only to find out that no background progress is occuring (OMPI + UCX w/ Mellanox NICs).

In my findings, MVAPICH, MPICH and Intel MPI all seem to feature this async progress thread, and I found it odd that scarcely any information was available for Open MPI.

[alex--m]

@bosilca - Would you know if generic progress threads (as opposed to BTL-specific) are still functional somehow?

[bosilca]

I do not recall the code being removed, so it might be still there, albeit non-functional. I will try to take a look.

[Yiltan]

Did anybody work this one out?

[arstgr]

Did anyone come up with a solution for this?

[janjust]

Hi - as far as UCX, it progresses only asynchronous events (eg. connect/disconnect), the message progress Isend, Irecv are done via calls to ucp_worker_progress(). So the asynchronous thread should come from upper level opal/ompi, and it doesn’t look like we support it.
To add a bit more, if HW resources are available and a message can be immediately posted then HW progression is "asynchronous" to the software layer, but there's no UCX progress thread to check the HW queue and keep it full (on the sender side) if there's pending send activity queued up in the software side.

[alex--m]

@janjust - your answer is regarding progress threads on the UCX level, right?
I think the original approach was to have a progress thread on OMPI's level, which would periodically call internal MPI progress (e.g. opal_progress) - which in turn would also call UCX's explicit progress function (i.e. ucp_worker_progress).

In the absence of such an OMPI-level progress thread - how can async. communication (e.g. MPI_Isend) overlap with computation?

[janjust]

It would be up to the user to implement calls to ompi wait routines which I think would call into opal progress.

[alex--m]

If communication progress only happens during wait calls - does it still qualify as computation/communication overlapping? Maybe it's just me, but I think the answer is no. :(

[Yiltan]

I think you could also use MPI_Test to engage the progress engine.

Something like

while(!flag) {
  do_work();
  MPI_Test(request, flag);
}

Still non-ideal but would allow for some more overlap than just an MPI_Wait

[alex--m]

Yes, looks like using MPI_Test will make OMPI call opal_progress if the request is not ready... though it won't help applications not written/modified to use this technique, unfortunately.

With OMPI seeming to have dropped support for async. progress threads - looks like this is the next best thing.

[yosefe]

@MichaelLaufer in your example with MPI_Isend/Irecv, what is the message size?
Can you try to set UCX_RC_MLX5_TM_ENABLE=y (assuming ConnectX-5 or newer NIC)?

[jsquyres]

Just to clarify, Open MPI has not dropped support for progress threads.

The old macro OPAL_ENABLE_PROGRESS_THREADS is a red herring. There are several low-impact progress threads in MPI processes today (e.g., PMIx, libevent, etc.). Network layers are free to implement their own progress threads if they want to.

Be aware that there are costs associated with low-latency-network progression threads. These costs don't typically impact large message performance, but they can definitely affect short message performance (e.g., latency). Hence, such progress threads need to be implemented quite carefully. Make sure that these threads do not impact the latency of any other network transport, for example.

Finally, it's somewhat of a misnomer to state that Open MPI with UCX doesn't have "asynchronous progress". There are multiple types of asynchronous progress, at least some of which are implemented by firmware and/or hardware. I think that this issue is specifically asking about asynchronous progress of large messages that have to be fragmented by the software layer. Perhaps NVIDIA people can explain how that works in UCX, and/or whether there is asynchronous progress for all fragments in a large message in UCX and/or Open MPI's support for UCX.

[yosefe]

Finally, it's somewhat of a misnomer to state that Open MPI with UCX doesn't have "asynchronous progress". There are multiple types of asynchronous progress, at least some of which are implemented by firmware and/or hardware. I think that this issue is specifically asking about asynchronous progress of large messages that have to be fragmented by the software layer. Perhaps NVIDIA people can explain how that works in UCX, and/or whether there is asynchronous progress for all fragments in a large message in UCX and/or Open MPI's support for UCX.

👍
With UCX and Mellanox NICs, OpenMPI progress thread is probably not the right approach; we'd like to offload to HW as much as possible.
Large messages would use rendezvous protocol that is fragmented to packets by RDMA HW, but there could be a problem of RTS received after MPI_Irecv ("expected") and data fetch not started before going to MPI_Wait. That's what UCX_RC_MLX5_TM_ENABLE=y is aimed to address.