Implementation of the current state of MPI Continuations proposal

Continuations provide a mechanism for attaching callbacks to outstanding
operation requests. A call to MPIX_Continue takes a request, a
function pointer, a user-provided data pointer, and a status object (or
`MPI_STATUS_IGNORE`), along with a continution request:

```
MPI_Request req;
// status object has to remain valid until the callback is invoked
MPI_Status *status = malloc(sizeof(MPI_Status));
char *buf = ...;
MPI_Irecv(buf, ..., MPI_ANY_SOURCE, ... &req);
MPIX_Continue(&req, &complete_cb, buf, status, cont_req);
assert(req == MPI_REQUEST_NULL);
```

The ownership of non-persistent requests is returned to MPI and the pointer
to the request will be set to `MPI_REQUEST_NULL`. The callback is passed
the status pointer and the user-provided data pointer:

```
void complete_cb(MPI_Status *status, void *user_data) {
  printf("Send completed\n");
  char *buf = (char*)user_data;
  process_msg(buf, status->MPI_SOURCE);
  free(buf); // free the send buffer
  free(status);    // free the status
}
```

The status has to remain valid until the invocation of the callback
and is set according to the operation before the callback is invoked.

The continuation is *registered* with the provided contination
request. The continuation request is a request allocated with `MPIX_Continue_init`:

```
MPIX_Continue_init(info, &cont_req);
```

Continuation requests may be used to test/wait for completion of all
continuations registered with cont_req using `MPI_Test/Wait`. Supported
info keys are:

- "mpi_continue_poll_only": only execute continuations when MPI_Test/Wait is
  called on the continuation request (default: false)
- "mpi_continue_enqueue_complete": if true, the continuation is executed
  immediately if the operations are already complete when MPIX_Continue is
  called. Execution is deferred otherwise (default: false)
- "mpi_continue_max_poll": the maximum number of continuations to execute
  when calling MPI_Test on the continuation request (default: -1, meaning unlimited)

A continuation may in turn be attached to a continuation request, in which
case the continuation request will be executed once all continuations
registered with the continuation request have completed.

In addition to MPIX_Continue, the proposal also includes MPIX_Continueall
which attaches a continuation to a set of requests such that the
continuation is executed once all operations have completed.

The implementation reflects the current state of the proposal and may
change at any time, following discussions in the MPI Forum. The goal
of this Open MPI extension is to provide a working implementation to
the community to experiment with this API.

Signed-off-by: Joseph Schuchart <[email protected]>
Signed-off-by: George Bosilca <[email protected]>

Author: devreal

ompi/mpiext/continue/Makefile.am

@@ -0,0 +1,22 @@
+# -*- shell-script -*-
+#
+# Copyright (c) 2021      The University of Tennessee and The University
+#                         of Tennessee Research Foundation.  All rights
+#                         reserved.
+# $COPYRIGHT$
+#
+# Additional copyrights may follow
+#
+# $HEADER$
+#
+
+# This Makefile is not traversed during a normal "make all" in an OMPI
+# build.  It *is* traversed during "make dist", however.  So you can
+# put EXTRA_DIST targets in here.
+#
+# You can also use this as a convenience for building this MPI
+# extension (i.e., "make all" in this directory to invoke "make all"
+# in all the subdirectories).
+
+SUBDIRS = c
+

ompi/mpiext/continue/c/Makefile.am

@@ -0,0 +1,41 @@
+#
+# Copyright (c) 2021      The University of Tennessee and The University
+#                         of Tennessee Research Foundation.  All rights
+#                         reserved.
+# $COPYRIGHT$
+#
+# Additional copyrights may follow
+#
+# $HEADER$
+#
+
+# OMPI_BUILD_MPI_PROFILING is enabled when we want our generated MPI_* symbols
+# to be replaced by PMPI_*.
+# In this directory, we need it to be 0
+
+AM_CPPFLAGS = -DOMPI_BUILD_MPI_PROFILING=0 -DOMPI_COMPILING_FORTRAN_WRAPPERS=0
+
+include $(top_srcdir)/Makefile.ompi-rules
+
+noinst_LTLIBRARIES = libmpiext_continue_c.la
+
+# This is where the top-level header file (that is included in
+# <mpi-ext.h>) must be installed.
+ompidir = $(ompiincludedir)/mpiext
+
+# This is the header file that is installed.
+nodist_ompi_HEADERS = mpiext_continue_c.h
+
+libmpiext_continue_c_la_SOURCES = \
+    continuation.c \
+    continue.c \
+    continueall.c \
+    continue_init.c \
+    mpiext_continue_module.c
+
+libmpiext_continue_c_la_LDFLAGS = -module -avoid-version
+
+ompi_HEADERS = $(headers)
+
+MAINTAINERCLEANFILES = $(nodist_libmpiext_continue_c_la_SOURCES)
+