Stream-ordered wait{_any,_all}, fix wait_any implementation

[cwpearson]

This adds wait, wait_any, wait_all functions like

template<KokkosComm::KokkosExecutionSpace Space>
KokkosComm::mpi::wait(const Space &exec, KokkosComm:Req<Mpi> req);

Semantically, this inserts the "wait" into the provided execution space instance - i.e., existing work in the instance is allowed to complete, and future work must wait for the communication to complete.
The 1-argument version without an execution space just calls the two-argument version with Kokkos::DefaultExecutionSpace.

• documentation
• more tests

[devreal]

I wonder whether this fence is necessary. If the semantics of KokkosExecutionSpace is such that enqueued operations are ordered then it is sufficient to block the enqueuing thread until the communication is complete. Subsequent submissions are necessarily ordered with the (then-completed) MPI operations.

[janciesko]

I think we do not need that fence there. There are two more occurrences of superfluous fences.

[janciesko]

Consider moving the wait_* functions to a separate file.

[cwpearson]

Semantically we're trying to order this "wait" after anything else already in the execution space instance. There's no way to actually enqueue an arbitrary host function in a Kokkos execution space instance, so the only way to ensure everything already in the instance is done is to fence it. Then we can call the MPI functions to complete the communication.

[devreal]

The difference is not observable. The operations have been started so they have seen the execution space at the time the operation was started. Subsequent kernels should not interfere with these operations, nor do operations submitted after the wait (because they will themselves synchronize the execution space before starting the operation). And I don't think wait_all should guarantee synchronization of the space because that is more than strictly needed.

[dssgabriel]

Some of this was addressed in #216. Currently, the semantics of Requests are to synchronize only the communication (which fences on the execution space associated with the comm operation iff there was some unpacking to do in the callbacks).
Can you confirm we can close this @cwpearson?