MPIX Stream

[hzhou]

Summary

MPIX Stream introduces a new concept/object into MPI to represent the execution context, by default, a serial execution context. MPIX streams can represent CPU threads so that users can convey threading context explicitly to MPI. It also can be amended by info hints to represent a GPU execution stream and to support async operations that are specific to GPU runtimes. Internally, an MPI implementation maps MPIX stream to hardware network endpoints, allowing users explicit control of communication performance.

APIs

MPIX Stream creation

• int MPIX_Stream_create(MPI_Info info, MPIX_Stream *stream)
• int MPIX_Stream_free(MPIX_Stream *stream)

Stream communicator

• int MPIX_Stream_comm_create(MPI_Comm parent_comm, MPIX_Stream stream, MPI_Comm *stream_comm)
  • Works with existing MPI operations with the additional MPIX Stream semantics

Multiplex stream communicator

• int MPIX_Stream_comm_create_multiplex(MPI_Comm parent_comm, int num_streams, MPIX_Stream streams[], MPI_Comm *multiplex_comm)
  • point-to-point APIs with rank + index addressing scheme

GPU enqueue APIs

• int MPIX_Send_enqueue(buf, count, datatype, dest, tag, comm)
• int MPIX_Recv_enqueue(buf, count, datatype, source, tag, comm, status)
• int MPIX_Isend_enqueue(buf, count, datatype, dest, tag, comm, request)
• int MPIX_Irecv_enqueue(buf, count, datatype, source, tag, comm, request)
• int MPIX_Wait_enqueue(request, status)
• int MPIX_Waitall_enqueue(count, array_of_requests, array_of_statuses)

discussion

• Identical signatures to existing APIs with the additional enqueue semantics
• Readily to be extended to collectives and one-sided APIs
• Once we have an async graph object, in principle, we can extend with -
  * int MPIX_Graph_enqueue(graph, comm)

References

• Original discussion
• 2022-06-24 -- HACC meeting
• 2022-09-28 -- EuroMPI presentation
• MPIX Stream paper