Provide put & get message passing primitives for the bsp1d backend

[GiovaGa]

Provide an interface for point-to-point communication between processes run by the bsp1d backend

[GiovaGa]

More precisely this feature will provide the following functions, inside the namespace grb::spmd (?) :

• put(...), get(...) to read/write messages to/from another process, with specializations for grb::Vector ;
• grb::collectives::register_global(...), register_local(...) to register memory to be used for inter-process one-way memory access;
• Registered memory should have a special type.
• Some kind of lock or way to synchronize only a pair of processes. Currently only a global sync is available, but this would be very useful for my specific use-case

@anyzelman am I missing something important?

[anyzelman]

I suggest to create a new namespace instead of collectives, called grb::rdma<>. The put, get, register_{global,local} should go in that namespace. For the third point, in your use case, are you sending ALP objects or non-ALP objects? (Are you always communicating reference vectors, cf. #403, for example?)

If the latter (i.e., it's really memory that we expose to users), the roadmap looks good but if it's the former (which seems more C++-y to me), then whether an ALP object had been registered for RDMA communication should be part of the internal ALP state.

[anyzelman]

Note btw (but I think you already noticed) that the related functionality is actually already implemented, so it's really all about the interface right?

(most of the implementation is here: https://github.com/Algebraic-Programming/ALP/blob/develop/include/graphblas/bsp1d/init.hpp#L50 )

[GiovaGa]

I suggest to create a new namespace instead of collectives, called grb::rdma<>. The put, get, register_{global,local} should go in that namespace. For the third point, in your use case, are you sending ALP objects or non-ALP objects? (Are you always communicating reference vectors, cf. #403, for example?)

I will be sending both ALP objects and scalars. In any case the RDMA interface for ALP objects will use the same functions as scalars, so we might as well expose it

[GiovaGa]

Note btw (but I think you already noticed) that the related functionality is actually already implemented, so it's really all about the interface right?

(most of the implementation is here: https://github.com/Algebraic-Programming/ALP/blob/develop/include/graphblas/bsp1d/init.hpp#L50 )

Seemingly there is something, but it seems very little

[anyzelman]

I suggest to create a new namespace instead of collectives, called grb::rdma<>. The put, get, register_{global,local} should go in that namespace. For the third point, in your use case, are you sending ALP objects or non-ALP objects? (Are you always communicating reference vectors, cf. #403, for example?)

I will be sending both ALP objects and scalars. In any case the RDMA interface for ALP objects will use the same functions as scalars, so we might as well expose it

In this case I suggest that put, get, register_{local,global} are strongly typed functions that take only the arguments that you need at the moment -- e.g., grb::Vector< T, reference > and POD types (i.e., scalars). Then ALP internally can worry about translating it to registrations and RDMA requests of the right raw memory areas.

[anyzelman]

Note btw (but I think you already noticed) that the related functionality is actually already implemented, so it's really all about the interface right?
(most of the implementation is here: https://github.com/Algebraic-Programming/ALP/blob/develop/include/graphblas/bsp1d/init.hpp#L50 )

Seemingly there is something, but it seems very little

All BSP1D functionality goes back to the structures and functions there, although in the majority of cases we revert to collectives rather than direct RDMA. Still, for LPF-based collectives, the structs defined here are used, and also memory registration I believe works through the code there.

Also worth noting is that the code linked to is not a simple LPF wrapper to C++, the difference is the following:

LPF is super minimal in that one needs to actively reserve enough memory slots and enough space for in-flight messages before registering memory and before issuing RDMA requests. The ALP code linked to instead turns this into active management: the reserved buffer sizes are recorded, and if a memory registration is requested while the related buffer size is insufficient, ALP will make sure the related buffers are resized. This also implies a warning for this issue though: the way the buffer management works needs to be ensured to be compatible with the new exposed RDMA interface. This is much harder to do than previously, because in principle, a user can issue any arbitrary communication pattern in any form of overlap.

I think we can just take the easy way out, at least for a first version: if the user requests a registration that violates an active buffer constraint, we just throw an exception. A user can catch that exception and if caught, ask ALP to increase its buffer (in essence, thus, going back to LPF's semantics)

[anyzelman]

Some kind of lock or way to synchronize only a pair of processes. Currently only a global sync is available, but this would be very useful for my specific use-case

This one is tricky btw -- LPF has functionality for it, the so-called synchronisation attributes which are designed to allow "weakening" synchronisation semantics while keeping the code structured. The weakest is the so-called zero-cost synchronisation, but I have strong reservations about exposing that to the ALP user directly: it's powerful, but very easy to make mistakes with. Let's maybe first get your use case to work with global fences, and then look at the code together to see how we can make it both performant and simple to use?

[GiovaGa]

Thinking about a way to make RDMA work in a user-friendly way, I had the following idea:

We could allow RDMA to work only between a specified set of "associated" memory regions: when the global registration is done. We store internally the pairing between the given address and its memory slot.
When an RDMA memory access is done, the user will just give the local address (or object) and the PID whose memory he wants to access. The lpf memory slot is then recovered internally and the RDMA done between the objects globally registered together and thus associated at that time.

@anyzelman let me know if this seems a good way to proceed.

[anyzelman]

Yes that's a good idea. In discussion just now (first test by replacing nonblocking in the below with reference), the API could be as follows:

• grb::RC grb::rdma< BSP1D >::register( const grb::Vector< T, nonblocking > &vector )
• grb::RC grb::rdma< BSP1D >::register( const T &scalar)
• grb::RC grb::rdma< BSP1D >::localRegisterSize( const size_t )
• grb::RC grb::rdma< BSP1D >::put( const grb::Vector< T, nonblocking > &src, const size_t dst_pid, grb::Vector< T, nonblocking > &dst )
• grb::RC grb::rdma< BSP1D >::put( const T &src, const size_t dst_pid, T &dst )
• grb::RC grb::rdma< BSP1D >::get( const size_t src_pid, const grb::Vector< T, nonblocking > &src, grb::Vector< T, nonblocking > &dst )
• grb::RC grb::rdma< BSP1D >::get( const size_t src_pid, const T &src, T &dst )
• grb::RC grb::rdma< BSP1D >::fence( ... )

Offsets, matrices, etc. we will not put in this MR, as the use case does not require it.

[GiovaGa]

Small note: register is a C keyword, so I will use register_global