YMQ IOSocketType definitions

[gxuu]

TLDR: See the bottom.
This post purposes the definition of IOSocketType that's being used by YMQ. There will be the following types:

1. ROUTER
2. DEALER
3. UNICAST
4. MULTICAST

1). This type requires the user to pass in the remote IOSocketIdentity when send/recv messages. That is, user passed in argument for IOSocket::sendMessage(message, callback) will have message->address contains a non null pointer; there will also be a new API overloading IOSocket::recvMessage(Bytes address, callback), where address specify the remote IOSocketIdentity and contains a non null pointer.
2). This type requires the user to pass in the remote IOSocketIdentity when send messages; It will also ignore the user passed in remote IOSocketIdentity when recv messages. Note that this is generally what the current implementation does.
3). This type ignores the user passed in remote IOSocketIdentity if presents when send/recv messages. It is assumed that there is only 0-1 established connection at any given point of execution. When there is 1 established connection it is assumed that user wants to send/recv to that connection. When the user call IOSocket::sendMessage or IOSocket::recvMessage and NO established connection is found, then the message is DROPPED, with no guarantee of message delivery.
4). This type requires the user to passed in remote IOSocketIdnetity when send messages. The core will perform a prefix match to all the established connections' identities in the system. The message will be sent to all the connections that are matched. Messages passed in will be completed immediately, with no guarantee of message delivery.

Rationale:

• ROUTER and DEALER are meant to be used with worker and client in scaler, where we do need guaranteed message delivery.
• UNICAST and MULTICAST are meant to be used with connectorMonitor in scaler, where we allow messages to be dropped.
• MULTICAST with empty identity passed in will match every established connections in the system, therefore achieving the PUB in ZMQ, MULTICAST can also be used to achieve topic filtering.

Here's a table that presents characteristics users might care about:

IOSocketTypes	Guaranteed delivery	User callback call time	operations	dest required	# conns
ROUTER	YES	after sendMessage completes	S/R	S/R	n
DEALER	YES	after sendMessage completes	S/R	S	n
UNICAST	NO	immediately after calling sendMessage	S/R	NA	1
MULTICAST	NO	immediately after calling sendMessage	S	S	n

@sharpener6 @magniloquency Please review and see if it fits the use case.

[magniloquency]

1. I'm not sure I see the point in passing in the identity on recv. That's not a behaviour the scaler requires, so I think we can simplify and just return a message from any connection.
2. I'm not sure I understand your definition of DEALER. It doesn't seem to have any relation to ZMQ's dealer, and doesn't seem meaningfully different to your ROUTER. I think this type should round-robin sending, like ZMQ's dealer does.
3. UNICAST seems ok. Seems like a replacement for ZMQ's pair, but the name is accurate. In ZMQ, pair sockets block when muted, though I'm not sure if that is important for the scaler or not, but it might be.
4. MULTICAST seems like ZMQ's pub, and it sounds ok.
5. I'm not sure why we would call the user's callback at different times depending on the socket type. I think that might make it harder to use?

[gxuu]

(1), (2), (3) as per @sharpener6 's instruction. ROUTER and DEALER are not meant to mimic what ZMQ does, but I couldn't figure out a better name.
(5) because it's easier to implement. Also user generally don't care (do they?) if that message has been sent, as the nature of these messages is that they only make sense if they would be picked up by client immediately (real time statistics).

[magniloquency]

5. I'm not sure, maybe it doesn't matter? We could provide a "flush" operation if we really need it. In that case, we might not need a callback for send at all?

[gxuu]

Yea I don't think we need the callbacks for (5) if they always succeed. Primary reason to have a callback is to make the implementation easier as we don't need to worry about the callback ever being nil. That way we don't need to write if(callback) callback() everywhere.

[magniloquency]

Hmm that's true actually, we still need a way to return errors.. As for if (callback) callback(), we could wrap that in a method or function call, if we want.

[rafa-be]

Had a call with @gxuu, here is our updated table:

IOSocketType	Guaranteed delivery	send	receive	# conns
BINDER	Yes	send(dest, msg)	src, msg = recv()	0-N
CONNECTOR	Yes	send(msg)	msg = recv()	0-1
MULTICAST	No	send(channel, msg)	N.A.	0-N
UNICAST	No	N.A.	msg = recv()	0-1

• We renamed ROUTER into BINDER, and DEALER into CONNECTOR, that better match the names used internally within Scaler;
• MULTICAST is a publisher (should it be named PUB?), while UNICAST connects to a MULTICAST (should be renamed SUB?);
• UNICAST could technically send messages to the MULTICAST pair, but we disable it.

In addition, there is one place in Scaler where a ZMQ.Pair socket is used, between the client and its agent.
I don't think we should worry too much about that one, as we could either rewrite it using Binder + Connector, or using a pipe-like object.

[gxuu]

The above definitions posted by @rafa-be are now implemented in the latest commit of PR #149. The behavior follows:

• BINDER and CONNECTOR behaved similar as the original implementation. With the exception that CONNECTOR ignores the address that user may pass in, replace it with empty string as the unique identity.
• MULTICAST send completes immediately, and is always successful. Messages are not queued.
• UNICAST still queues the recv request, because we don't want the user to write a busy loop. Upon disconnects, queued requests are completed with "disconnected" error or something like that. I.e. no message guaranteed delivery after reconnect.

@rafa-be @magniloquency

[magniloquency]

This breaks with some of the semantics of ZMQ, and I don't know which of those the scaler relies upon (e.g. if it relies on guaranteed message delivery, different mute behaviours of different socket types, etc.). I think that things are simpler when we keep the interface across socket types consistent, but as long as it works with the scaler, I'm fine with it

[rafa-be]

This breaks with some of the semantics of ZMQ

Are you referring to the Unicast/Multicast sockets not guaranteeing delivery?

We reviewed Scaler's use of these sockets with @gxuu, and these semantics match what Scaler needs.

That being said, guaranteeing delivery or not for these Unicast/Multicast sockets do not matter much I think. If it's easier to make all sockets guaranteeing delivery, then its fine too. @gxuu ?

[magniloquency]

Are you referring to the Unicast/Multicast sockets not guaranteeing delivery?

I'm referring to a few things, but as long as none of them break the scaler I'm fine with it.

Is there a reason to only guarantee delivery for some of the socket types? Why not guarantee it for all socket types?

How do these socket types handle muting (when a socket has no peers)? In zmq, socket types either block or drop messages on mute. For example, router drops messages that are sent to an address which is not connected, and naturally this means that it drops on mute as well.

In zmq, I think send() only completes once the message has actually been written over all the relevant tcp sockets. I think it would be a good idea for us to maintain this behaviour in our transport layer for correctness, otherwise it means something like the following won't work:

# send message on MULTICAST socket
# note: returns before message is actually sent
await socket.send(Message(payload=b"hello, world"))

# the program might exit without the message being sent
# there is no way to guarantee it was written
sys.exit(0)

[gxuu]

@rafa-be

That being said, guaranteeing delivery or not for these Unicast/Multicast sockets do not matter much I think. If it's easier to make all sockets guaranteeing delivery, then its fine too. @gxuu ?

@magniloquency

Is there a reason to only guarantee delivery for some of the socket types? Why not guarantee it for all socket types?

Harder to implement; Also in the use case it is used for monitors. When you monitor you would like to see real time data instead of data from the back. So it doesn't make too much sense to guarantee delivery with that.

@magniloquency

How do these socket types handle muting (when a socket has no peers)? In zmq, socket types either block or drop messages on mute. For example, router drops messages that are sent to an address which is not connected, and naturally this means that it drops on mute as well.

First two blocks (in the sense that they never call the user provided callback). Last two drops (in the sense that they call user provided callback immediately). The first two types always queued the message, as there's no mute/onmute notion in YMQ.

@magniloquency

[...]maintain this behaviour in our transport layer for correctness, otherwise it means something like the following won't work:[...]

Don't do this. IOSocket are expected to be removed from the core by calling IOContext::removeIOSocket. Upon destruction, messages not sent out will be printed or logged so the user know those messages were not written. Again, for monitoring, we don't really care whether a message has been written.