Auto Reconnect not working on ROUTER-ROUTER pattern

[Jeducious]

Description of problem

Background

I have a distributed computing app of which I am an author/maintainer (crowdrender - distributed rendering for Blender).
I have recently been moving the core of this app to a different architecture, one based on event driven patterns. There are two components that are essential, one is an event bus, implemented using PUB/SUB, and another for streaming binary data (files, binary blobs, streams etc) which uses the ROUTER-ROUTER pattern for peer to peer streaming of data.

Problem

One of the common problems I want the app to handle is a peer going to sleep. When this happens I see that the PUB/SUB channel, after a timeout, notices that the sleeping peer has dropped its connection, and begins the automatic recovery, here's what I see on that channel for the PUB and SUB sockets for the connecting peer (the sleeping peer is the one that binds in this example)

2025-04-26 22:39:16,445 ; Client EventBusLogger ; INFO ;None; EVENT:{ b'clients'.b'eventbusbridge.socket_event'; body: ZMQEvent: DISCONNECTED, endpoint: tcp://192.168.1.145:9691, value 3872 }
2025-04-26 22:39:18,366 ; Client EventBusLogger ; INFO ;None; EVENT:{ b'clients'.b'eventbusbridge.socket_event'; body: ZMQEvent: DISCONNECTED, endpoint: tcp://192.168.1.145:9697, value 3852 }
2025-04-26 22:42:52,792 ; Client EventBusLogger ; INFO ;None; EVENT:{ b'clients'.b'eventbusbridge.socket_event'; body: ZMQEvent: CLOSED, endpoint: tcp://[::ffff:192.168.1.145]:9691, value 3812 }
2025-04-26 22:42:52,792 ; Client EventBusLogger ; INFO ;None; EVENT:{ b'clients'.b'eventbusbridge.socket_event'; body: ZMQEvent: CONNECT_RETRIED, endpoint: tcp://[::ffff:192.168.1.145]:9691, value 5000 }
2025-04-26 22:42:54,720 ; Client EventBusLogger ; INFO ;None; EVENT:{ b'clients'.b'eventbusbridge.socket_event'; body: ZMQEvent: CLOSED, endpoint: tcp://[::ffff:192.168.1.145]:9697, value 4196 }
2025-04-26 22:42:54,721 ; Client EventBusLogger ; INFO ;None; EVENT:{ b'clients'.b'eventbusbridge.socket_event'; body: ZMQEvent: CONNECT_RETRIED, endpoint: tcp://[::ffff:192.168.1.145]:9697, value 5000 }

The attempts to reconnect continue forever on the PUB-SUB channel. However, on the ROUTER-ROUTER channel, although the socket events give a DISCONNECTED event, there are NO attempts at all to reconnect.

Environment and configuration

Environment

I am using a windows peer on the connecting side running windows 11, and on the binding peer's side, running MacOS. I am using the pyzmq bindings, version 26.4 which runs libzmq 4.3.5. on both machines. I am using python 3.11.11 installed in Blender.

Socket configuration

For the ROUTER-ROUTER socket configuration I am using the following options;

# Connecting peer 
_out_sock.setsockopt(zmq.IPV6, 1)
_out_sock.setsockopt(zmq.CONNECT_TIMEOUT, 10 * 1000)
_out_sock.setsockopt(zmq.RECONNECT_IVL_MAX, 5000)
_out_sock.setsockopt(zmq.ROUTER_MANDATORY, 1)
_out_sock.setsockopt(zmq.ROUTER_HANDOVER, 1)
_out_sock.curve_publickey = self.cr_auth_ctx.client_public_key
_out_sock.curve_secretkey = self.cr_auth_ctx.client_secret_key

# Binding Peer
_out_sock.setsockopt(zmq.IPV6, 1)
_out_sock.setsockopt(zmq.CONNECT_TIMEOUT, 10 * 1000)
_out_sock.setsockopt(zmq.RECONNECT_IVL_MAX, 5000)
_out_sock.setsockopt(zmq.ROUTER_MANDATORY, 1)
_out_sock.setsockopt(zmq.ROUTER_HANDOVER, 1)
_out_sock.curve_publickey = self.cr_auth_ctx.client_public_key
_out_sock.curve_secretkey = self.cr_auth_ctx.client_secret_key
_out_sock.curve_server = True

NOTE: These sockets connect and disconnect perfectly fine, with the expected socket events according to the protocol specification. The connect and binding peer both setup their own keys, supplied by an auth object which guarantees a unique set of keys for each peer.

Expected vs Actual behaviour

Expected Behaviour

If the binding peer is put or allowed to go into sleep mode, the connecting peer detects the drop in connection, issues a ZMQ_EVENT_DISCONNECTED event and then attempts to reconnect, evidenced by ZMQ_EVENT_CONNECT_RETRIED events corresponding to the endpoint of the sleeping peer.

Actual Behaviour

When the binding side goes to sleep, sometimes there is a DISCONNECTED event, sometimes not, there are NO ZMQ_EVENT_CONNECT_RETRIED events at all. Viewing the list of TCP/IP connections for the connecting peer, shows that there are no TCP/IP connections listed for the sleeping peer's endpoints, whereas for the PUB/SUB channel, there are TCP/IP connections listed for the sleeping peer's endpoints for the PUB/SUB channel, which are showing SYN_SENT as the state.

Final Remarks

I have been using pyzmq for ten years, though I do not necessarily consider myself an expert. I would be glad of any corrections, requests for more information on my approach so that I can handle sleeping peers. If this is indeed a bug, I would be happy to contribute my time to helping with a fix, I am highly motivated to make this pattern work or substitute it for something more suitable.

Basically what I am saying is, sometimes, I'm an idiot, but I mean well, and I'm always grateful when someone helps me get past my own limitations of understanding 😄

Thanks!

[Jeducious]

Hi @bluca 😄

A polite request, if you have any capacity, would really appreciate your comments on whether this issue is a skill issue on my behalf, or if there really is a problem. Usually I find that 99.9% of the time, its a skill issue on my part, in which case, this issue will be an easy close.

Many thanks in advance :)

James

[shishirpy]

Hi @Jeducious, ROUTER-ROUTER pattern is talked about in the Freelance pattern in the guide. It also says.

There’s a small paradox here. We need to know when servers become connected and available for work. In the Freelance pattern, unlike the broker-based patterns we saw earlier in this chapter, servers are silent until spoken to. Thus we can’t talk to a server until it’s told us it’s online, which it can’t do until we’ve asked it.

This might be of some help - https://zguide.zeromq.org/docs/chapter4/#Brokerless-Reliability-Freelance-Pattern

[Jeducious]

Thanks @shishirpy, I've taken a look through that link, which, I think I've read before, but a long time ago, good to refresh my memory.

However, from what I've read (again correct me if you think I missed something here), that article on the Freelance pattern mainly talks about router socket identities and how to ensure that each peer's ROUTER socket can be aware of the other peer's identity so that messages can be prepended with the correct ID for the peer, and be properly routed.

That already is working, when the peers first connect, msgs can flow in both directions perfectly, rather the problem that I am seeing is a failure of the automatic reconnect feature of zmq when the binding side peer goes to sleep.

In my post above, I talked about the socket events for the ROUTER-ROUTER connection being different to the PUB/SUB connection. When binding peer goes to sleep, the PUB/SUB connection shows socket events indicating zmq is trying to reconnect the connecting peer to the now sleeping, binding peer. However, the ROUTER socket on the connecting peer shows no activity related to reconnection, and if the sleeping, binding peer comes back online, the connection is not re-established, leaving the system broken.

So the question here is, why does the auto-reconnect work in the PUB/SUB pattern, but not for the ROUTER/ROUTER pattern?

I assume the possible answers are:

1. I am an idiot, need to read the manual (betting on this one, it is the case 99% of the time)
2. Its a bug in zmq (not sure how likely this is given the extensive use of zmq, sure this would have been noticed be someone else?); and
3. the unknown unknown

Any help with this would be greatly appreciated as I currently can't handle sleeping peers in an automated fashion, I don't know, but maybe telling the user that the peer has disconnected and letting them manage that might be the best way out of this, but if there's a way to sensibly retry, that would be great, its just at the moment, I can't see a way of doing that as zmq doesn't appear to be reconnecting in the ROUTER/ROUTER pattern I've setup.

Cheers!