-
Notifications
You must be signed in to change notification settings - Fork 59
Commit 45d60ed
Make NetTx<->NetRx handshake two-way to prevent errant reconnections (#793)
Summary:
The purpose of this diff is to handle the following scenario:
1. Process A starts serving a NetRx.
2. Process B creates a NetTx that connects to process A's NetRx.
3. B sends a few messages to A, and the messages are acked.
4. Process A dies/is killed, while B stays alive.
5. A new Process C starts serving a NetRx on the same channel as from step 1.
6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*.
7. B sends messages to C, starting from where it left off with A.
8. C rejects all of B's messages because of invalid sequence numbers.
9. B's NetTx eventually times out after a long time with no acks.
In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages.
Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller.
Differential Revision: D796070921 parent ddfb022 commit 45d60edCopy full SHA for 45d60ed
File tree
Expand file treeCollapse file tree
1 file changed
+292
-27
lines changedFilter options
- hyperactor/src/channel
Expand file treeCollapse file tree
1 file changed
+292
-27
lines changed
0 commit comments