Make NetTx<->NetRx handshake two-way to prevent errant reconnections #793
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 7, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 7, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 7, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 8, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 8, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 8, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
8302f53 to
de5a4da
Compare
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 8, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
de5a4da to
9b7a8c8
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 8, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
9b7a8c8 to
070b3ad
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 8, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
2 times, most recently
from
c817b33 to
2241ac2
Compare
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 12, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 12, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
2241ac2 to
2794dd2
Compare
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 12, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 12, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 12, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
2794dd2 to
7667e2c
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 12, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
7667e2c to
03c06e8
Compare
samlurye
force-pushed
the
export-D79607092
branch
from
03c06e8 to
efce7a5
Compare
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 13, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Reviewed By: mariusae Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 13, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Reviewed By: mariusae Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
efce7a5 to
b2495c2
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 13, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Reviewed By: mariusae Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
b2495c2 to
4f2b17b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 13, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Reviewed By: mariusae Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
2 times, most recently
from
31d9002 to
1c87225
Compare
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 19, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 19, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 19, 2025
…eta-pytorch#793) Summary: The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 19, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
samlurye
force-pushed
the
export-D79607092
branch
from
1c87225 to
2711802
Compare
samlurye
added a commit
to samlurye/monarch-1
that referenced
this pull request
Aug 19, 2025
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092 Reviewed By: mariusae
…eta-pytorch#793) Summary: Pull Request resolved: meta-pytorch#793 The purpose of this diff is to handle the following scenario: 1. Process A starts serving a NetRx. 2. Process B creates a NetTx that connects to process A's NetRx. 3. B sends a few messages to A, and the messages are acked. 4. Process A dies/is killed, while B stays alive. 5. A new Process C starts serving a NetRx on the same channel as from step 1. 6. B's NetTx connects to C's NetRx, *with no way of knowing it has connected to a different process than before*. 7. B sends messages to C, starting from where it left off with A. 8. C rejects all of B's messages because of invalid sequence numbers. 9. B's NetTx eventually times out after a long time with no acks. In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages. Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller. Differential Revision: D79607092
|
This pull request was exported from Phabricator. Differential Revision: D79607092 |
samlurye
force-pushed
the
export-D79607092
branch
from
2711802 to
de46bac
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The purpose of this diff is to handle the following scenario:
In order to distinguish among connections from different NetTx instances to the same NetRx instance, each NetTx generates a random unique session id. This session id gets sent as part of an initial handshake from NetTx -> NetRx before the NetTx starts sending normal messages.
Currently, though, NetTx doesn't wait for any handshake before starting to send messages. To resolve the issue described above, this diff introduces a global (per-process) "rx session id". When a NetTx first connects to a NetRx, the NetRx responds with its rx session id as part of the handshake. The NetTx waits for the handshake response and extracts the rx session id. If this is the first time the NetTx is connecting, the NetTx stores the rx session id. On subsequent connection attempts, the NetTx will validate the rx session id it receives from the handshake against the rx session id it previously stored; if there is a mismatch, the NetTx returns the appropriate error to its caller.
Differential Revision: D79607092