Distance based latency #858
Open
+447
−483
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D80141665 |
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 14, 2025
Summary: Now that the simnet has awareness of which compute resource each ProcId maps to, when messages are being sent we can simply look at the sender and destination ProcIds and compute the distance the message is being sent in order to determine the latency. Latency is randomly sample from a beta distribution where the min and max for each distance is configured Implementation details (follow along numbers in comments): 1. In the previous diff when Procs were allocated, their coordinates (region, dc, zone, rack, host, gpu) were registered to the Simnet 2. When SimTx posts a message, we can safely assume that it is a MessageEnvelope. MessageEnvelopes contain information about the sender and receiver so we can determine which ProcIds the message is being sent between, which in turn means we can identify which coordinates they are being sent between 3. We determine distance between 2 coordinates by identifying the most major dimension in which they differ 4. We create a struct called LatencyConfig which holds a distribution for sampling, as well as minimum and maximum values for each distance. 5. We use the identified distance to get a sample for what the latency should be for that send 6. We pass in that latency to the MessageDeliveryEvent to use as its duration 7. The old network configuration which was an all-to-all map of edges with latencies between nodes has been removed along with all related structs 8. Unit tests have been refactored such that when we need a particular message to be sent with a particular latency, we register the ProcIds with the appropriate coordinates, and configure the interdistance latency test_allocator_registers_resources in alloc/sim.rs demonstrates that when we allocate a ProcMesh using the sim allocator, our Procs are registered as compute resources and the latencies are computed based on distance Differential Revision: D80141665
thomasywang
force-pushed
the
export-D80141665
branch
from
a44e6bf to
21ec77b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D80141665 |
thomasywang
force-pushed
the
export-D80141665
branch
from
21ec77b to
667531b
Compare
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 14, 2025
Summary: Pull Request resolved: meta-pytorch#858 Now that the simnet has awareness of which compute resource each ProcId maps to, when messages are being sent we can simply look at the sender and destination ProcIds and compute the distance the message is being sent in order to determine the latency. Latency is randomly sample from a beta distribution where the min and max for each distance is configured Implementation details (follow along numbers in comments): 1. In the previous diff when Procs were allocated, their coordinates (region, dc, zone, rack, host, gpu) were registered to the Simnet 2. When SimTx posts a message, we can safely assume that it is a MessageEnvelope. MessageEnvelopes contain information about the sender and receiver so we can determine which ProcIds the message is being sent between, which in turn means we can identify which coordinates they are being sent between 3. We determine distance between 2 coordinates by identifying the most major dimension in which they differ 4. We create a struct called LatencyConfig which holds a distribution for sampling, as well as minimum and maximum values for each distance. 5. We use the identified distance to get a sample for what the latency should be for that send 6. We pass in that latency to the MessageDeliveryEvent to use as its duration 7. The old network configuration which was an all-to-all map of edges with latencies between nodes has been removed along with all related structs 8. Unit tests have been refactored such that when we need a particular message to be sent with a particular latency, we register the ProcIds with the appropriate coordinates, and configure the interdistance latency test_allocator_registers_resources in alloc/sim.rs demonstrates that when we allocate a ProcMesh using the sim allocator, our Procs are registered as compute resources and the latencies are computed based on distance Differential Revision: D80141665
Summary: buck run fbcode//monarch/examples/meta/paft:actor_paft -- --mesh_type thread --group_size 250 --max_groups 10 --min_healthy 7 Differential Revision: D78917462
Summary: There was an open TODO to remove the global mailbox for SimClock. We don't actually even need mailboxes for sim clock and a oneshot works just fine Differential Revision: D80029571
Summary: When we increase the number of actors in our simulation it takes longer for all the events at a certain time to complete so we need to wait for longer. If we wait to long then the simulation just runs slower than it needs to so its nice to make this configurable. In the long term we will come up with a more robust solution to this but in the meantime that is not a priority. See EX528476 to understand the underlying problem the debounce is remedying Differential Revision: D80137965
Summary: The sim allocator will now register the location (region, dc, zone, rack, host, gpu) of every ProcId upon creation with the simnet. Differential Revision: D80137963
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 14, 2025
Summary: Now that the simnet has awareness of which compute resource each ProcId maps to, when messages are being sent we can simply look at the sender and destination ProcIds and compute the distance the message is being sent in order to determine the latency. Latency is randomly sample from a beta distribution where the min and max for each distance is configured Implementation details (follow along numbers in comments): 1. In the previous diff when Procs were allocated, their coordinates (region, dc, zone, rack, host, gpu) were registered to the Simnet 2. When SimTx posts a message, we can safely assume that it is a MessageEnvelope. MessageEnvelopes contain information about the sender and receiver so we can determine which ProcIds the message is being sent between, which in turn means we can identify which coordinates they are being sent between 3. We determine distance between 2 coordinates by identifying the most major dimension in which they differ 4. We create a struct called LatencyConfig which holds a distribution for sampling, as well as minimum and maximum values for each distance. 5. We use the identified distance to get a sample for what the latency should be for that send 6. We pass in that latency to the MessageDeliveryEvent to use as its duration 7. The old network configuration which was an all-to-all map of edges with latencies between nodes has been removed along with all related structs 8. Unit tests have been refactored such that when we need a particular message to be sent with a particular latency, we register the ProcIds with the appropriate coordinates, and configure the interdistance latency test_allocator_registers_resources in alloc/sim.rs demonstrates that when we allocate a ProcMesh using the sim allocator, our Procs are registered as compute resources and the latencies are computed based on distance Differential Revision: D80141665
thomasywang
force-pushed
the
export-D80141665
branch
from
667531b to
d4889c9
Compare
Summary: Pull Request resolved: meta-pytorch#858 Now that the simnet has awareness of which compute resource each ProcId maps to, when messages are being sent we can simply look at the sender and destination ProcIds and compute the distance the message is being sent in order to determine the latency. Latency is randomly sample from a beta distribution where the min and max for each distance is configured Implementation details (follow along numbers in comments): 1. In the previous diff when Procs were allocated, their coordinates (region, dc, zone, rack, host, gpu) were registered to the Simnet 2. When SimTx posts a message, we can safely assume that it is a MessageEnvelope. MessageEnvelopes contain information about the sender and receiver so we can determine which ProcIds the message is being sent between, which in turn means we can identify which coordinates they are being sent between 3. We determine distance between 2 coordinates by identifying the most major dimension in which they differ 4. We create a struct called LatencyConfig which holds a distribution for sampling, as well as minimum and maximum values for each distance. 5. We use the identified distance to get a sample for what the latency should be for that send 6. We pass in that latency to the MessageDeliveryEvent to use as its duration 7. The old network configuration which was an all-to-all map of edges with latencies between nodes has been removed along with all related structs 8. Unit tests have been refactored such that when we need a particular message to be sent with a particular latency, we register the ProcIds with the appropriate coordinates, and configure the interdistance latency test_allocator_registers_resources in alloc/sim.rs demonstrates that when we allocate a ProcMesh using the sim allocator, our Procs are registered as compute resources and the latencies are computed based on distance Differential Revision: D80141665
|
This pull request was exported from Phabricator. Differential Revision: D80141665 |
thomasywang
force-pushed
the
export-D80141665
branch
from
d4889c9 to
85d1018
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Now that the simnet has awareness of which compute resource each ProcId maps to, when messages are being sent we can simply look at the sender and destination ProcIds and compute the distance the message is being sent in order to determine the latency.
Latency is randomly sample from a beta distribution where the min and max for each distance is configured
Implementation details (follow along numbers in comments):
test_allocator_registers_resources in alloc/sim.rs demonstrates that when we allocate a ProcMesh using the sim allocator, our Procs are registered as compute resources and the latencies are computed based on distance
Differential Revision: D80141665