tuned: use tree instead of bruck at scale [v6.0.x] #13481
Closed
+1
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The switch from tree to bruck between 512 and 1023 processes leads to unexpected latency changes in benchmarks of other collectives. We should be consistent here. There is no good reason for why bruck would perform better in that range but not beyond. Signed-off-by: Joseph Schuchart <[email protected]> (cherry picked from commit 9bd7757)
|
Honestly I am extremely cautious about this change, because it is not based on a valid experiment. I'm not saying that @devreal runs are not correct, I am saying that data is not enough to justify such a change because that timing is highly dependent on the network and the process placement and it's match (or mismatch) with the topology used in the barrier algorithm. I have yet to see an in depth analysis of such data related to this change. |
|
@devreal since you opened the revert - do you want to close this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The switch from tree to bruck between 512 and 1023 processes leads to unexpected latency changes in benchmarks of other collectives. We should be consistent here. There is no good reason for why bruck would perform better in that range but not beyond.
Backport of #11023 to v6.0.x