[AICOMRCCL-350] Allow for advanced tuning of P2P channels/part mapping via RCCL_P2P_SHIFT_SIZE #2999
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This feature allows for further investigation to see if balancing channels across XCCs can yield improved performance.
Technical Details
Addition of new complementary ncclP2pChannelForPart / ncclP2pChannelToPart functions that take in a shiftSize variable that adjusts how work is assigned.
Modifications to the two key functions listed above, then additions to ncclComm / ncclDevComm to carry this new information.
A new RCCL environment variable to adjust, as well as a function for properly setting the value / establishing default shiftSize.
This PR was originally created by Gilbert Lee: PR2067
JIRA ID
AICOMRCCL-350
Test Plan
Correctness tests on gfx942 with MX and AINIC.
Perf. comparisons with RCCL_P2P_SHIFT_SIZE values of "1,2,3,4,5" compared to default run and RCCL develop branch.
Test Result
On 4-node AINIC runs, RCCL_P2P_SHIFT_SIZE =1 improved peak bw.
Submission Checklist