Skip to content

[Example] One shot all reduce #245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 8, 2025
Merged

[Example] One shot all reduce #245

merged 1 commit into from
Aug 8, 2025

Conversation

joydddd
Copy link
Contributor

@joydddd joydddd commented Jul 8, 2025 •
edited
Loading

joydddd added a commit that referenced this pull request Jul 8, 2025
@joydddd
One shot all reduce & symm mem sync
19e6b5c 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from cd18d79 to 19e6b5c Compare July 8, 2025 19:07
This was referenced Jul 8, 2025
Merged
Merged
Merged
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 8, 2025
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 8, 2025 20:23
joydddd added a commit that referenced this pull request Jul 8, 2025
@joydddd
One shot all reduce & symm mem sync
2c79dd9 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 19e6b5c to 2c79dd9 Compare July 8, 2025 20:23
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 8, 2025 20:24
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 8, 2025 20:29
joydddd added a commit that referenced this pull request Jul 8, 2025
@joydddd
One shot all reduce & symm mem sync
374dfa3 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 2c79dd9 to 374dfa3 Compare July 8, 2025 20:29
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 8, 2025 20:29
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 18:19
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
1887088 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 374dfa3 to 1887088 Compare July 9, 2025 18:20
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 9, 2025 18:20
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 19:51
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
f3199f8 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 1887088 to f3199f8 Compare July 9, 2025 19:51
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 9, 2025 19:51
@joydddd joydddd force-pushed the joydddd/stack/11 branch from 6815f03 to 3f492ce Compare July 9, 2025 19:52
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
027a16b 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from f3199f8 to 027a16b Compare July 9, 2025 19:52
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 21:27
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
2f52133 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 027a16b to 2f52133 Compare July 9, 2025 21:28
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 9, 2025 21:28
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 21:30
joydddd added a commit that referenced this pull request Aug 4, 2025
@joydddd
One shot all reduce Example
a76965c 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd changed the base branch from main to joydddd/stack/13 August 4, 2025 21:22
@joydddd joydddd changed the base branch from joydddd/stack/13 to main August 4, 2025 21:22
@joydddd joydddd changed the base branch from main to joydddd/stack/13 August 4, 2025 21:23
@joydddd joydddd changed the base branch from joydddd/stack/13 to main August 4, 2025 21:44
@joydddd joydddd changed the base branch from main to joydddd/stack/13 August 4, 2025 21:44
jansel
jansel reviewed Aug 5, 2025
View reviewed changes
examples/all_reduce.py


@helion.jit(
config=helion.Config(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we able to autotune this yet?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. Unfortunately.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the blockers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need support for collaborative autotuning on multiple torchrun initiated processes.

I have event-based benchmarking infra ready in #393 (autotuner/benchmarker) which reports timing results on process 0.

We need to:

  1. Make sure all processes benchmark the same configs in the same order. (Is there any randomization in the autotuning process?)
  2. Use the event based benchmarker when in torchrun env inside autotuner. (easy)
  3. Communicate results from process 0 to all processes, OR process 0 makes a decision and communicate the optimal config to all processes. (Through caching?)

joydddd added a commit that referenced this pull request Aug 5, 2025
@joydddd
One shot all reduce Example
15b3f75 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd changed the base branch from joydddd/stack/13 to main August 5, 2025 18:52
@joydddd joydddd changed the base branch from main to joydddd/stack/13 August 5, 2025 20:23
@joydddd joydddd changed the base branch from joydddd/stack/13 to main August 5, 2025 20:39
@joydddd joydddd changed the base branch from main to joydddd/stack/13 August 5, 2025 20:39
@joydddd joydddd changed the base branch from joydddd/stack/13 to main August 5, 2025 20:44
@joydddd joydddd changed the title One shot all reduce Example [Example] One shot all reduce Aug 5, 2025
@joydddd joydddd changed the base branch from main to joydddd/stack/13 August 5, 2025 20:44
joydddd added a commit that referenced this pull request Aug 5, 2025
@joydddd
[Example] One shot all reduce
2c0a1be 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd changed the base branch from joydddd/stack/13 to main August 5, 2025 20:47
@joydddd joydddd force-pushed the joydddd/stack/12 branch 2 times, most recently from cb9d73e to 4273b27 Compare August 5, 2025 22:36
@joydddd
[Example] One shot all reduce
abf1f4b 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd merged commit ca3f4d6 into main Aug 8, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jansel jansel jansel left review comments

@yf225 yf225 Awaiting requested review from yf225

@drisspg drisspg Awaiting requested review from drisspg

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@joydddd @jansel @facebook-github-bot