Skip to content

Enable MoE TP with thunderfx #2611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: dtensor-tp-colwise-test
Choose a base branch
from
Draft

Enable MoE TP with thunderfx #2611

wants to merge 2 commits into from
+97 −35

Conversation

kshitij12345
Copy link
Collaborator

@kshitij12345 kshitij12345 commented Oct 7, 2025 •
edited
Loading

Depends on #2598

Changes

  • Allow nvFuser executor to handle AsyncCollectiveTensor as it may show up when using programs with DTensor.
  • Update nvFuser executor to create separate fusion region (and FusionDefinition) for multi-device and single-device.
  • Update dtensor_mul to correctly handle broadcasting and type promotion.
  • Add dtensor_silu dtensor symbol composed of existing prims.
  • Enable thunderfx path for the distributed test_moe.py.

Generated trace from thunderfx - https://gist.github.com/kshitij12345/c02ecef47a9974216d8350bb0e0db4ca

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mruberry mruberry Awaiting requested review from mruberry mruberry will be requested when the pull request is marked ready for review mruberry is a code owner

@lantiga lantiga Awaiting requested review from lantiga lantiga will be requested when the pull request is marked ready for review lantiga is a code owner

@t-vi t-vi Awaiting requested review from t-vi t-vi will be requested when the pull request is marked ready for review t-vi is a code owner

@KaelanDt KaelanDt Awaiting requested review from KaelanDt KaelanDt will be requested when the pull request is marked ready for review KaelanDt is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kshitij12345