Skip to content

Support multimodule communication #2031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yaoyu-33 merged 23 commits into from
Jan 24, 2026
Merged

Support multimodule communication #2031

yaoyu-33 merged 23 commits into from
Jan 24, 2026

Conversation

@yaoyu-33
Copy link
Contributor

@yaoyu-33 yaoyu-33 commented Oct 29, 2025

This is a ported gitlab PR 4069. Expert review is done, needs final review.

shifangx and others added 17 commits October 5, 2025 16:59
@shifangx
Support multimodule communication
5aec371
@shifangx
add test
01f2b4f
@shifangx
add test for MultiModulePipelineCommunicator init
3425fac
@shifangx
add test for compute_total_pipeline_stages
cc2b6ee
@shifangx
add test_send_forward_recv_forward
e940fcb
@shifangx
add test_send_backward_recv_backward
b689f16
@shifangx
add test_send_forward_recv_backward_send_backward_recv_forward
9565a98
@shifangx
skip if torch version is low
381986a
@shifangx
delete duplicated dist.broadcast in BridgeCommunicator
f064f05
@shifangx
add test_send_forward_recv_forward_with_transformer_blocks
11ae642
@shifangx
add test_send_forward_recv_forward_with_transformer_blocks_and_differ…
698e454 
…ent_parallelisms
chore: Format files
babbb54
@shifangx
update comment and logging
2ac3216
@shifangx
rename from pp_stage to pp_rank, and import some unit function from t…
e9d03ba 
…est_bridge_communicator
@shifangx
add test for the case when pp(image_encoder) != pp(audio_encoder)
7c10cb0
chore: Format files
b28287f
@shifangx
add test with llm pp size larger than 2, such as llm pp=4
fbeb6e9
@yaoyu-33 yaoyu-33 requested review from a team as code owners October 29, 2025 18:27
@copy-pr-bot
Copy link

copy-pr-bot bot commented Oct 29, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

jaredcasper
jaredcasper reviewed Oct 29, 2025
View reviewed changes
@yashaswikarnati yashaswikarnati mentioned this pull request Nov 13, 2025
Open
6 tasks
@yaoyu-33 yaoyu-33 requested review from a team as code owners November 13, 2025 20:23
@yanring yanring added the Expert Review Apply this label to indicate that your PR is ready for expert review. label Nov 19, 2025
@yanring yanring added this to the Core 0.16 milestone Nov 19, 2025
@shifangx
Copy link
Contributor

shifangx commented Jan 6, 2026

hi, @ko3n1g, can you help to check why this pr CI test failed for Codeovers approval?
https://github.com/NVIDIA/Megatron-LM/actions/runs/20747699475/job/59568935015?pr=2031

you and yash, jared have approved, but I donot know why the Codeovers approval test still failed.

@ko3n1g
Copy link
Contributor

ko3n1g commented Jan 6, 2026

core-nemo is missing

@shifangx
Copy link
Contributor

shifangx commented Jan 7, 2026

core-nemo is missing

Thanks @ko3n1g

@shifangx
Copy link
Contributor

shifangx commented Jan 7, 2026 •
edited
Loading

Hi, @pablo-garay, Could you help explain what this error message means? I am trying to fix this error, but I am not sure what is the root cause of this error.
https://github.com/NVIDIA/Megatron-LM/actions/runs/20747699360/job/59568927927?pr=2031
This pr does not change megatron.core.model_parallel_config.ModelParallelConfig.init , but it is mentioned in error message.

Change 1: megatron.core.model_parallel_config.ModelParallelConfig.init(pipeline_dtype)
Clean: megatron.core.model_parallel_config.ModelParallelConfig.init
Clean repr: 'megatron.core.model_parallel_config.ModelParallelConfig.init'
✗ NO MATCH

@shifangx
Copy link
Contributor

shifangx commented Jan 7, 2026

/ok to test 19f0ecd

@yaoyu-33
Copy link
Contributor Author

yaoyu-33 commented Jan 16, 2026

/ok to test 2fe2eeb

@shifangx
Copy link
Contributor

shifangx commented Jan 24, 2026

/ok to test 762ec65

@yaoyu-33 yaoyu-33 added this pull request to the merge queue Jan 24, 2026
Merged via the queue into NVIDIA:main with commit 485ed18 Jan 24, 2026
52 of 54 checks passed
@yaoyu-33 yaoyu-33 deleted the shifang/multi_module_comm branch January 24, 2026 04:05
ko3n1g added a commit to ko3n1g/Megatron-LM that referenced this pull request Jan 24, 2026
@ko3n1g
Revert "Support multimodule communication (NVIDIA#2031)"
913178e 
This reverts commit 485ed18.
ko3n1g added a commit that referenced this pull request Jan 24, 2026
@ko3n1g
Revert "Support multimodule communication (#2031)" (#3068)
0972f02
ko3n1g added a commit that referenced this pull request Jan 27, 2026
@ko3n1g
Reapply "Support multimodule communication (#2031)" (#3068)
35e85a6 
This reverts commit 0972f02.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jaredcasper jaredcasper jaredcasper approved these changes

@ericharper ericharper ericharper approved these changes

@ko3n1g ko3n1g ko3n1g approved these changes

@yashaswikarnati yashaswikarnati yashaswikarnati approved these changes

Assignees

No one assigned

Labels

complexity: low Run tests

Projects

None yet

Milestone

Core 0.16

Development

Successfully merging this pull request may close these issues.

7 participants

@yaoyu-33 @shifangx @ko3n1g @jaredcasper @ericharper @yashaswikarnati @yanring