Merge main into dev by ilml · Pull Request #3865 · NVIDIA/Megatron-LM

ilml · 2026-03-13T19:54:25Z

What does this PR do ?

Sync the latest main branch changes into the development line and preserve ongoing dev-only work through the required conflict resolutions.

Summary

Pull the latest upstream main changes into the merge branch, including broad updates across CI/workflows, docs, examples, inference, FSDP/resharding, and test coverage.
Preserve active dev work around emerging optimizers and layer-wise optimizer refactoring, HyperConnection, Dynamic-CP / THD handling, MoE, MLA/MTP, and related attention and training paths.
Resolve merge conflicts in the key overlap areas so dev behavior is retained while adopting the latest main changes.
Keep the existing dev uv.lock during the merge because regenerating it with the installed uv currently fails on upstream nemo-run metadata.

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

Pre-checks

I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

When your PR is ready, click Ready for Review.
An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
- Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

Signed-off-by: Maanu Grover <maanug@nvidia.com>

…3579)

…DIA#3584)

…passes (NVIDIA#3525)

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: Maanu Grover <maanug@nvidia.com>

Co-authored-by: Xin Yao <xiny@nvidia.com>

Signed-off-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Xin Yao <xiny@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

…VIDIA#3591)

…IA#2765) Co-authored-by: Xin Yao <xiny@nvidia.com>

…VIDIA#3300) Co-authored-by: Xin Yao <xiny@nvidia.com>

Signed-off-by: nvidia <nvidia@TRY-64956-gpu01.nvidialaunchpad.com>

NVIDIA#3630)

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

… Manager patch, docs (NVIDIA#3507) Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Made-with: Cursor

ilml · 2026-03-19T22:12:40Z

/ok to test 8096db6

Preseed uv-created environments with setuptools and related build tools so git-source dependencies resolve reliably in CI, align the Transformer Engine source pin with the preserved lockfile, and restore the missing grouped_gemm helper that broke install-time imports. Made-with: Cursor

ilml · 2026-03-19T23:07:35Z

/ok to test c2cbe54

- Add InferenceGroupedMLP class to experts.py from main (was lost in merge conflict resolution while backends.py imported it) - Add megatron/core/inference/moe/ module from main (dependency of InferenceGroupedMLP) - Update mxfp8_tensor.py and add mxfp8_quantize.py from main (needed for triton backend support in inference MoE) - Fix duplicate autodoc items: remove duplicate _EMERGING_OPTIMIZERS placeholder, duplicate fsdp_all_gather_in_start_param_sync field, duplicate logger assignments in spec_utils.py and token_dispatcher.py - Fix docs warnings: replace H3 heading in moe_utils.py docstring with bold text, remove orphaned docs/source/api-guide/router_replay.md, remove redundant docs/api-guide/fine_grained_activation_offloading.md, exclude deepseek reproduce guide from Sphinx toctree check Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ilml · 2026-03-21T20:57:33Z

/ok to test 57a5344

- param_and_grad_buffer.py: keep layerwise optimizer all_gather path from main and dev's grad_enabled caching + no_grad wrapper for the standard distributed optimizer path - transformer_config.py: keep both fused_residual_rmsnorm (main) and use_transformer_engine_op_fuser (dev) config fields - test_mamba_moe_model.py: keep golden config entries from both branches Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ilml · 2026-03-21T21:01:57Z

/ok to test d057244

The process_mtp_loss function now passes input_ as keyword arg (from dev branch changes), but the test mock expected a positional 'hidden' parameter. Updated the mock signature to match. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ilml · 2026-03-21T21:37:05Z

/ok to test 1e9a599

ilml · 2026-03-21T22:39:30Z

/ok to test 1e9a599

The float32 variant deterministically times out with an NCCL ALLREDUCE timeout (SeqNum=361) in some CI shards while passing in others. The test and fusion code are identical to dev branch, indicating a pre-existing infrastructure issue with multi-GPU JIT compilation timing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ilml · 2026-03-21T23:35:42Z

/ok to test fc2d334

…nction The router.py passes dense_output=True for inference mode but the merge took dev's version of moe_utils.py which lacks this parameter. Added from main to fix TypeError in InferenceTopKRouter. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ilml · 2026-03-22T00:22:47Z

/ok to test 6521ee2

ilml · 2026-03-22T01:30:43Z

/ok to test bd44a67

The test was passing layer_wise_distributed_optimizer as a keyword arg to get_megatron_muon_optimizer(), but that function doesn't accept it. Set it on the OptimizerConfig object instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ilml · 2026-03-22T02:52:06Z

/ok to test 5c04917

Pass async_allgather and model_chunks from optimizer config to LayerWiseDistributedOptimizer constructor so overlap param gather works correctly with layer-wise optimizers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ilml · 2026-03-22T03:54:09Z

/ok to test d8caf0a

maanug-nv and others added 30 commits February 25, 2026 00:59

Move config src files into a dedicated dir (NVIDIA#3570)

b17248a

Signed-off-by: Maanu Grover <maanug@nvidia.com>

chore: rotate oncall schedule

9bb2375

Revert "remove encoder_and_decoder from enums (NVIDIA#3406)" (NVIDIA#…

0dba6a0

…3579)

Fix default cuda graph persist arg. Add persist to rl common.sh. (NVI…

b359981

…DIA#3584)

Optimize away add request overheads in dummy ep cuda-graphed forward …

60a25aa

…passes (NVIDIA#3525)

ci: Test docs build (NVIDIA#3583)

e7d80bd

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Fix docs build for release (NVIDIA#3597)

80ef2ae

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Remove secrets (NVIDIA#3598)

2561a59

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Define secrets (NVIDIA#3599)

19444b9

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: gh-release-from-tag (NVIDIA#3600)

a08ad25

Signed-off-by: oliver könig <okoenig@nvidia.com>

Ko3n1g/ci/remove twine username (NVIDIA#3601)

ac449c4

Signed-off-by: oliver könig <okoenig@nvidia.com>

Add training code to MCore wheel (NVIDIA#3573)

9088d4f

Signed-off-by: Maanu Grover <maanug@nvidia.com>

FP8 attention knob for nvFP4 recipe (NVIDIA#3363)

dd4c0b7

Co-authored-by: Xin Yao <xiny@nvidia.com>

Fix error with --load-main-params-from-ckpt (NVIDIA#3569)

c2782fd

Signed-off-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Xin Yao <xiny@nvidia.com>

ci: Create comment (NVIDIA#3610)

c921891

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Skip cleanup-taint-node jobs during deployments (NVIDIA#3612)

9100119

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: No comment for release workflow (NVIDIA#3615)

6161f7a

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Re-add release tag prefix (NVIDIA#3619)

36d8a9d

Signed-off-by: oliver könig <okoenig@nvidia.com>

docs: Fix version picker urls (NVIDIA#3621)

1b1f5c4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

ci: Increase changelog generation max PRs fetched (NVIDIA#3620)

027e0f3

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Add debug info to an assert. (NVIDIA#3588)

18c94d2

fix: async_utils: explicit GC in persistent checkpoint worker loop (N…

e1a9ac9

…VIDIA#3591)

Fix: Perform sigmoid calculation in fp32 for aux loss stability (NVID…

5f668c1

…IA#2765) Co-authored-by: Xin Yao <xiny@nvidia.com>

fix: forward use_te_activation_func flag in non-MoE GPT layer spec (N…

d3c10df

…VIDIA#3300) Co-authored-by: Xin Yao <xiny@nvidia.com>

Multimodal: Limit transformer version in Dockerfile (NVIDIA#3448)

7e3f670

Signed-off-by: nvidia <nvidia@TRY-64956-gpu01.nvidialaunchpad.com>

Track and plot per-token off-policy in RL (NVIDIA#3515)

6c0b9c6

Revert "Add single-process checkpoint save to avoid forked multiproce… (

a7c207f

NVIDIA#3630)

Multimodal: fix VQA dataset selection (NVIDIA#3464)

6503bf8

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Multimodal: Fix multimodal training example - tokenizer, Triton Cache…

2f549e5

… Manager patch, docs (NVIDIA#3507) Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Support TP > GQA for inference (NVIDIA#3627)

afbce84

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

style: black-format test_absorbed_mla for CI lint

8096db6

Made-with: Cursor

copy-pr-bot bot temporarily deployed to test March 19, 2026 23:08 Inactive

copy-pr-bot bot temporarily deployed to test March 21, 2026 20:58 Inactive

copy-pr-bot bot temporarily deployed to test March 21, 2026 21:03 Inactive

copy-pr-bot bot temporarily deployed to test March 21, 2026 21:38 Inactive

copy-pr-bot bot temporarily deployed to test March 21, 2026 23:37 Inactive

copy-pr-bot bot temporarily deployed to test March 22, 2026 00:24 Inactive

copy-pr-bot bot temporarily deployed to test March 22, 2026 01:32 Inactive

ilml force-pushed the merge-main-into-dev branch from bd44a67 to 5c04917 Compare March 22, 2026 02:52

copy-pr-bot bot temporarily deployed to test March 22, 2026 02:53 Inactive

copy-pr-bot bot deployed to test March 22, 2026 03:55 Active

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge main into dev#3865

Merge main into dev#3865
ilml wants to merge 154 commits intoNVIDIA:devfrom
ilml:merge-main-into-dev

ilml commented Mar 13, 2026 •

edited

Loading

Uh oh!

ilml commented Mar 19, 2026

Uh oh!

ilml commented Mar 19, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

ilml commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Summary

Contribution process

Pre-checks

Code review

Step 1: Mark PR as "Ready for Review"

Step 2: Final Review

Step 3: Approved

Merge

Uh oh!

ilml commented Mar 19, 2026

Uh oh!

ilml commented Mar 19, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 21, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

ilml commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

ilml commented Mar 13, 2026 •

edited

Loading