Merge branch 'main' into kunlunl/megatron-fsdp-fp8-params_main

Author: cspades

.github/ISSUE_TEMPLATE/bug_report.md

@@ -9,7 +9,8 @@ assignees: ''
 
 **Describe the bug**
 
-A clear and concise description of what the bug is.
+A clear and concise description of what the bug is. Tag the [@megatron-oncall](https://github.com/orgs/NVIDIA/teams/megatron-oncall) 
+to get oncall's attention to this issue.
 
 **Steps/Code to reproduce bug**
 
@@ -25,4 +26,4 @@ A clear and concise description of what you expected to happen.
 
 **Additional context**
 
-Add any other context about the problem here.
+Add any other context about the problem here. 

.github/ISSUE_TEMPLATE/feature_request.md

@@ -10,6 +10,9 @@ assignees: ''
 **Is your feature request related to a problem? Please describe.**
 A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
 
+Tag the [@megatron-oncall](https://github.com/orgs/NVIDIA/teams/megatron-oncall) 
+to get oncall's attention to this issue.
+
 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.
 

.github/ISSUE_TEMPLATE/question.md

@@ -9,4 +9,5 @@ assignees: ''
 ---
 
 **Your question**
-Ask a clear and concise question about Megatron-LM.
+Ask a clear and concise question about Megatron-LM. Tag the [@megatron-oncall](https://github.com/orgs/NVIDIA/teams/megatron-oncall) 
+to get oncall's attention to this issue.

.github/ISSUE_TEMPLATE/regression.md

@@ -8,7 +8,8 @@ assignees: ''
 ---
 
 **Describe the regression**
-A clear and concise description of what the regression is.
+A clear and concise description of what the regression is. Tag the [@megatron-oncall](https://github.com/orgs/NVIDIA/teams/megatron-oncall) 
+to get oncall's attention to this issue.
 
 **To Reproduce**
 Steps to reproduce the behavior. The easier it is to reproduce the faster it will get maintainer attention.

.github/copy-pr-bot.yaml

@@ -1,4 +1,4 @@
 enabled: true
 auto_sync_draft: false
 auto_sync_ready: true
-trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "ChenhanYu", "FDecaYed", "HaochenYuan", "ISEEKYAN", "JRD971000", "Phlip79", "QiZhangNV", "ShriyaRishab", "Victarry", "Wohox", "ZhiyuLi-Nvidia", "ahmadki", "aklife97", "ananthsub", "asolergi-nv", "buptzyb", "chtruong814", "cspades", "cuichenx", "deepakn94", "dimapihtar", "duncanriach", "erhoo82", "ericharper", "fanshiqing", "frsun-nvda", "gautham-kollu", "gdengk", "guyueh1", "hxbai", "jalbericiola", "jaredcasper", "jenchen13", "jiemingz", "jkamalu", "jon-barker", "kanz-nv", "kevalmorabia97", "ko3n1g", "kunlunl", "kvareddy", "layalir", "lhb8125", "lmcafee-nvidia", "maanug-nv", "mathemakitten", "matthieule", "mehraakash", "mkhona-nvidia", "pablo-garay", "parthmannan", "pthombre", "rogerwaleffe", "sanandaraj5597", "santhnm2", "sbak5", "shanmugamr1992", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sudhakarsingh27", "tdene", "theothermike", "thomasdhc", "trintamaki", "tylerpoon", "wdykas", "xiaoyao0115", "xuwchen", "yanring", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yuzhongw-nvidia", "zhongbozhu"]
+trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "ChenhanYu", "FDecaYed", "HaochenYuan", "ISEEKYAN", "JRD971000", "Phlip79", "QiZhangNV", "ShriyaRishab", "Victarry", "Wohox", "ZhiyuLi-Nvidia", "ahmadki", "aklife97", "ananthsub", "asolergi-nv", "buptzyb", "chtruong814", "cspades", "cuichenx", "deepakn94", "dimapihtar", "duncanriach", "erhoo82", "ericharper", "fanshiqing", "frsun-nvda", "gautham-kollu", "gdengk", "guyueh1", "hxbai", "jalbericiola", "jaredcasper", "jenchen13", "jiemingz", "jkamalu", "jon-barker", "jstjohn", "kanz-nv", "kevalmorabia97", "ko3n1g", "kunlunl", "kvareddy", "layalir", "lhb8125", "lmcafee-nvidia", "maanug-nv", "mathemakitten", "matthieule", "mehraakash", "mkhona-nvidia", "pablo-garay", "parthmannan", "pthombre", "rogerwaleffe", "sanandaraj5597", "santhnm2", "sbak5", "shanmugamr1992", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sudhakarsingh27", "tdene", "theothermike", "thomasdhc", "trintamaki", "tylerpoon", "wdykas", "xiaoyao0115", "xuwchen", "yanring", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yuzhongw-nvidia", "zhongbozhu"]

.github/oncall_schedule.json

@@ -1,8 +1,4 @@
 [
-    {
-        "user": "Phlip79",
-        "date": "2025-12-31"
-    },
     {
         "user": "Phlip79",
         "date": "2026-01-07"
@@ -46,5 +42,9 @@
     {
         "user": "ko3n1g",
         "date": "2026-03-18"
+    },
+    {
+        "user": "Phlip79",
+        "date": "2026-03-25"
     }
 ]

.github/workflows/_build_test_publish_wheel.yml

@@ -157,7 +157,7 @@ jobs:
           - PACKAGE: megatron-core
             PLATFORM: amd64
           - PACKAGE: megatron-fsdp
-            IMAGE: quay.io/pypa/manylinux_2_28_x86_64
+            PLATFORM: amd64
     env:
       PACKAGE: ${{ matrix.PACKAGE }}
     steps:
@@ -173,7 +173,19 @@ jobs:
           TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }}
           TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
           TWINE_REPOSITORY: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/r')) && 'pypi' || 'testpypi' }}
+          PLATFORM: ${{ matrix.PLATFORM }}
         run: |
-          ls -al dist/$PACKAGE*
+
+          # Delete sdist for arm64 since we already upload it with amd64.
+          if [ "$PLATFORM" == "arm64" ]; then
+            rm dist/*.tar.gz
+          fi
+
+          ls -al dist/
           pip install twine
-          twine upload -r $TWINE_REPOSITORY -u $TWINE_USERNAME -p $TWINE_PASSWORD dist/$PACKAGE*
+          twine upload \
+            --verbose \
+            -r $TWINE_REPOSITORY \
+            -u $TWINE_USERNAME \
+            -p $TWINE_PASSWORD \
+            dist/*

.github/workflows/_release_library.yml

@@ -60,6 +60,7 @@ jobs:
     with:
       dry-run: true
       ref: ${{ inputs.release-ref }}
+      no-publish: true
     secrets:
       TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }}
       TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
@@ -74,7 +75,7 @@ jobs:
       )
       && !cancelled()
     outputs:
-      version: ${{ needs.bump-version-mcore.outputs.release-version }}
+      release-version: ${{ needs.bump-version-mcore.outputs.release-version }}
     env:
       IS_DRY_RUN: ${{ inputs.dry-run }}
     steps:
@@ -92,6 +93,7 @@ jobs:
           SRC_DIR: ''
           PYPROJECT_NAME: 'megatron.core'
         run: |
+          set +u
           cd ${{ github.run_id }}
 
           PACKAGE_INFO_FILE="$SRC_DIR${PYPROJECT_NAME//.//}/package_info.py"
@@ -101,7 +103,7 @@ jobs:
           PATCH=$(cat $PACKAGE_INFO_FILE | awk '/^PATCH = /' | awk -F"= " '{print $2}')
           PRERELEASE=$(cat $PACKAGE_INFO_FILE | awk '/^PRE_RELEASE = /' | awk -F"= " '{print $2}' | tr -d '"' | tr -d "'")
 
-          echo "release-version=$MAJOR.$MINOR.$NEXT_PATCH$NEXT_PRERELEASE" | tee -a "$GITHUB_OUTPUT"
+          echo "release-version=$MAJOR.$MINOR.$PATCH$PRERELEASE" | tee -a "$GITHUB_OUTPUT"
 
           if [[ "$PRERELEASE" != "" ]]; then
             if [[ "$PRERELEASE" == *rc* ]]; then
@@ -130,6 +132,8 @@ jobs:
           SRC_DIR: 'megatron/core/distributed/fsdp/src/'
           PYPROJECT_NAME: 'megatron_fsdp'
         run: |
+          set +u
+          
           cd ${{ github.run_id }}
 
           PACKAGE_INFO_FILE="$SRC_DIR${PYPROJECT_NAME//.//}/package_info.py"

docs/developer/contribute.md

@@ -46,11 +46,11 @@ You should receive a response within 2 business days.
 
 ### I need help, who should I ping?
 
-Use `@megatron-oncall`.
+Use [@megatron-oncall](https://github.com/orgs/NVIDIA/teams/megatron-oncall).
 
 ### If my issue or PR isn't getting attention, what should I do?
 
-After 2 business days, use `@megatron-oncall`.
+After 2 business days, tag the user [@megatron-oncall](https://github.com/orgs/NVIDIA/teams/megatron-oncall).
 
 ### Is there a policy for issues and PRs that haven't been touched in X days? Should they be closed?
 

docs/developer/oncall.md

@@ -1,33 +1,45 @@
 # Oncall Overview
 
-During your oncall week, you will be assigned to all PRs marked “Ready for Review”. From a high-level, your responsibilities include:
+During your oncall week, you will be assigned to all PRs marked “Ready for 
+Review”. From a high-level, your responsibilities include:
 
-  - Review all new PRs
-  - Uphold and enforce the Megatron coding standard
-  - Accelerate the review process for expert reviewers (when necessary)
+- Review all new PRs
+- Accelerate the review process
+- Ensure issues and discussion questions are answered
 
-## Checklist
+## PR Responsibilities
 
 Below is the checklist that the oncall needs to go through for each PR.
 
-- [ ] Should the PR remain a single PR?
+- Should the PR remain a single PR?
   - Each PR should have at most 1 expert reviewer, although there will be some outlier cases
-- [ ] Label PR as “complexity: low”, “complexity: medium”, or “complexity: high” depending on complexity
-  - Low: <100 lines changed
-  - Medium: 100 < lines changed < 500
-  - High: > 500 lines changed
-- [ ] Does this PR have proper testing coverage?
+- Label PR as “complexity: low”, “complexity: medium”, or “complexity: high” depending on complexity
+  - Expert reviewers have final say, oncall just sets the initial complexity level
+  - Initial complexity level guideline
+    - Low: <100 lines changed
+    - Medium: 100 < lines changed < 500
+    - High: > 500 lines changed
+- Does this PR have proper testing coverage?
   - If new logic is added, is the new logic tested?
-- [ ] Should the PR add documentation for any new features?
-- [ ] Does the PR conform to our style guidelines?
+- Should the PR add documentation for any new features?
+- Does the PR conform to our style guidelines?
   - Code structure
   - Cleanliness
   - Comments
   - File structure
-- [ ] Do all tests pass?
+- Do all tests pass?
   - Oncall will need to kick off testing suite for external reviewers
   - Comment “/ok to test commid_id” to kick off testing suite
-- [ ] Add the “Expert Review” label
-  - Expert reviewers should review within 1 business day. Message the assigned reviewer if it is taking longer.
-  - After 2 business days, the expert reviewer waives the right to review.
-- [ ] Add the “Final Review” label after experts approve
+- Add the “Expert Review” label
+  - Select an expert reviewer from each expert group as a reviewer. If you’re unsure who to select, pick a “maintainer” or manager.
+  - **Expert reviewers should review within 1 business day.** Message the assigned reviewer if it is taking longer. The reviewer either needs to review the PR or suggest an alternate reviewer.
+  - If the reviewer is not responding after 2 business days, escalate to the reviewer's manager.
+- Add the “Final Review” label after experts approve
+  - Final reviewers should review within 1 business day. Message the assigned reviewer if it is taking longer.
+  - If the reviewer is not responding after 2 business days, escalate to the reviewer's manager.
+
+## Issues and Discussion Questions
+
+On a daily basis, check for new [issues](https://github.com/NVIDIA/Megatron-LM/issues) 
+and [discussions](https://github.com/NVIDIA/Megatron-LM/discussions). If you 
+do not know how to answer that's ok! Delegate the issue or discussion to someone who does.