Fix Github workflows issues #2636

pggPL · 2026-01-30T19:05:26Z

Description

This PR fixes following issues:

Deploy nightly docs fails, because of non-compatible packages. I tested it in my own fork and version changes fix the issue,
Build jobs are red, because of OoM - the MAX_JOBS=1 envvar was not propagated correctly inside the containers,
PyTorch build job needed more disk space, so I changed container to JAX one and installed pytorch manually - it takes much less space than any other option,

Fixes # (issue)

Type of change

Documentation change (change only to the documentation, either a fix or a new content)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Infra/Build change
Code refactoring

Checklist:

I have read and followed the contributing guidelines
The functionality is complete
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps · 2026-01-30T19:07:48Z

Greptile Overview

Greptile Summary

This PR resolves three critical CI/CD issues:

Documentation deployment: Updated deprecated GitHub Actions versions (actions/upload-pages-artifact v1→v3, actions/deploy-pages v2→v4) and removed the deprecated name parameter, fixing compatibility issues with current GitHub Pages infrastructure
OOM build failures: Fixed MAX_JOBS=1 environment variable propagation by replacing Docker-in-Docker architecture with direct container execution, ensuring memory limits are respected during builds
Disk space optimization: Eliminated the maximize-build-space action and switched PyTorch builds to a unified CUDA 13.0.0 base container with manual dependency installation, significantly reducing disk usage

All four build jobs (core, pytorch, jax, all) now use consistent CUDA 13.0.0 containers with properly configured dependency installations. The NVTE_CUDA_ARCHS: "90" environment variable and git config safe.directory have been added to all jobs for proper operation in containerized environments.

Confidence Score: 4/5

This PR is safe to merge with proper testing of the build workflows
The changes are well-structured infrastructure improvements that address real CI/CD issues. The CUDA version upgrade (12.x → 13.0) across all jobs is consistent, and the Docker architecture simplification properly fixes the MAX_JOBS propagation issue. However, given this changes the fundamental build infrastructure and CUDA version, thorough testing of all build jobs is essential before merge.
Build workflow should be tested across all four jobs (core, pytorch, jax, all) to verify CUDA 13.0 compatibility

Important Files Changed

Filename	Overview
.github/workflows/deploy_nightly_docs.yml	Updated GitHub Actions versions (v1→v3, v2→v4) and removed deprecated `name` parameter; added manual trigger
.github/workflows/build.yml	Unified all jobs to CUDA 13.0, replaced Docker-in-Docker with direct containers, fixed MAX_JOBS propagation; minor CUDA version consistency concern

Sequence Diagram

sequenceDiagram
    participant GHA as GitHub Actions
    participant OLD as Old Build (Docker-in-Docker)
    participant NEW as New Build (Direct Container)
    participant Build as Build Process
    
    Note over GHA,Build: Old Architecture (Docker-in-Docker)
    GHA->>OLD: Start ubuntu-latest runner
    OLD->>OLD: maximize-build-space action
    OLD->>OLD: docker run builder container
    OLD->>OLD: docker exec (MAX_JOBS not propagated)
    OLD->>Build: pip install (OOM due to no MAX_JOBS limit)
    
    Note over GHA,Build: New Architecture (Direct Container)
    GHA->>NEW: Start with CUDA 13.0 container
    NEW->>NEW: apt-get install dependencies
    NEW->>NEW: git config safe.directory
    NEW->>NEW: MAX_JOBS=1 in env (properly propagated)
    NEW->>Build: pip install (memory controlled)
    Build-->>NEW: Success (no OOM)
    
    Note over GHA,Build: Documentation Deployment
    GHA->>GHA: upload-pages-artifact@v3 (was v1)
    GHA->>GHA: deploy-pages@v4 (was v2)
    GHA->>GHA: Fixes compatibility issues

greptile-apps

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

.github/workflows/deploy_nightly_docs.yml

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

.github/workflows/build.yml

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T04:23:36Z

.github/workflows/build.yml

+          root-reserve-mb: 4096
          temp-reserve-mb: 32
-          swap-size-mb: 10240
+          swap-size-mb: 4096


Verify that reduced memory allocation (root: 5120→4096 MB, swap: 10240→4096 MB) is sufficient for PyTorch builds to avoid OOM issues.

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T04:28:36Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install cmake==3.21.0 pybind11[global] ninja pydantic importlib-metadata>=1.0 packaging numpy einops onnxscript
+          pip install torch --no-cache-dir


Missing --index-url https://download.pytorch.org/whl/cu130 which was present before. This may install CPU-only PyTorch instead of the CUDA version needed for testing.

Suggested change

pip install torch --no-cache-dir

pip install torch --no-cache-dir --index-url https://download.pytorch.org/whl/cu130

greptile-apps · 2026-02-03T04:28:36Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install cmake==3.21.0 pybind11[global] einops onnxscript
+          pip install torch --no-cache-dir


Missing --index-url https://download.pytorch.org/whl/cu130 which was present before. This may install CPU-only PyTorch instead of the CUDA version.

Suggested change

pip install torch --no-cache-dir

pip install torch --no-cache-dir --index-url https://download.pytorch.org/whl/cu130

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T04:34:55Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] ninja pydantic importlib-metadata>=1.0 packaging numpy einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 when installing PyTorch. Without this, pip will install the default PyTorch from PyPI, which may be CPU-only or have incompatible CUDA version. This was present in the base commit for the all job and is needed here too.

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

greptile-apps · 2026-02-03T04:34:56Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 when installing PyTorch. The base commit explicitly used this index URL to ensure CUDA 13.0 support matching the JAX container (see commit 4cf2f12). Without it, the default PyPI version will be installed, which may be CPU-only.

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T04:43:10Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] ninja pydantic importlib-metadata>=1.0 packaging numpy einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 for PyTorch installation. The JAX container has CUDA 13.0, and commit 4cf2f12 explicitly added this index URL to ensure compatibility. Without it, pip may install a CPU-only or incompatible CUDA version from PyPI.

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

greptile-apps · 2026-02-03T04:43:11Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 for PyTorch installation. This was explicitly added in commit 4cf2f12 to match the CUDA 13.0 version in the JAX container. Without it, the default PyPI version will be installed, which may be CPU-only or have incompatible CUDA version.

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T04:46:22Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] ninja pydantic importlib-metadata>=1.0 packaging numpy einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 for PyTorch installation. This was explicitly added in commit 4cf2f12 to match CUDA 13.0 in the JAX container. Without it, pip installs the default PyPI version (likely CPU-only or wrong CUDA version).

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

greptile-apps · 2026-02-03T04:46:23Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 for PyTorch installation. This was present in the base commit and is required to match CUDA 13.0 in the JAX container (see commit 4cf2f12).

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T04:50:52Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] ninja pydantic importlib-metadata>=1.0 packaging numpy einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 for PyTorch. The JAX container has CUDA 13.0, but without this flag pip installs the default PyPI version (likely CPU-only or wrong CUDA version). This was explicitly added in commit 4cf2f12 (#2308) for this exact reason.

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

greptile-apps · 2026-02-03T04:50:53Z

.github/workflows/build.yml

+      - name: 'Dependencies'
+        run: |
+          pip install --no-cache-dir cmake==3.21.0 pybind11[global] einops onnxscript
+          pip install --no-cache-dir torch


Missing --index-url https://download.pytorch.org/whl/cu130 for PyTorch. Without this, pip installs the default PyPI version which may not match the JAX container's CUDA 13.0. This was present in the base commit for this exact job.

Suggested change

pip install --no-cache-dir torch

pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cu130

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T04:54:15Z

.github/workflows/build.yml

        run: NVTE_USE_CCACHE=1 NVTE_CCACHE_BIN=sccache pip install --no-build-isolation . -v
        env:
          NVTE_FRAMEWORK: none
-          MAX_JOBS: 1
          SCCACHE_GHA_ENABLED: "true"
+          SCCACHE_CACHE_SIZE: "5G"


MAX_JOBS=1 removed from Core job. Check that the build completes successfully without this limit to prevent OOM issues.

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T07:42:02Z

.github/workflows/build.yml

+      image: nvcr.io/nvidia/cuda:12.1.0-devel-ubuntu22.04
+      options: --user root


Check that CUDA 12.1.0 (downgraded from 12.8.0) doesn't break PyTorch compatibility or cause runtime issues

greptile-apps · 2026-02-03T07:42:03Z

.github/workflows/build.yml

    container:
-      image: ghcr.io/nvidia/jax:jax
+      image: nvcr.io/nvidia/cuda:12.1.0-devel-ubuntu22.04
      options: --user root


Switched from ghcr.io/nvidia/jax:jax to base CUDA container - verify JAX[cuda12] install is compatible with CUDA 12.1 and includes all necessary dependencies

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-03T10:12:29Z

.github/workflows/build.yml

+          apt-get update
+          apt-get install -y git python3.9 pip cudnn9-cuda-12
+          pip install cmake==3.21.0 pybind11[global] ninja packaging
+          pip install jax


pip install jax installs CPU-only JAX by default. Need jax[cuda12] to match CUDA 13.0 runtime

Suggested change

pip install jax

pip install jax[cuda12]

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

fix

ee0cca1

Signed-off-by: Pawel Gadzinski <[email protected]>

pggPL changed the title ~~PR to debug github workflows fails~~ PR to debug github workflows failures Jan 30, 2026

greptile-apps bot reviewed Jan 30, 2026

View reviewed changes

fix

df6a81b

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps bot reviewed Jan 30, 2026

View reviewed changes

fix

daa3bf3

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps bot reviewed Jan 30, 2026

View reviewed changes

.github/workflows/deploy_nightly_docs.yml Show resolved Hide resolved

pggPL changed the title ~~PR to debug github workflows failures~~ Fix Github workflows issues Jan 30, 2026

fix

d47399a

Signed-off-by: Pawel Gadzinski <[email protected]>

greptile-apps bot reviewed Jan 31, 2026

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

ccache size limit

d2091f2

Signed-off-by: Pawel Gadzinski <[email protected]>