Skip to content

Fix source build cache invalidation in CI#4202

Open
mitchdz wants to merge 2 commits intomainfrom
fix-source-build-caching
Open

Fix source build cache invalidation in CI#4202
mitchdz wants to merge 2 commits intomainfrom
fix-source-build-caching

Conversation

@mitchdz
Copy link
Copy Markdown
Collaborator

@mitchdz mitchdz commented Mar 23, 2026

Fix source build cache invalidation in CI

Problem

The "Load source build cache" job (source_build / prereqs stage) was rebuilding almost every run because the prereqs layers were invalidated on nearly every commit.

The cause was ADD .git/modules/tpls/llvm/HEAD and ADD .git/modules/tpls/pybind11/HEAD in assets.Dockerfile. These files depend on the current checkout, so they change whenever:

  • A different PR runs (different merge commit or submodule refs)
  • Submodule pins change on main or the PR base

When that ADD layer changes, BuildKit invalidates all later layers (git init, LLVM build, GCC ldd check), so the expensive prereqs stage was rarely cached.

Solution

Use build args llvm_commit and pybind11_commit instead of adding .git/modules/ HEAD files from the build context. The prereqs layer cache is now driven by submodule commit hashes instead of the current checkout.

  • Same llvm_commit/pybind11_commit → cache hit (e.g. PRs based on main that don’t change submodules)
  • Different submodule refs → full rebuild, as expected

Changes

  • docker/build/assets.Dockerfile: Add ARG llvm_commit and ARG pybind11_commit; replace ADD of submodule HEAD files with a RUN that creates them from these args.
  • ci.yml: Pass llvm_commit and pybind11_commit into the source_build job.
  • create_cache_command.yml: Pass submodule commits into the source_build_caches job.
  • deployments.yml: Pass submodule commits into the source_build job.
  • prebuilt_binaries.yml: Compute and pass llvm_commit and pybind11_commit for builds where the cache may be cold.

Signed-off-by: mdzurick <mdzurick@nvidia.com>
github-actions bot pushed a commit that referenced this pull request Mar 23, 2026
@github-actions
Copy link
Copy Markdown

CUDA Quantum Docs Bot: A preview of the documentation can be found here.

Signed-off-by: mdzurick <mdzurick@nvidia.com>
@github-actions
Copy link
Copy Markdown

CUDA Quantum Docs Bot: A preview of the documentation can be found here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant