openspec: packer S3 build cache proposal

[rustyrazorblade]

Add OpenSpec change proposal for caching compiled packer artifacts (BCC, Cassandra source builds) in S3 to reduce AMI build time by 15–40 minutes on cache hits.

Closes #569

[rustyrazorblade]

@claude openspec apply

[claude]

Claude finished @rustyrazorblade's task in 3m 2s —— View job

---

Applying OpenSpec Change: Packer S3 Build Cache

• Read OpenSpec proposal, design, and tasks
• Task 1: Create packer/lib/s3_cache.sh
• Task 2: Add BCC cache to install_bcc.sh
• Task 3: Add Cassandra source build cache to install_cassandra.sh
• Task 4: Update packer/README.md with documentation
• Push changes

Summary

Implemented all 4 tasks from the OpenSpec change:

packer/lib/s3_cache.sh (new file) — shared helpers:

• s3_cache_get <bucket> <key> <dest> — downloads from S3, returns 0 on hit, 1 on miss/failure
• s3_cache_put <bucket> <key> <src> — uploads to S3, best-effort (never fails the build)
• Both no-op when PACKER_CACHE_BUCKET is unset or PACKER_CACHE_SKIP=1

install_bcc.sh — S3 cache integrated:

• Cache key: packer-build-cache/bcc/bcc-v{VERSION}-{ARCH}.tar.gz
• On cache hit: extracts tarball to /, verifies Python import, skips compilation (~15–20 min saved)
• On cache miss: compiles as normal, then uploads installed artifacts to S3
• Python import check runs on both paths

install_cassandra.sh — S3 cache integrated for git-branch builds only:

• After git clone, captures GIT_SHA=$(git rev-parse --short=12 HEAD)
• Cache key: packer-build-cache/cassandra/{VERSION}-{GIT_SHA}.tar.gz
• On cache hit: extracts to /usr/local/cassandra/, skips ant build (~20 min saved)
• On cache miss: runs ant build, then uploads built directory to S3
• Configuration block (conf backup, cassandra.in.sh append) still runs on both paths

packer/README.md — new "S3 Build Cache" section documenting:

• PACKER_CACHE_BUCKET and PACKER_CACHE_SKIP env vars
• S3 key structure table
• Required IAM permissions (s3:GetObject, s3:PutObject, s3:HeadObject)
• How to manually invalidate cache entries

[github-actions]

Code Coverage

Total Project Coverage	67.03%	❌

[claude]

PR Review: openspec packer S3 build cache proposal

Good proposal overall — the problem is real and the approach is sound. A few issues to address before implementation:

---

Critical: Script ordering issue in base.pkr.hcl

install_bcc.sh runs at line 119, but install_awscli.sh runs at line 123-125 — after BCC. The s3_cache_get/s3_cache_put functions call aws s3 cp, but the AWS CLI won't exist yet. Either reorder the provisioners (install AWS CLI first), or the helper functions need to handle the case where aws is not in PATH. The design/tasks should call this out explicitly.

---

Critical: packer/lib/s3_cache.sh won't be on the remote build machine

Packer's script = "install/install_bcc.sh" provisioner uploads and runs a single file. A source /path/to/s3_cache.sh inside it will fail because packer/lib/ is never uploaded. Task 4 needs to add a provisioner "file" step to base.pkr.hcl and cassandra.pkr.hcl that uploads the helper to a known path before any script that sources it, or inline the helper into each script (worse option). This is a blocker for implementation.

---

Arch value consistency

The BCC cache key uses {ARCH}. The design should specify the exact command — uname -m returns x86_64/aarch64 while dpkg --print-architecture returns amd64/arm64. Either works, but it should match consistently across all builds (including CI). The packer var.arch variable is amd64/arm64, so the design should clarify which format is used to avoid CI vs. local mismatches.

---

BCC artifact list may miss headers

The cached artifacts list includes /usr/lib/libbcc*, /usr/lib/libbpf*, /usr/share/bcc/, and Python bindings, but omits /usr/include/bcc/ (C headers). This is fine if BCC is only used at runtime on the AMI, but if any tools compile against it, the cache restore will be incomplete. Worth a note in the design to confirm headers are intentionally excluded.

---

Cache invalidation for base image changes

The BCC cache key (bcc-v{VERSION}-{ARCH}) doesn't account for OS changes. If the base AMI moves from Ubuntu 22.04 → 24.04, the cached binary would be wrong but the key would match. Consider whether to include an OS component in the key (e.g., $(lsb_release -cs) → noble) or document this as a known limitation requiring manual PACKER_CACHE_SKIP=1 on base image changes.

---

Testing strategy not addressed

The existing testPackerBase/testPackerCassandra Gradle tasks run scripts in Docker. There's no mention of how S3 cache logic will be tested locally. Consider adding a LocalStack or PACKER_CACHE_BUCKET being unset (which correctly no-ops) scenario to the task list. At minimum, the helper should be tested with bucket unset (no-op path) and the full S3 path (LocalStack or real bucket in CI).

---

Minor: ~/.m2 cleanup on cache hit

The existing install_cassandra.sh removes ~/.m2 after the ant build (around line 258). On a cache hit, no ant/Maven runs, so no .m2 pollution — this is fine and no change needed. Just confirming this was considered.

---

The proposal is well-structured and the task breakdown is clear. Addressing the script ordering and file provisioner issues before starting Task 1 will save a debugging cycle.

[claude]

test comment

[claude]

PR Review: Packer S3 Build Cache - Full Review

Good overall design. The best-effort fallback philosophy is sound and the shared helper pattern in packer/lib/s3_cache.sh is clean. A few issues worth addressing before merging.

Bug 1: Corrupted cache archives will fail the build, not fall back

In install_bcc.sh, the cache-restore block runs under set -euo pipefail. If a partially-uploaded archive passes the aws s3 cp download but fails tar extraction, set -euo pipefail causes the script to exit immediately instead of falling back to a normal build. The design doc says the tar extraction will fail and the build will fall back to compiling from scratch - but that is not what the code does. The same issue exists in install_cassandra.sh on the cache-restore path. Fix: wrap the tar extraction in its own error handler and fall through to the build on failure, or temporarily disable errexit for the restore block.

Bug 2: Cache archive owned by root may not be readable during upload

In install_bcc.sh, sudo tar -czf creates the CACHE_ARCHIVE file owned by root with restrictive permissions. The aws s3 cp in s3_cache_put runs as the normal user and will likely get a permission-denied error - which then gets swallowed by the OR-true guard. The cache write silently fails every time. Fix: either sudo chmod a+r the archive before calling s3_cache_put, or run the entire aws s3 cp under sudo. The same pattern in install_cassandra.sh has the same problem.

Minor: Cloned repo not cleaned up on cache hit (install_cassandra.sh)

On a cache hit, the version clone directory is left behind in the working directory. For Cassandra repos this can be several hundred MB. Should add rm -rf of the clone dir after a successful cache restore.

Minor: Misleading log message in s3_cache_put

The message Cache write complete (errors are non-fatal) is printed even when the upload failed. Consider using a conditional to log the actual outcome.

Observation: tasks.md shows unchecked tasks but implementation is already in the PR

The tasks.md file has all items unchecked but the implementation scripts are already included in this PR. If this is intentional (proposal + implementation together), the tasks should be marked complete. If the tasks are meant to track future work, the implementation files should not be here yet.

Minor: CACHE_ARCHIVE temp file not covered by cleanup trap (install_bcc.sh)

If the script exits during cache restoration, the CACHE_ARCHIVE temp file is not deleted - the existing cleanup trap only handles WORK_DIR. Low impact since packer instances are ephemeral.

Design looks solid otherwise: S3 key design (version + arch for BCC, version + git SHA for Cassandra) is content-addressed correctly. PACKER_CACHE_SKIP bypass is a good escape hatch. Documentation in packer/README.md is thorough and clear. IAM policy in both the docs and the design doc is correct. The shared helper avoids duplication correctly.

The two bugs (corrupted-cache fallback under errexit, and sudo-owned archive not readable by aws-cli user) are the most impactful and should be addressed before merging.

[rustyrazorblade]

@claude address the feedback

[claude]

Claude finished @rustyrazorblade's task in 4m 30s —— View job

---

Addressed all review feedback. Here's what was changed:

Tasks

• Read relevant files
• Fix: Script ordering — moved install_awscli.sh before install_bcc.sh in base.pkr.hcl
• Fix: Upload s3_cache.sh via provisioner "file" to /tmp/s3_cache.sh in both base.pkr.hcl and cassandra.pkr.hcl; scripts now source from that fixed path (the old SCRIPT_DIR relative path broke in packer runs since scripts are uploaded to a random /tmp location)
• Fix: Corrupted cache fallback — wrapped tar extraction in a nested if so a bad archive falls back to a normal build instead of aborting under set -euo pipefail
• Fix: sudo-owned archive — added sudo chmod a+r on cache archives before passing to s3_cache_put so the non-root aws user can read them
• Fix: Clone cleanup on cache hit — added rm -rf "$version" after successful cache restore in install_cassandra.sh
• Fix: Misleading log message — s3_cache_put now logs actual success or failure rather than always printing "complete"
• Fix: Marked all tasks.md items complete
• Address: Arch consistency — uname -m was already documented in design.md; added OS-change cache invalidation risk and file provisioner deployment notes to the design
  · Branch

[claude]

test

[claude]

PR Review: Packer S3 Build Cache

Overall this is a solid, well-structured proposal. The design doc is thorough, the shared helper pattern is the right call, and the best-effort semantics are implemented correctly throughout. A few issues to address before merging.

Bug: Cache env vars not forwarded to remote packer shell provisioners

Packer shell provisioners do not inherit the caller's environment variables automatically -- only variables in 'environment_vars' reach the remote script. Neither 'PACKER_CACHE_BUCKET' nor 'PACKER_CACHE_SKIP' appear in 'environment_vars' in either HCL file.

The 'install_bcc.sh' provisioner has no 'environment_vars' at all. The 'install_cassandra.sh' provisioner only forwards 'INSTALL_CASSANDRA=1'.

The effect: both scripts will always see 'PACKER_CACHE_BUCKET' as unset, the helper functions will no-op, and the cache will silently never work even when the user sets the env var locally.

Fix for 'base.pkr.hcl' (same pattern needed in 'cassandra.pkr.hcl'): add both vars to environment_vars using packer's env() interpolation. env() returns empty string when the var is unset, which is the correct safe default (cache skipped).

Issue: Hard failure sourcing /tmp/s3_cache.sh outside packer

Both scripts unconditionally source '/tmp/s3_cache.sh'. If someone runs the script directly -- local Docker-based packer test, the testPackerScript Gradle task, manual debugging -- the file won't exist and the script aborts immediately before doing any real work.

TESTING.md and the testPackerScript Gradle task specifically support running individual provisioner scripts in isolation. This change silently breaks that workflow.

A runtime guard handles both cases: check if the file exists first with [[ -f /tmp/s3_cache.sh ]], and if not, define no-op stubs for s3_cache_get (return 1) and s3_cache_put (return 0).

Minor: BCC cache key does not include OS version

The design doc acknowledges this risk: if the base AMI moves to a new Ubuntu LTS, the existing cache entry will match by key but the binary may be incompatible. The documented mitigation is manual (PACKER_CACHE_SKIP=1).

Adding lsb_release -rs to the key (e.g. bcc-v0.35.0-x86_64-24.04.tar.gz) would make invalidation automatic on OS upgrades and prevent a class of hard-to-debug runtime failures. Not a blocker, but worth considering.

Nit: Integer flag compared as string

In install_cassandra.sh, CACHE_HIT is compared as a string with == 0. Since it is an integer flag, the arithmetic form (( ! CACHE_HIT )) is more idiomatic.

Things done well

• s3_cache.sh is clean and well-commented. Centralizing the no-op logic (bucket unset, skip flag) means neither calling script has to repeat those checks.
• mktemp for download destinations avoids path collisions.
• compgen -G to test for optional libbpf files before expanding the glob is the correct pattern under set -euo pipefail.
• sudo chmod a+r on the temp archive before non-root aws s3 cp is a subtle but necessary detail.
• Cassandra cache tarballs archive with relative paths and extract to the correct target directory -- directory structure round-trips correctly.
• || true on tar create and return 0 in s3_cache_put consistently enforce the best-effort contract throughout.

The env var forwarding issue would prevent the cache from ever working. The sourcing issue would break local testing. Both should be fixed before merging.