Skip to content

ci: cut prebuild API cost by ~99% (paginated gh api + curl asset fetch)#13

Merged
widgetii merged 1 commit into
mainfrom
ci/prebuild-api-frugal
Jul 5, 2026
Merged

ci: cut prebuild API cost by ~99% (paginated gh api + curl asset fetch)#13
widgetii merged 1 commit into
mainfrom
ci/prebuild-api-frugal

Conversation

@widgetii

@widgetii widgetii commented Jul 5, 2026

Copy link
Copy Markdown
Member

Rethink of PR #12

The retry crutch treats the symptom. Real root cause: the prebuild consumes ~9000 API calls per fresh-cache run and loses the race against every other workflow sharing the org-wide installation-token bucket. Fix the greed, don't paper over the failures.

What was expensive

Step Calls
gh release list --limit 200 ~1 / source
gh release view <tag> × retention ~90 / source
gh release download <tag> --pattern sizes.*.json (per-asset API call inside) ~98 × 90 = ~8800 / source at cold cache
gh release download again for kconfig assets ~200 more
actions/configure-pages@v5 in pages.yml 1
Total ~9000 per source per fresh-cache run

What replaces it

  • One paginated gh api /repos/<repo>/releases?per_page=100 per source. The response already carries body, assets[], and browser_download_url — so per-tag release view isn't needed. Even at 200-release retention this stays at ~2 calls per source.
  • Asset bytes via anonymous HTTPS on browser_download_url, called with curl. On public repos this consumes zero of the installation token bucket — github.com's 302 to the CDN and the CDN fetch are both unauthenticated. curl's own --retry 5 --retry-delay 5 --retry-all-errors handles transient network flakes.
  • actions/configure-pages@v5 dropped from pages.yml. Its only output was a Pages URL we don't consume (Vite base is hardcoded) and its only side-effect was another rate-limited API call. deploy-pages@v4 doesn't require it as long as Pages is enabled — which it is, source configured to build via GitHub Actions.

Net API budget

Cold cache Warm cache
Before ~9000 / source ~200 / source
After ~2 / source ~2 / source

Abstraction shape

New HttpFn type sits alongside GhFn; both are injectable so tests still drive runPrebuild deterministically without touching the network. GhReleaseDetail.assets[] gains an optional downloadUrl field.

Tests

  • Rewrote makeGh mock to serve the paginated /releases endpoint (only endpoint the new prebuild hits).
  • New makeHttp mock materialises asset content when the runtime curls at a browser_download_url that resolves in the fake set.
  • New fetchReleases describe block: single-page early stop, multi-page pagination.
  • 99 active tests pass locally.
  • End-to-end build against the real OpenIPC/firmware/nightly-20260702-db82859 release fetched all 96 sizes shards + 192 kconfig files via curl and rebuilt the site cleanly.

Test plan

  • npm test — 99 pass, 3 live skipped
  • Local FORCE_REFETCH=1 npm run build against live releases — succeeds
  • Dispatch on this branch (test+build green, deploy rejected by branch policy)
  • Post-merge: watch the next 3 nightly runs; the 07:20–07:50 UTC bucket contention should no longer show up.

🤖 Generated with Claude Code

Three consecutive nightly deploys failed with HTTP 403 "API rate limit
exceeded for installation":

  28646280479 (2026-07-03) prebuild → gh release download OpenIPC/builder
  28698970626 (2026-07-04) actions/configure-pages@v5 → Get Pages site
  28733698722 (2026-07-05) actions/configure-pages@v5 → Get Pages site

The consequential root cause isn't which step happens to bomb — it's
that we were consuming ~9000 API calls per run and losing the race
against every other workflow sharing the org-wide installation-token
bucket. Retry papers over that; the real fix is to stop being greedy.

What was expensive

  * `gh release list --limit 200`                        ~1 call/source
  * `gh release view <tag>` × retention                 ~90 calls/source
  * `gh release download <tag> --pattern sizes.*.json`  ~1 + 1-per-asset
    call/tag = ~98 × 90 = ~8800 calls/source at cold cache
  * `gh release download` again for kconfig assets      ~200 more
  * `actions/configure-pages@v5` in the workflow          1 call
  ─────────────────────────────────────────────────────
  ~9000 API calls per source, per fresh-cache run.

What replaces it

  * ONE paginated `gh api /repos/<repo>/releases?per_page=100` per
    source. The API response already carries body + assets +
    `browser_download_url` for every release, so per-tag `release view`
    isn't needed anymore. Even at 200-release retention this stays at
    ~2 calls/source.

  * Asset bytes come from `browser_download_url` via anonymous HTTPS
    with `curl`. On public repos this consumes zero of the installation
    token bucket — github.com serves a 302 to a signed CDN URL, both
    hops are unauthenticated, neither counts against the API limit.
    curl's built-in `--retry 5 --retry-delay 5 --retry-all-errors`
    covers transient network flakes without our own loop.

  * `actions/configure-pages@v5` dropped from pages.yml — its only
    output was a page URL we don't consume (Vite `base` is hardcoded)
    and its only side-effect was another rate-limited API call.
    `deploy-pages@v4` doesn't require it; Pages is already enabled
    for this repo, source configured to build via GitHub Actions.

Net API budget per prebuild run:

  before: ~9000 calls / source at cold cache, ~200 warm
  after:  ~2 calls / source, cache-independent

Sequential vs parallel curl

  Kept sequential — this fix is about rate-limit exposure, not
  wall-clock. Real end-to-end run at LIMIT=1 (fresh cache) fetched
  96 sizes shards + 192 kconfig files + 1 API call in ~5 min. That's
  fine for a nightly cron.

Abstraction shape

  New `HttpFn` type sits alongside `GhFn`; both are injectable, so
  tests still drive runPrebuild deterministically without touching
  the network. `GhReleaseDetail.assets[]` gains an optional
  `downloadUrl` field so the runtime knows where to `curl`.

Tests

  - Rewrote `makeGh` in tests to mock the paginated `/releases`
    endpoint (only mock the prebuild now uses).
  - New `makeHttp` mock materialises asset content when the runtime
    curls at a browser_download_url that resolves in the fake set.
  - New `fetchReleases` describe block: single-page early stop,
    multi-page pagination.
  - 99 active tests pass; end-to-end build against real
    OpenIPC/firmware nightly-20260702-db82859 fetched all 96 sizes
    shards + kconfig assets via curl and rebuilt the site.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@widgetii widgetii merged commit 41403c3 into main Jul 5, 2026
2 of 3 checks passed
@widgetii widgetii deleted the ci/prebuild-api-frugal branch July 5, 2026 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant