ci: cut prebuild API cost by ~99% (paginated gh api + curl asset fetch)#13
Merged
Conversation
Three consecutive nightly deploys failed with HTTP 403 "API rate limit
exceeded for installation":
28646280479 (2026-07-03) prebuild → gh release download OpenIPC/builder
28698970626 (2026-07-04) actions/configure-pages@v5 → Get Pages site
28733698722 (2026-07-05) actions/configure-pages@v5 → Get Pages site
The consequential root cause isn't which step happens to bomb — it's
that we were consuming ~9000 API calls per run and losing the race
against every other workflow sharing the org-wide installation-token
bucket. Retry papers over that; the real fix is to stop being greedy.
What was expensive
* `gh release list --limit 200` ~1 call/source
* `gh release view <tag>` × retention ~90 calls/source
* `gh release download <tag> --pattern sizes.*.json` ~1 + 1-per-asset
call/tag = ~98 × 90 = ~8800 calls/source at cold cache
* `gh release download` again for kconfig assets ~200 more
* `actions/configure-pages@v5` in the workflow 1 call
─────────────────────────────────────────────────────
~9000 API calls per source, per fresh-cache run.
What replaces it
* ONE paginated `gh api /repos/<repo>/releases?per_page=100` per
source. The API response already carries body + assets +
`browser_download_url` for every release, so per-tag `release view`
isn't needed anymore. Even at 200-release retention this stays at
~2 calls/source.
* Asset bytes come from `browser_download_url` via anonymous HTTPS
with `curl`. On public repos this consumes zero of the installation
token bucket — github.com serves a 302 to a signed CDN URL, both
hops are unauthenticated, neither counts against the API limit.
curl's built-in `--retry 5 --retry-delay 5 --retry-all-errors`
covers transient network flakes without our own loop.
* `actions/configure-pages@v5` dropped from pages.yml — its only
output was a page URL we don't consume (Vite `base` is hardcoded)
and its only side-effect was another rate-limited API call.
`deploy-pages@v4` doesn't require it; Pages is already enabled
for this repo, source configured to build via GitHub Actions.
Net API budget per prebuild run:
before: ~9000 calls / source at cold cache, ~200 warm
after: ~2 calls / source, cache-independent
Sequential vs parallel curl
Kept sequential — this fix is about rate-limit exposure, not
wall-clock. Real end-to-end run at LIMIT=1 (fresh cache) fetched
96 sizes shards + 192 kconfig files + 1 API call in ~5 min. That's
fine for a nightly cron.
Abstraction shape
New `HttpFn` type sits alongside `GhFn`; both are injectable, so
tests still drive runPrebuild deterministically without touching
the network. `GhReleaseDetail.assets[]` gains an optional
`downloadUrl` field so the runtime knows where to `curl`.
Tests
- Rewrote `makeGh` in tests to mock the paginated `/releases`
endpoint (only mock the prebuild now uses).
- New `makeHttp` mock materialises asset content when the runtime
curls at a browser_download_url that resolves in the fake set.
- New `fetchReleases` describe block: single-page early stop,
multi-page pagination.
- 99 active tests pass; end-to-end build against real
OpenIPC/firmware nightly-20260702-db82859 fetched all 96 sizes
shards + kconfig assets via curl and rebuilt the site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rethink of PR #12
The retry crutch treats the symptom. Real root cause: the prebuild consumes ~9000 API calls per fresh-cache run and loses the race against every other workflow sharing the org-wide installation-token bucket. Fix the greed, don't paper over the failures.
What was expensive
gh release list --limit 200gh release view <tag>× retentiongh release download <tag> --pattern sizes.*.json(per-asset API call inside)gh release downloadagain for kconfig assetsactions/configure-pages@v5in pages.ymlWhat replaces it
gh api /repos/<repo>/releases?per_page=100per source. The response already carriesbody,assets[], andbrowser_download_url— so per-tagrelease viewisn't needed. Even at 200-release retention this stays at ~2 calls per source.browser_download_url, called withcurl. On public repos this consumes zero of the installation token bucket — github.com's 302 to the CDN and the CDN fetch are both unauthenticated. curl's own--retry 5 --retry-delay 5 --retry-all-errorshandles transient network flakes.actions/configure-pages@v5dropped from pages.yml. Its only output was a Pages URL we don't consume (Vitebaseis hardcoded) and its only side-effect was another rate-limited API call.deploy-pages@v4doesn't require it as long as Pages is enabled — which it is, source configured to build via GitHub Actions.Net API budget
Abstraction shape
New
HttpFntype sits alongsideGhFn; both are injectable so tests still driverunPrebuilddeterministically without touching the network.GhReleaseDetail.assets[]gains an optionaldownloadUrlfield.Tests
makeGhmock to serve the paginated/releasesendpoint (only endpoint the new prebuild hits).makeHttpmock materialises asset content when the runtimecurls at abrowser_download_urlthat resolves in the fake set.fetchReleasesdescribe block: single-page early stop, multi-page pagination.OpenIPC/firmware/nightly-20260702-db82859release fetched all 96 sizes shards + 192 kconfig files via curl and rebuilt the site cleanly.Test plan
npm test— 99 pass, 3 live skippedFORCE_REFETCH=1 npm run buildagainst live releases — succeeds🤖 Generated with Claude Code