Skip to content

[rubysrc2cpg] Retry ruby_ast_gen download on HTTP errors - #6265

Merged
ml86 merged 1 commit into
masterfrom
max/rubysrc2cpg-astgen-dl-retries
Sep 8, 2026
Merged

ml86 merged 1 commit into
masterfrom
max/rubysrc2cpg-astgen-dl-retries

Conversation

@max-leuthaeuser

Copy link
Copy Markdown
Contributor

Motivation

We've seen transient HTTP errors fail the build during rubysrc2cpg's astGenResourceTask, which downloaded the ruby_ast_gen zip via a single-shot urlInputStream + IO.transfer with no retry logic.

Change

Route the download through DownloadHelper.ensureIsAvailable (backed by UrlRetry.downloadWithRetries: 5 attempts, exponential backoff, non-2xx status check) — the same helper already used by joernTypeStubsDlTask and all other frontends' astgen downloads.

Behavior is otherwise unchanged: the download still only happens when the unpacked astgen version doesn't match, and the zip is still deleted after unzipping. Since the URL bookkeeping in .local/source-urls is only written after a successful download, a failed attempt always triggers a fresh retry on the next build.

Verification

  • Forced the download path by removing the unpacked version.rb, then ran sbt rubysrc2cpg/astGenResourceTask: download went through ensureIsAvailable ([INFO] downloading ...), unzip + zip cleanup succeeded.
  • Re-ran the task: correctly short-circuits with no download once the version matches.
  • Audited all other .sbt files for raw download primitives (urlInputStream, IO.transfer, toURL, openStream, curl/wget): no remaining occurrences — every build-time HTTP download in the repo now goes through the retrying helper.

Use DownloadHelper.ensureIsAvailable (backed by UrlRetry with
exponential backoff) in astGenResourceTask instead of a single-shot
urlInputStream transfer, matching the other frontends' astgen
downloads.
@max-leuthaeuser
max-leuthaeuser requested a review from ml86 September 8, 2026 06:01
@ml86
ml86 merged commit a76dfb2 into master Sep 8, 2026
8 checks passed
@max-leuthaeuser
max-leuthaeuser deleted the max/rubysrc2cpg-astgen-dl-retries branch September 9, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants