feat!: migrate from Google Cloud Storage to go.dev downloads API#117
Merged
andrewkroh merged 8 commits intomainfrom Oct 17, 2025
Merged
feat!: migrate from Google Cloud Storage to go.dev downloads API#117andrewkroh merged 8 commits intomainfrom
andrewkroh merged 8 commits intomainfrom
Conversation
BREAKING CHANGE: The binary download mechanism has been completely rearchitected to use the official Go downloads API instead of parsing Google Cloud Storage XML responses. Changes: - Add new api.go module with JSON structures matching the go.dev API response - GoRelease struct represents a Go release with version info and files - GoFile struct represents downloadable files with OS/arch/kind metadata - fetchGoReleases() fetches data from https://go.dev/dl/?mode=json&include=all - findArchiveFile() finds the correct archive for a given OS/arch combination - constructDownloadURL() builds URLs in the format https://go.dev/dl/{filename} - Update binrepo.go to use the new API - Rewrite AvailableBinaries() to fetch and parse JSON instead of XML - Update installBinary() to use the new API to find files and construct URLs - Remove all XML parsing logic and iterXMLDirListing function - Remove unused imports (encoding/xml, net/http, net/url, regexp) - Deprecate GoStorageHome field in Manager struct - Field is kept for backward compatibility but no longer used - Added deprecation comment explaining the new API endpoint The new implementation: - Properly filters files by kind='archive' (excludes installers) - Handles platform-specific file extensions (.zip for Windows, .tar.gz for others) - Supports all OS/arch combinations provided by the API - Maintains compatibility with existing version parsing and installation logic Tested with: - Various OS/arch combinations (darwin/arm64, linux/amd64, windows/amd64) - Successfully lists versions from 1.16 to 1.25.3 - Successfully downloads and installs Go binaries using new URLs This change modernizes gvm to use the official API, improving reliability and ensuring continued compatibility as the Go project evolves their distribution infrastructure.
Add package-level documentation comment to gvm.go to satisfy staticcheck ST1000 rule that requires at least one file in a package to have a package comment. The comment describes the purpose of the gvm package and its main capabilities.
andrewkroh
commented
Oct 17, 2025
- Fix ARM architecture handling: check for goarch == 'arm' and convert to 'armv6l' instead of the redundant check for 'armv6l' converting to 'armv6l' - Remove unnecessary matchesArch() function and use direct comparison - Simplify code by removing redundant special cases The ARM special case is needed because: - runtime.GOARCH returns 'arm' on ARM systems - Go only provides pre-built binaries for ARMv6 (labeled as 'armv6l') - The API returns files with arch='armv6l' for ARM binaries - We need to map 'arm' -> 'armv6l' when searching for files
- Change GoStorageHome default from 'https://storage.googleapis.com/golang' to 'https://go.dev/dl' - Use GoStorageHome as the base URL for both API calls and download URLs - Add proper URL handling to ensure correct slash placement in URLs - This allows users to configure alternative download sources if needed The implementation: - fetchGoReleases() constructs the API URL from GoStorageHome + query params - constructDownloadURL() builds download URLs from GoStorageHome + filename - Handles trailing slashes properly to avoid double slashes in URLs - Maintains backward compatibility by keeping the GoStorageHome field
- Update README to reflect the new Go downloads API usage - Update CLI help text to match the new implementation
andrewkroh
commented
Oct 17, 2025
|
|
||
| func (m *Manager) installBinary(version *GoVersion) (string, error) { | ||
| // Fetch releases to find the correct file | ||
| releases, err := m.fetchGoReleases() |
Owner
Author
There was a problem hiding this comment.
Note that this is a slight behavioral change from the previous implementation. The old implementation predicted the download file name from the version, GOOS, and GOARCH to avoid needing to use the index. This one always lists the files in the index then selects the file name based on the GOOS/GOARCH. From looking over the https://go.dev/dl/?mode=json&include=all output, the new approach is more reliable because the file names have changed over the history of releases.
There was a problem hiding this comment.
Pull Request Overview
Migrates gvm from scraping GCS XML listings to using the official go.dev JSON downloads API, modernizing binary discovery and installation and updating defaults/documentation accordingly.
- Replace XML parsing with JSON API client (go.dev/dl).
- Update default storage base URL and CLI/docs to go.dev.
- Add API types and selection logic (archive filtering, OS/arch matching, ARM mapping).
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| gvm.go | Updates package doc and default GoStorageHome to https://go.dev/dl. |
| cmd/gvm/gvm.go | Updates CLI usage message to reference go.dev/dl. |
| binrepo.go | Replaces XML-based listing and direct URL construction with JSON API-based release/file selection and download. |
| api.go | New JSON API client, release/file models, archive selection, and URL construction helpers. |
| README.md | Updates documentation to reference go.dev/dl. |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Based on code review feedback: - Remove custom hasExtension() helper function - Use strings.HasSuffix() directly for checking file extensions - This is more idiomatic Go and handles boundary conditions correctly - Reduces code complexity by removing unnecessary abstraction
Based on code review feedback: - Replace manual string slicing with strings.TrimPrefix() - More idiomatic and clearer intent - Handles edge cases correctly (e.g., versions without 'go' prefix) - Simplifies the code from 4 lines to 1 line
Owner
Author
|
LGTM |
orestisfl
added a commit
to orestisfl/elastic-agent-system-metrics
that referenced
this pull request
Dec 10, 2025
The CI builds were failing with HTTP 403 errors when downloading Go: gvm: error: failed downloading from https://storage.googleapis.com/golang/go1.24.7.windows-amd64.zip: download failed with http status 403 See andrewkroh/gvm#117 that describes the issue on gvm's side. Tested locally: $ /tmp/gvm-0.6.0 --version 0.6.0 $ eval "$(/tmp/gvm-0.6.0 1.24.7)" && go version go version go1.24.7 linux/amd64 Updated gvm version in: - .buildkite/pipeline.yml (SETUP_GVM_VERSION env var) - .buildkite/scripts/run-win-tests.ps1 (hardcoded URL) This aligns with the beats repository configuration.
orestisfl
added a commit
to elastic/elastic-agent-system-metrics
that referenced
this pull request
Dec 11, 2025
## What does this PR do? This PR fixes multiple CI failures: ### 1. Update gvm to v0.6.0 (fixes Go download 403 errors) The CI builds were failing with HTTP 403 errors when downloading Go: ``` gvm: error: failed downloading from https://storage.googleapis.com/golang/go1.24.7.windows-amd64.zip: download failed with http status 403 ``` gvm v0.6.0 migrated to the official Go downloads API at `go.dev/dl`. See: andrewkroh/gvm#117 Updated gvm version in: - `.buildkite/pipeline.yml` (SETUP_GVM_VERSION env var) - `.buildkite/scripts/run-win-tests.ps1` (hardcoded URL) Tested locally: ```sh $ /tmp/gvm-0.6.0 --version 0.6.0 $ eval "$(/tmp/gvm-0.6.0 1.24.7)" && go version go version go1.24.7 linux/amd64 ``` This aligns with the beats repository configuration. ### 2. Skip flaky cgroup tests (issue #270) Container tests fail when cgroups are unavailable due to: - **Private cgroup namespace**: cgroup paths contain `/../..` which can't be resolved - **Non-root user**: permission denied accessing cgroup files These are treated as non-fatal errors in production code. Our testing suite was designed to predict exactly when these failures are going to happen but something has gone wrong in the meantime, more investigation is needed. Changes: - Skip cgroup assertion when `stats.Cgroup == nil` in `TestContainerMonitoringFromInsideContainer` and `TestSelfMonitoringFromInsideContainer` - Pass `CGROUPNSMODE` env var from test framework to inner tests - Only assert cgroups in `validateProcResult` when `cgroupNSMode == "host" && userID == 0` - Filter "Non-fatal error" messages from `FatalLogMessages` check - Improve test logging (replace `Verbose` field with `t.Logf`), go automatically prints logs with -v or on failure. - Improve assertions in `validateProcResult`. The function does not fail immediately anymore and [keeps going](https://go.dev/wiki/TestComments#keep-going) to maximize failing test's context. ## Why is it important? CI is completely broken - no PRs can be merged. ## Checklist - [x] My code follows the style guidelines of this project - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have added tests that prove my fix is effective or that my feature works - [ ] ~~I have added an entry in `CHANGELOG.md`~~
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Code was primarily written by Cursor.
This PR migrates gvm from using Google Cloud Storage XML API to the official Go downloads JSON API at
go.dev/dl. Downloads like https://storage.googleapis.com/golang/go1.24.9.linux-amd64.tar.gz began failing with HTTP 403.Breaking Changes
Key Changes
New API Client (
api.go)GoReleasestruct represents a Go release with version info and downloadable filesGoFilestruct represents individual files with OS/arch/kind metadatafetchGoReleases()fetches data fromhttps://go.dev/dl/?mode=json&include=allfindArchiveFile()finds the correct archive for a given OS/arch combination"arm"to"armv6l"to match API)Updated Binary Repository (
binrepo.go)AvailableBinaries()now fetches and parses JSON from the new APIinstallBinary()uses the new API to find files and construct download URLsManager Configuration (
gvm.go)GoStorageHomedefault fromhttps://storage.googleapis.com/golangtohttps://go.dev/dlGoStorageHomefieldImplementation Details
kind: "archive"(excludes installers)GoStorageHomefield configurableTesting
This modernizes gvm to use the official API, improving reliability and ensuring continued compatibility as the Go project evolves their distribution infrastructure.