Skip to content

Commit b31b2ea

Browse files
cgrindelclaude
andauthored
feat: add utilities to generate rv_checksums and excluded_gems (#283)
* feat: add utilities to generate rv_checksums and excluded_gems Add two Bazel-compatible utilities to automate maintenance of rv-ruby checksums and excluded gems: 1. generate_rv_checksums: Fetches SHA256 checksums from rv-ruby GitHub releases and updates ruby.toolchain() in MODULE.bazel 2. generate_excluded_gems: Queries stdgems.org for native gems and updates ruby.bundle_fetch() excluded_gems in MODULE.bazel Both utilities: - Support --dry-run mode for previewing changes - Use buildozer for automatic MODULE.bazel updates - Include comprehensive test coverage - Use cgrindel_bazel_starlib for consistent error handling Dependencies added: - rules_shell for sh_binary and sh_test support - cgrindel_bazel_starlib for fail/warn/assertions utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> * feat: add generate_rv_checksums * feat: add integration tests and fix buildozer commands for excluded_gems - Add integration test for generate_excluded_gems.sh to verify buildozer updates work correctly - Refactor buildozer command to use array structure and proper list operations ('remove' + 'add' instead of 'set') - Use -types flag with name-based targeting instead of extension syntax - Fix tab indentation in error messages - Add test for name filtering to ensure only target bundle is updated The buildozer command now matches the pattern used in generate_rv_checksums.sh for consistency. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> * fix: use BUILD_WORKSPACE_DIRECTORY for default MODULE.bazel path - Update both generate_rv_checksums.sh and generate_excluded_gems.sh to use BUILD_WORKSPACE_DIRECTORY when available for the default MODULE.bazel path - This fixes the issue where running via `bazel run` couldn't find MODULE.bazel because it was looking in the execution directory instead of the workspace root - When BUILD_WORKSPACE_DIRECTORY is not set (direct script execution), it falls back to ./MODULE.bazel as before - Move buildifier_prebuilt from dev_dependency to regular dependency since it's required by the generator tools at runtime - Fix line wrapping in integration test comments 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> * docs: update rv-ruby section with utility documentation - Remove WORKSPACE examples from rv-ruby section (focus on Bzlmod) - Add "Configure rv-ruby Downloads" section documenting generate_rv_checksums utility - Add "Configure Excluded Gems" section documenting generate_excluded_gems utility - Fix rb_bundle_fetch to ruby.bundle_fetch for Bzlmod syntax - Add utility options documentation (--name, --module-bazel, --dry-run) - Improve formatting and organization of rv-ruby configuration steps - Add note about compilation errors when excluded_gems is not set The documentation now provides clear, step-by-step instructions for using the automated utilities instead of manually configuring checksums and excluded gems. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> * refactor: improve shell script formatting and remove debug code - Remove redundant set -o errexit declarations (handled by runfiles.bash) - Add shellcheck disable directives for dynamic sourcing in test files - Remove debug echo statements from integration tests - Wrap long lines to improve readability - Fix line continuation for better formatting consistency 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> * feat: add curl timeouts and improve documentation - Add --max-time 30 to curl commands to prevent indefinite hangs - Enhance get_minor_version comment to explain Ruby versioning structure and why stdgems data is organized by minor version 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> * chore: add verbose failures and detailed test output on error * fix: try escaping the curly bracket * refactor: remove unnecessary echo output from tests Remove all echo statements to STDOUT from test files to keep test output clean. Tests should only output on failure. Changes: - Remove DEBUG echo statements from integration tests - Remove TEST/PASS progress messages from all tests - Remove summary "All tests passed!" messages - Keep echo statements redirected to stderr or files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.5 <[email protected]>
1 parent 3e6a87e commit b31b2ea

16 files changed

+1336
-29
lines changed

.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ jobs:
2424
bazelrc: common --announce_rc --color=yes
2525
repository-cache: true
2626
- run: bazel run :buildifier.check
27-
- run: bazel test ...
27+
- run: bazel test --verbose_failures --test_output=errors --test_summary=detailed ...
2828
- if: failure() && runner.debug == '1'
2929
uses: mxschmitt/action-tmate@v3
3030

MODULE.bazel

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,16 @@ module(
1212
# the versions resolved in users repositories.
1313
bazel_dep(name = "bazel_features", version = "1.9.0")
1414
bazel_dep(name = "bazel_skylib", version = "1.3.0")
15+
bazel_dep(name = "cgrindel_bazel_starlib", version = "0.27.0")
1516
bazel_dep(name = "platforms", version = "0.0.5")
1617
bazel_dep(name = "rules_cc", version = "0.0.9")
1718
bazel_dep(name = "rules_java", version = "7.2.0")
19+
bazel_dep(name = "rules_shell", version = "0.4.1")
20+
bazel_dep(name = "buildifier_prebuilt", version = "8.2.1.1")
1821

1922
# Ruleset development dependencies.
2023
bazel_dep(name = "aspect_bazel_lib", version = "2.22.2", dev_dependency = True)
2124
bazel_dep(name = "bazel_skylib_gazelle_plugin", version = "1.9.0", dev_dependency = True)
22-
bazel_dep(name = "buildifier_prebuilt", version = "8.2.1.1", dev_dependency = True)
2325
bazel_dep(name = "gazelle", version = "0.47.0", dev_dependency = True)
2426
bazel_dep(name = "rules_go", version = "0.59.0", dev_dependency = True)
2527
bazel_dep(name = "stardoc", version = "0.8.0", dev_dependency = True)

README.md

Lines changed: 44 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -130,24 +130,23 @@ For faster MRI installation on Linux and macOS, you can use prebuilt Ruby
130130
binaries from [rv-ruby][19] instead of compiling from source. This significantly
131131
reduces installation time and ensures consistent, portable Ruby environments.
132132

133-
**WORKSPACE:**
133+
**Configure rv-ruby Downloads**
134134

135-
```bazel
136-
load("@rules_ruby//ruby:deps.bzl", "rb_register_toolchains")
135+
To securely download and properly cache the Ruby binaries, the `ruby.toolchain`
136+
declaration must be updated with the `rv_version` and `rv_checksums` attributes.
137137

138-
rb_register_toolchains(
139-
version = "3.4.8",
140-
rv_version = "20251225",
141-
rv_checksums = {
142-
"linux-x86_64": "f36cef10365d370e0867f0c3ac36e457a26ab04f3cfbbd7edb227a18e6e9b3c3",
143-
"linux-arm64": "0c08c35a99f10817643d548f98012268c5433ae25a737ab4d6751336108a941d",
144-
"macos-x86_64": "e9da39082d1dd8502d322c850924d929bc45b7a1e35da593a5606c00673218d4",
145-
"macos-arm64": "cd9d7a1428076bfcc6c2ca3c0eb69b8e671e9b48afb4c351fa4a84927841ffef",
146-
},
147-
)
138+
We have provided the `generate_rv_checksums` utility to add/update these
139+
attributes for you. The utility needs to know the `rv-ruby` version to use
140+
(https://github.com/spinel-coop/rv-ruby/releases) and the version of Ruby to
141+
download. By default, it will use the Ruby version specified in the
142+
`.ruby-version` file.
143+
144+
```bash
145+
bazel run @rules_ruby//tools/generate_rv_checksums -- 20251225
148146
```
149147

150-
**Bzlmod:**
148+
After running the utility, the toolchain declaration in your `MODULE.bazel`
149+
should look something like the following:
151150

152151
```bazel
153152
ruby = use_extension("@rules_ruby//ruby:extensions.bzl", "ruby")
@@ -156,26 +155,39 @@ ruby.toolchain(
156155
version_file = "//:.ruby-version",
157156
rv_version = "20251225",
158157
rv_checksums = {
159-
"linux-x86_64": "f36cef10365d370e0867f0c3ac36e457a26ab04f3cfbbd7edb227a18e6e9b3c3",
160158
"linux-arm64": "0c08c35a99f10817643d548f98012268c5433ae25a737ab4d6751336108a941d",
161-
"macos-x86_64": "e9da39082d1dd8502d322c850924d929bc45b7a1e35da593a5606c00673218d4",
159+
"linux-x86_64": "f36cef10365d370e0867f0c3ac36e457a26ab04f3cfbbd7edb227a18e6e9b3c3",
162160
"macos-arm64": "cd9d7a1428076bfcc6c2ca3c0eb69b8e671e9b48afb4c351fa4a84927841ffef",
161+
"macos-x86_64": "e9da39082d1dd8502d322c850924d929bc45b7a1e35da593a5606c00673218d4",
163162
},
164163
)
165164
```
166165

167-
**Important:** When using rv-ruby, you must exclude default gems with C extensions
168-
from `rb_bundle_fetch` as these are pre-compiled in the rv-ruby binary:
166+
**Configure Excluded Gems**
167+
168+
When using `rv-ruby`, you must exclude _default_ gems with C extensions from
169+
`bundle_fetch` as these are pre-compiled in the `rv-ruby` binary. You may see
170+
compilation errors if you do not exclude these gems.
171+
172+
We have provided the `generate_excluded_gems` utility to update the declaration
173+
for you.
174+
175+
```bash
176+
bazel run @rules_ruby//tools/generate_excluded_gems
177+
```
178+
179+
The utility reads the Ruby version being used and checks
180+
https://raw.githubusercontent.com/janlelis/stdgems/main/default_gems.json to
181+
determine which gems should be excluded. The utility adds/updates the
182+
`excluded_gems` attribute with the correct list of gems. The `bundle_fetch`
183+
declaration will look something like the following:
169184

170185
```bazel
171-
rb_bundle_fetch(
186+
ruby.bundle_fetch(
172187
name = "bundle",
173188
gemfile = "//:Gemfile",
174189
gemfile_lock = "//:Gemfile.lock",
175190
excluded_gems = [
176-
# Default gems with C extensions from https://stdgems.org/3.4.8
177-
# These are pre-compiled in rv-ruby with portable dependencies.
178-
# IMPORTANT: These gems must also be pinned in your Gemfile.
179191
"date", "digest", "etc", "fcntl", "fiddle",
180192
"io-console", "io-nonblock", "io-wait", "json",
181193
"openssl", "pathname", "prism", "psych",
@@ -184,15 +196,20 @@ rb_bundle_fetch(
184196
)
185197
```
186198

187-
Find the list of default gems for your Ruby version at https://stdgems.org/\<version\>
188-
(e.g., https://stdgems.org/3.4.8 for Ruby 3.4.8). Only exclude gems with C
189-
extensions. Bundled gems should NOT be excluded.
199+
> [!NOTE]
200+
> You can find an HTML-rendered list of the default gems for a Ruby version at
201+
> https://stdgems.org/\<version\> (e.g., https://stdgems.org/3.4.8 for Ruby
202+
> 3.4.8).
190203
191204
**Notes:**
192205

193-
- rv-ruby is only supported on Linux and macOS (x86_64 and arm64).
206+
- `rv-ruby` is only supported on Linux and macOS (x86_64 and arm64).
194207
- On Windows, the toolchain automatically falls back to RubyInstaller.
195-
- Find available rv-ruby releases at https://github.com/spinel-coop/rv-ruby/releases
208+
- Find available `rv-ruby` releases at
209+
https://github.com/spinel-coop/rv-ruby/releases
210+
- The utilities support `--name` to target specific toolchains/bundles and
211+
`--module-bazel` to specify a custom MODULE.bazel path.
212+
- Run utilities with `--dry-run` to preview changes without modifying files.
196213

197214
### JRuby
198215

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
load("@rules_shell//shell:sh_binary.bzl", "sh_binary")
2+
load("@rules_shell//shell:sh_test.bzl", "sh_test")
3+
4+
sh_binary(
5+
name = "generate_excluded_gems",
6+
srcs = ["generate_excluded_gems.sh"],
7+
data = [
8+
"@buildifier_prebuilt//buildozer",
9+
],
10+
visibility = ["//visibility:public"],
11+
deps = [
12+
"@cgrindel_bazel_starlib//shlib/lib:fail",
13+
"@rules_shell//shell/runfiles",
14+
],
15+
)
16+
17+
sh_test(
18+
name = "generate_excluded_gems_test",
19+
srcs = ["generate_excluded_gems_test.sh"],
20+
data = [
21+
"testdata/default_gems.json",
22+
"testdata/expected_excluded_gems_output.txt",
23+
":generate_excluded_gems",
24+
],
25+
deps = [
26+
"@cgrindel_bazel_starlib//shlib/lib:assertions",
27+
"@rules_shell//shell/runfiles",
28+
],
29+
)
30+
31+
sh_test(
32+
name = "generate_excluded_gems_integration_test",
33+
srcs = ["generate_excluded_gems_integration_test.sh"],
34+
data = [
35+
"testdata/default_gems.json",
36+
":generate_excluded_gems",
37+
],
38+
deps = [
39+
"@cgrindel_bazel_starlib//shlib/lib:assertions",
40+
"@rules_shell//shell/runfiles",
41+
],
42+
)
43+
44+
filegroup(
45+
name = "all_files",
46+
srcs = glob(["**/*"]),
47+
visibility = ["//:__subpackages__"],
48+
)
Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
#!/usr/bin/env bash
2+
3+
# Generates excluded_gems for ruby.bundle_fetch() and updates MODULE.bazel.
4+
5+
# --- begin runfiles.bash initialization v3 ---
6+
# Copy-pasted from the Bazel Bash runfiles library v3.
7+
set -uo pipefail
8+
set +e
9+
f=bazel_tools/tools/bash/runfiles/runfiles.bash
10+
# shellcheck disable=SC1090
11+
source "${RUNFILES_DIR:-/dev/null}/${f}" 2>/dev/null \
12+
|| source "$(grep -sm1 "^${f} " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null \
13+
|| source "$0.runfiles/${f}" 2>/dev/null \
14+
|| source "$(grep -sm1 "^${f} " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null \
15+
|| source "$(grep -sm1 "^${f} " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null \
16+
|| {
17+
echo >&2 "ERROR: ${BASH_SOURCE[0]} cannot find ${f}"
18+
exit 1
19+
}
20+
f=
21+
set -e
22+
# --- end runfiles.bash initialization v3 ---
23+
24+
# MARK - Dependencies
25+
26+
fail_sh_location=cgrindel_bazel_starlib/shlib/lib/fail.sh
27+
fail_sh="$(rlocation "${fail_sh_location}")" \
28+
|| (echo >&2 "Failed to locate ${fail_sh_location}" && exit 1)
29+
# shellcheck disable=SC1090
30+
source "${fail_sh}"
31+
32+
# Locate buildozer via runfiles
33+
buildozer_location=buildifier_prebuilt/buildozer/buildozer
34+
buildozer="$(rlocation "${buildozer_location}")" \
35+
|| (echo >&2 "Failed to locate ${buildozer_location}" && exit 1)
36+
37+
# MARK - Default Values
38+
39+
# Default values
40+
dry_run=false
41+
name="bundle"
42+
module_bazel="${BUILD_WORKSPACE_DIRECTORY:-.}/MODULE.bazel"
43+
ruby_version=""
44+
45+
# MARK - Functions
46+
47+
# Read .ruby-version file
48+
read_ruby_version() {
49+
local version_file="${BUILD_WORKSPACE_DIRECTORY}/.ruby-version"
50+
if [[ ! -f ${version_file} ]]; then
51+
fail "Error: .ruby-version not found and --ruby-version not specified"
52+
fi
53+
tr -d '[:space:]' <"${version_file}"
54+
}
55+
56+
# Extract minor version from a Ruby version string (e.g., 3.4.8 -> 3.4)
57+
# Ruby uses MAJOR.MINOR.PATCH versioning, and stdgems data is organized by
58+
# minor version since default gems are typically consistent within a minor
59+
# release series.
60+
get_minor_version() {
61+
local version="$1"
62+
echo "${version}" | cut -d. -f1,2
63+
}
64+
65+
# MARK - Argument Handling
66+
67+
# Parse arguments
68+
while (("$#")); do
69+
case "${1}" in
70+
--dry-run)
71+
dry_run=true
72+
shift
73+
;;
74+
--ruby-version)
75+
ruby_version="${2}"
76+
shift 2
77+
;;
78+
--name)
79+
name="${2}"
80+
shift 2
81+
;;
82+
--module-bazel)
83+
module_bazel="${2}"
84+
shift 2
85+
;;
86+
-*)
87+
fail "Error: Unknown option: ${1}"
88+
;;
89+
*)
90+
fail "Error: Unexpected argument: ${1}"
91+
;;
92+
esac
93+
done
94+
95+
# Get Ruby version
96+
if [[ -z ${ruby_version} ]]; then
97+
ruby_version="$(read_ruby_version)"
98+
fi
99+
100+
# Get minor version
101+
minor_version=$(get_minor_version "${ruby_version}")
102+
103+
# MARK - Retrieve stdgems data
104+
105+
# Fetch stdgems data
106+
stdgems_url="${STDGEMS_URL:-https://raw.githubusercontent.com/janlelis/stdgems/main/default_gems.json}"
107+
response=$(curl -sL --max-time 30 "${stdgems_url}")
108+
109+
# Filter for native gems that exist for this Ruby version
110+
# The jq query:
111+
# 1. Select gems where native == true
112+
# 2. Select gems where versions contains the minor version key
113+
# 3. Extract the gem name
114+
# 4. Sort
115+
excluded_gems=$(echo "${response}" | jq -r --arg version "${minor_version}" \
116+
'.gems[] | select(.native == true) | select(.versions | has($version)) | .gem' \
117+
| sort)
118+
119+
# Check if we found any gems
120+
if [[ -z ${excluded_gems} ]]; then
121+
fail <<-EOT
122+
Error: No native gems found for Ruby ${ruby_version} (${minor_version})
123+
This Ruby version may not be supported in stdgems data
124+
EOT
125+
fi
126+
127+
# MARK - Update MODULE.bazel
128+
129+
# Generate output for dry-run or display
130+
output="excluded_gems = [\n"
131+
while IFS= read -r gem; do
132+
output+=" \"${gem}\",\n"
133+
done <<<"${excluded_gems}"
134+
output+="],"
135+
136+
if [[ ${dry_run} == "true" ]]; then
137+
# Dry-run: just output the excluded gems
138+
echo -e "${output}"
139+
exit 0
140+
fi
141+
142+
# Construct list of gems for buildozer 'add' command
143+
# Convert newline-separated list to space-separated
144+
gem_list=""
145+
while IFS= read -r gem; do
146+
gem_list+=" ${gem}"
147+
done <<<"${excluded_gems}"
148+
149+
# Update MODULE.bazel using buildozer
150+
buildozer_cmd=(
151+
"${buildozer}"
152+
-types ruby.bundle_fetch
153+
"remove excluded_gems"
154+
"add excluded_gems${gem_list}"
155+
"${module_bazel}:${name}"
156+
)
157+
if ! "${buildozer_cmd[@]}" 2>/dev/null; then
158+
fail <<-EOT
159+
Failed to update ${module_bazel}
160+
161+
Buildozer command failed. This could mean:
162+
- The file doesn't exist at ${module_bazel}
163+
- No ruby.bundle_fetch() call was found with name="${name}"
164+
- The file has syntax errors
165+
166+
You can use --dry-run to see what would be updated:
167+
$(echo -e "${output}")
168+
EOT
169+
fi
170+
171+
echo "Successfully updated excluded_gems in ${module_bazel}"

0 commit comments

Comments
 (0)