Skip to content

Commit 9ff730e

Browse files
konardkonard
authored
Merge pull request #47 from link-foundation/issue-46-2e5336b482c3
Fix CI failure: relative JSON path breaks under su - sandbox (Issue #46)
2 parents 7f1e343 + 06a5d80 commit 9ff730e

File tree

4 files changed

+405
-2
lines changed

4 files changed

+405
-2
lines changed

‎.changeset/fix-relative-json-path-under-su-sandbox.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
bump: patch
3+
---
4+
5+
Fix CI failure: relative JSON path breaks under `su - sandbox` (Issue #46)
6+
7+
`su - sandbox` (login shell) changes the working directory from the runner's workspace to the sandbox user's home directory. Convert `JSON_OUTPUT_FILE` to an absolute path using `realpath` before passing it to the sandbox user subprocess, and grant the sandbox user read/write access to the JSON file and its parent directory.

‎docs/case-studies/issue-46/CASE-STUDY.md‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# Case Study: Failing CI/CD Run — "No such file or directory" for JSON Measurements
2+
3+
**Issue**: [#46 - Fix failing CI/CD run](https://github.com/link-foundation/sandbox/issues/46)
4+
**CI Run**: [22261112919, Job 64399507098](https://github.com/link-foundation/sandbox/actions/runs/22261112919/job/64399507098)
5+
**Date**: 2026-02-21
6+
**Status**: Investigation Complete — Fix Applied
7+
8+
## Executive Summary
9+
10+
The "Measure Disk Space and Update README" workflow failed with:
11+
12+
```
13+
cat: data/disk-space-measurements.json: No such file or directory
14+
##[error]Process completed with exit code 1.
15+
```
16+
17+
The root cause is that `su - sandbox` (login shell) changes the working directory to the sandbox user's home (`/home/sandbox`). The relative path `data/disk-space-measurements.json` then resolves to `/home/sandbox/data/disk-space-measurements.json` — a file that was never created there. The JSON file was correctly initialized in the runner's working directory (`/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json`), but the sandbox-measure.sh script tried to read and write a different file.
18+
19+
**Fix**: Convert the relative JSON output path to an absolute path before passing it to `su - sandbox`.
20+
21+
## Timeline of Events
22+
23+
| Time (UTC) | Event |
24+
|------------|-------|
25+
| 2026-02-21T17:29:57 | Job started on ubuntu-24.04 runner (version 20260201.15.1) |
26+
| 2026-02-21T17:31:25 | `measure-disk-space.sh` invoked: `sudo ./scripts/measure-disk-space.sh --json-output data/disk-space-measurements.json` |
27+
| 2026-02-21T17:31:26 | JSON initialized at `data/disk-space-measurements.json` (relative to runner CWD: `/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json`) |
28+
| 2026-02-21T17:31:26 | Measurements begin for system components (Essential Tools, .NET, C/C++, etc.) |
29+
| 2026-02-21T17:32:57 | GitLab CLI recorded as last successful system measurement |
30+
| 2026-02-21T17:32:57 | `su - sandbox -c "bash /tmp/sandbox-measure.sh 'data/disk-space-measurements.json'"` executed |
31+
| 2026-02-21T17:32:57 | Login shell changes CWD to `/home/sandbox`; relative path resolves to wrong location |
32+
| 2026-02-21T17:32:58 | Bun installs successfully to `/home/sandbox/.bun/bin/bun` |
33+
| 2026-02-21T17:32:58 | `add_measurement "Bun"` calls `cat data/disk-space-measurements.json` — **file not found** |
34+
| 2026-02-21T17:32:58 | Script exits with code 1; `set -euo pipefail` propagates failure |
35+
| 2026-02-21T17:32:58 | Workflow step fails: `cat: data/disk-space-measurements.json: No such file or directory` |
36+
37+
## Root Cause Analysis
38+
39+
### Primary Cause: `su -` (Login Shell) Changes Working Directory
40+
41+
The `su -` command (equivalent to `su --login` or `su -l`) simulates a full login for the target user. According to the [Linux `su` man page](https://manpages.ubuntu.com/manpages/focal/en/man1/su.1.html) and [Linuxize's su documentation](https://linuxize.com/post/su-command-in-linux/):
42+
43+
> When using `su -`, the command **changes the current working directory to the home directory of the target user**. All environment variables are reset to the target user's environment.
44+
45+
The relevant code in `measure-disk-space.sh` (line 646):
46+
47+
```bash
48+
su - sandbox -c "bash /tmp/sandbox-measure.sh '$JSON_OUTPUT_FILE'"
49+
```
50+
51+
Where `JSON_OUTPUT_FILE` is `data/disk-space-measurements.json` (a relative path).
52+
53+
**Before `su - sandbox`**: Working directory = `/home/runner/work/sandbox/sandbox`
54+
- Relative path resolves to: `/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json` ✓ (file exists)
55+
56+
**After `su - sandbox`**: Working directory = `/home/sandbox`
57+
- Relative path resolves to: `/home/sandbox/data/disk-space-measurements.json` ✗ (file does not exist)
58+
59+
The sandbox-measure.sh script's `add_measurement` function then fails at:
60+
61+
```bash
62+
current_json=$(cat "$JSON_OUTPUT_FILE") # Fails: No such file or directory
63+
```
64+
65+
This error propagates through `set -euo pipefail`, causing the sandbox-measure.sh to exit with code 1, which then causes the outer script and the workflow step to fail.
66+
67+
### Why Bun Appears in the Error
68+
69+
The failure appears immediately after Bun installation because Bun is the **first measurement** in sandbox-measure.sh. The successful Bun installation output (`bun was installed successfully to ~/.bun/bin/bun`) appears before the JSON read failure because:
70+
71+
1. `measure_install "Bun" "Runtime" install_bun` — calls `install_bun` which succeeds
72+
2. `add_measurement "Bun" ...` — calls `cat "$JSON_OUTPUT_FILE"` which fails
73+
74+
Bun itself is **not the cause** of the failure. The error would have occurred even if Bun installation was skipped, as the JSON file issue affects all measurements.
75+
76+
### Why the JSON File Was "Missing"
77+
78+
The JSON file was never truly missing — it was created correctly at:
79+
```
80+
/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json
81+
```
82+
83+
But the sandbox user's script looked for it at:
84+
```
85+
/home/sandbox/data/disk-space-measurements.json
86+
```
87+
88+
This directory (`/home/sandbox/data/`) was never created, making the path completely inaccessible to the sandbox user. Additionally, even if the directory existed, the sandbox user would not normally have write access to the runner's working directory structure.
89+
90+
### Contributing Factors
91+
92+
1. **`set -euo pipefail` in outer script**: `set -euo pipefail` at the top of `measure-disk-space.sh` (line 2) ensures any failure in the subprocess chain propagates correctly — this is correct behavior, but amplifies the visibility of the root cause bug.
93+
94+
2. **`su -` vs `su`**: The login shell option (`-`) was chosen deliberately to give the sandbox user a clean environment for installation. This is the correct approach for user-space installations, but requires careful handling of paths. According to [Baeldung's analysis](https://www.baeldung.com/linux/su-command-options), `su -` "simulates a full login" and "resets environment variables and changes to the user's home directory."
95+
96+
3. **Relative path throughout**: The JSON output file is passed as a relative path from the CLI argument `--json-output data/disk-space-measurements.json`. This relative path is never converted to an absolute path, so it becomes invalid after a working directory change.
97+
98+
## Solution
99+
100+
### Fix: Convert to Absolute Path Before `su - sandbox`
101+
102+
The fix is to resolve the JSON output path to an absolute path before executing the sandbox user's sub-script. This ensures the path remains valid regardless of what working directory the sub-shell starts in.
103+
104+
```bash
105+
# Convert JSON_OUTPUT_FILE to absolute path before passing to su
106+
JSON_OUTPUT_FILE_ABS="$(realpath "$JSON_OUTPUT_FILE")"
107+
108+
if [ "$EUID" -eq 0 ]; then
109+
su - sandbox -c "bash /tmp/sandbox-measure.sh '$JSON_OUTPUT_FILE_ABS'"
110+
else
111+
sudo -i -u sandbox bash /tmp/sandbox-measure.sh "$JSON_OUTPUT_FILE_ABS"
112+
fi
113+
```
114+
115+
Additionally, the `sandbox-measure.sh` script needs to create the parent directory if it doesn't exist, since the sandbox user may not have the `data/` directory in their home:
116+
117+
```bash
118+
# At the start of sandbox-measure.sh
119+
mkdir -p "$(dirname "$JSON_OUTPUT_FILE")"
120+
```
121+
122+
### Why This Fix Is Correct
123+
124+
- `realpath` converts relative paths to absolute paths using the current working directory
125+
- The absolute path `/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json` is unambiguous and valid from any working directory
126+
- The sandbox user needs write access to this path — since the runner runs with elevated privileges and the outer script (running as root) creates the file, the sandbox user needs read/write access, which can be ensured by pre-creating the file with appropriate permissions
127+
128+
### Alternative Approaches Considered
129+
130+
1. **Use `su` without `-`** (no login shell): Would preserve the working directory but break user-space tool installations that expect a clean home environment. Not recommended.
131+
132+
2. **Pass absolute path directly from CLI**: Requires users to always specify absolute paths in the workflow, which is error-prone.
133+
134+
3. **Use `cd` inside `su -` command**: Adding `cd /path/to/workdir &&` before the script call would work but is fragile.
135+
136+
4. **Write JSON to `/tmp/` instead of `data/`**: Would avoid the working directory issue but changes the documented output location.
137+
138+
## Technical Details
139+
140+
### Environment Comparison
141+
142+
| Context | Working Directory | `data/disk-space-measurements.json` resolves to |
143+
|---------|------------------|--------------------------------------------------|
144+
| Runner shell | `/home/runner/work/sandbox/sandbox` | `/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json` ✓ |
145+
| Root (outer script) | `/home/runner/work/sandbox/sandbox` | `/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json` ✓ |
146+
| `su - sandbox` | `/home/sandbox` | `/home/sandbox/data/disk-space-measurements.json` ✗ |
147+
148+
### Affected Script Section
149+
150+
**File**: `scripts/measure-disk-space.sh`, lines 644–649
151+
152+
```bash
153+
# Execute sandbox user measurements
154+
if [ "$EUID" -eq 0 ]; then
155+
su - sandbox -c "bash /tmp/sandbox-measure.sh '$JSON_OUTPUT_FILE'"
156+
else
157+
sudo -i -u sandbox bash /tmp/sandbox-measure.sh "$JSON_OUTPUT_FILE"
158+
fi
159+
```
160+
161+
The `sudo -i -u sandbox` also creates a login shell (`-i` flag = simulate initial login), so the same path issue would affect that branch too.
162+
163+
### Bun Installation Script Behavior
164+
165+
For reference, Bun's install script (`https://bun.sh/install`) uses `set -euo pipefail` and adds `~/.bun/bin` to `$HOME/.bash_profile`. The script exits 0 on success and 1 on failure. In this CI run, Bun installed successfully (exit 0), confirming that Bun itself is not the failure source.
166+
167+
## CI Run Data
168+
169+
- **Workflow**: "Measure Disk Space and Update README"
170+
- **Job**: "Measure Component Disk Space"
171+
- **Run ID**: 22261112919
172+
- **Job ID**: 64399507098
173+
- **Runner**: ubuntu-24.04, version 20260201.15.1
174+
- **Commit**: 38673b0f4aa8c91069a9473a5df1d157c8522584 (main)
175+
- **Trigger**: Push to main branch
176+
177+
### Components Successfully Recorded Before Failure
178+
179+
| Component | Category | Size |
180+
|-----------|----------|------|
181+
| Essential Tools | System | 0MB |
182+
| .NET SDK 8.0 | Runtime | 481MB |
183+
| C/C++ Tools (CMake, Clang, LLVM, LLD) | Build Tools | 56MB |
184+
| Assembly Tools (NASM, FASM) | Build Tools | 3MB |
185+
| R Language | Runtime | 115MB |
186+
| Ruby Build Dependencies | Dependencies | 0MB |
187+
| Python Build Dependencies | Dependencies | 40MB |
188+
| GitHub CLI | Development Tools | 0MB |
189+
| GitLab CLI | Development Tools | 27MB |
190+
191+
### Components Not Recorded (Sandbox User Installations)
192+
193+
All sandbox user installations failed due to the JSON path issue:
194+
- Bun, gh-setup-git-identity, glab-setup-git-identity, Deno, NVM + Node.js 20
195+
- Pyenv + Python, Go, Rust, SDKMAN + Java 21, Kotlin, Lean, Opam + Rocq
196+
- Homebrew, PHP 8.3, Perlbrew + Perl, rbenv + Ruby, Swift
197+
198+
## References
199+
200+
### External Links
201+
202+
1. [Ubuntu Manpage: su (focal)](https://manpages.ubuntu.com/manpages/focal/en/man1/su.1.html) — official documentation for `su` behavior
203+
2. [Linuxize: Su Command in Linux](https://linuxize.com/post/su-command-in-linux/) — `su -` changes to target user's home directory
204+
3. [Baeldung: Why Do We Use su – and Not Just su?](https://www.baeldung.com/linux/su-command-options) — explains login shell environment reset
205+
4. [Bun Installation Documentation](https://bun.sh/docs/installation) — Bun install script behavior
206+
5. [Bun install script source](https://bun.sh/install) — uses `set -euo pipefail`, adds to `~/.bash_profile`
207+
208+
### Internal Logs
209+
210+
- Full CI log: `ci-run-log.txt` (saved locally during investigation)
211+
- CI run: `gh run view 22261112919 --repo link-foundation/sandbox --log`
212+
213+
---
214+
215+
*Case study compiled: 2026-02-21*
216+
*Investigation by: AI Issue Solver*

‎docs/case-studies/issue-46/DATA-COLLECTION.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Data Collection: Issue #46 CI/CD Failure
2+
3+
## Raw Data Sources
4+
5+
### CI Run Information
6+
7+
- **Workflow Run URL**: https://github.com/link-foundation/sandbox/actions/runs/22261112919/job/64399507098
8+
- **Run ID**: 22261112919
9+
- **Job ID**: 64399507098
10+
- **Job Name**: "Measure Component Disk Space"
11+
- **Workflow**: "Measure Disk Space and Update README"
12+
- **Runner**: ubuntu-24.04 (version 20260201.15.1)
13+
- **Commit**: 38673b0f4aa8c91069a9473a5df1d157c8522584 (main branch)
14+
- **Trigger**: Push to main
15+
16+
### Log Collection Commands
17+
18+
```bash
19+
# List recent runs
20+
gh run list --repo link-foundation/sandbox --limit 5 --json databaseId,conclusion,createdAt,headSha
21+
22+
# Download run logs
23+
gh run view 22261112919 --repo link-foundation/sandbox --log > ci-logs/measure-disk-space-22261112919.log
24+
25+
# View specific job
26+
gh run view 22261112919 --repo link-foundation/sandbox --job 64399507098
27+
```
28+
29+
## Key Log Excerpts
30+
31+
### Error Occurrence (Lines 1618-1619 of full log)
32+
33+
```
34+
2026-02-21T17:32:58.8658778Z cat: data/disk-space-measurements.json: No such file or directory
35+
2026-02-21T17:32:58.8708101Z ##[error]Process completed with exit code 1.
36+
```
37+
38+
### JSON Initialization (Line 254 of full log)
39+
40+
```
41+
2026-02-21T17:31:26.0172612Z [*] Initialized JSON output at data/disk-space-measurements.json
42+
```
43+
44+
### Context Switch to Sandbox User (Lines 1598-1617)
45+
46+
```
47+
2026-02-21T17:32:57.2186148Z [✓] Recorded: GitLab CLI - 27MB
48+
2026-02-21T17:32:57.2186516Z
49+
2026-02-21T17:32:57.2186621Z ==> Preparing Homebrew Directory
50+
...
51+
2026-02-21T17:32:57.2266635Z ==> Measuring Sandbox User Installations
52+
...
53+
2026-02-21T17:32:57.4165096Z [*] Measuring: Bun
54+
2026-02-21T17:32:58.0291267Z #=#=#
55+
2026-02-21T17:32:58.0292030Z
56+
2026-02-21T17:32:58.0480819Z ######################################################### 80.3%
57+
2026-02-21T17:32:58.0481863Z ######################################################################## 100.0%
58+
2026-02-21T17:32:58.8174069Z bun was installed successfully to ~/.bun/bin/bun
59+
2026-02-21T17:32:58.8182037Z
60+
2026-02-21T17:32:58.8235749Z Added "~/.bun/bin" to $PATH in "~/.bash_profile"
61+
2026-02-21T17:32:58.8236217Z
62+
2026-02-21T17:32:58.8237067Z To get started, run:
63+
2026-02-21T17:32:58.8237409Z
64+
2026-02-21T17:32:58.8284531Z source /home/sandbox/.bash_profile
65+
2026-02-21T17:32:58.8285158Z bun --help
66+
2026-02-21T17:32:58.8658778Z cat: data/disk-space-measurements.json: No such file or directory
67+
2026-02-21T17:32:58.8708101Z ##[error]Process completed with exit code 1.
68+
```
69+
70+
### Workflow Step Command (Lines 224-246)
71+
72+
```
73+
2026-02-21T17:31:25.9988774Z ##[group]Run set -o pipefail
74+
2026-02-21T17:31:25.9991794Z sudo ./scripts/measure-disk-space.sh --json-output data/disk-space-measurements.json 2>&1 | tee measurement.log
75+
```
76+
77+
## Environment
78+
79+
### Runner Configuration
80+
81+
| Property | Value |
82+
|----------|-------|
83+
| OS | Ubuntu 24.04.3 LTS |
84+
| Runner Version | 2.331.0 |
85+
| Image Version | 20260201.15.1 |
86+
| Azure Region | westus |
87+
| Initial Disk Space | 145G total, 53G used, 92G available |
88+
89+
### Disk Space After Free-up Step
90+
91+
```
92+
2026-02-21T17:31:26.0172612Z Baseline disk usage: 33541MB
93+
```
94+
95+
Directories removed:
96+
- `/usr/share/dotnet`
97+
- `/usr/local/lib/android`
98+
- `/opt/ghc`
99+
- `/opt/hostedtoolcache`
100+
- `/usr/local/share/boost`
101+
- `$AGENT_TOOLSDIRECTORY`
102+
103+
## Script Analysis
104+
105+
### `scripts/measure-disk-space.sh` Key Sections
106+
107+
**Argument Parsing (lines 13-16)**:
108+
```bash
109+
JSON_OUTPUT_FILE="${1:-/tmp/disk-space-measurements.json}"
110+
if [[ "$1" == "--json-output" ]] && [[ -n "${2:-}" ]]; then
111+
JSON_OUTPUT_FILE="$2"
112+
fi
113+
```
114+
The path `data/disk-space-measurements.json` is stored as-is (relative).
115+
116+
**JSON Initialization (lines 85-94)**:
117+
```bash
118+
init_json_output() {
119+
cat > "$JSON_OUTPUT_FILE" << 'EOF'
120+
{...}
121+
EOF
122+
log_info "Initialized JSON output at $JSON_OUTPUT_FILE"
123+
}
124+
```
125+
File created successfully at the relative path — works because CWD is the runner's workspace.
126+
127+
**Sandbox User Execution (lines 644-649)**:
128+
```bash
129+
if [ "$EUID" -eq 0 ]; then
130+
su - sandbox -c "bash /tmp/sandbox-measure.sh '$JSON_OUTPUT_FILE'"
131+
else
132+
sudo -i -u sandbox bash /tmp/sandbox-measure.sh "$JSON_OUTPUT_FILE"
133+
fi
134+
```
135+
The `-` flag to `su` and `-i` flag to `sudo` both create login shells, changing CWD to `/home/sandbox`.
136+
137+
**sandbox-measure.sh `add_measurement` (lines 335-353)**:
138+
```bash
139+
add_measurement() {
140+
local current_json
141+
current_json=$(cat "$JSON_OUTPUT_FILE") # <-- FAILS: reads wrong path
142+
...
143+
echo "$current_json" > "$JSON_OUTPUT_FILE" # Would also fail
144+
...
145+
}
146+
```
147+
148+
## Proof of Root Cause
149+
150+
The evidence chain:
151+
152+
1. **JSON initialized at relative path** in runner's CWD (`/home/runner/work/sandbox/sandbox/data/disk-space-measurements.json`) — log line 254 confirms this.
153+
154+
2. **`su - sandbox` changes CWD** to `/home/sandbox` — this is documented behavior of login shells.
155+
156+
3. **First read of JSON fails** in sandbox-measure.sh's `add_measurement`, which runs after `install_bun` succeeds — log lines 1605-1618 show Bun installing successfully before the error.
157+
158+
4. **No other errors** in the log between GitLab CLI recording and the JSON error — confirming no other failure points.
159+
160+
## Files Collected
161+
162+
| File | Description |
163+
|------|-------------|
164+
| `ci-run-log.txt` | Full CI run log (1633 lines, 198KB) |
165+
| `issue-46-raw.json` | Raw GitHub issue JSON |
166+
| `pr-47-raw.json` | Raw PR #47 JSON |
167+
| `pr-47-conversation-comments.json` | PR conversation comments (empty) |
168+
| `pr-47-review-comments.json` | PR review comments (empty) |
169+
| `issue-46-comments.json` | Issue comments (empty) |

0 commit comments

Comments
 (0)