Skip to content

Commit 13e7ae4

Browse files
konardclaude
andcommitted
Restore benchmark images and add case study documentation
The benchmark images (bench_rust.png and bench_rust_log_scale.png) were accidentally deleted in commit b8573e9 when applying best practices from Comparisons.Neo4jVSDoublets. This commit restores them from git history. Also adds a detailed case study analysis documenting: - Timeline of events leading to the bug - Root cause analysis - Evidence from CI logs and git history - Proposed solutions and lessons learned Fixes #15 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 850e64c commit 13e7ae4

File tree

8 files changed

+1181
-0
lines changed

8 files changed

+1181
-0
lines changed

Docs/.gitkeep

Whitespace-only changes.

Docs/bench_rust.png

44.1 KB
Loading

Docs/bench_rust_log_scale.png

44.3 KB
Loading
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# Case Study: Broken Image Links in README.md (Issue #15)
2+
3+
## Summary
4+
5+
Benchmark execution resulted in broken image links in README.md. The images `bench_rust.png` and `bench_rust_log_scale.png` were not present in the `Docs/` folder, causing broken image links on GitHub.
6+
7+
## Timeline of Events
8+
9+
### Phase 1: Initial State (Before December 27, 2025)
10+
- Benchmark images existed in the `Docs/` folder
11+
- README.md referenced images at `Docs/bench_rust.png` and `Docs/bench_rust_log_scale.png`
12+
- Images were being generated and committed by CI on push to main
13+
14+
### Phase 2: The Problematic Commit (December 27, 2025, 12:15 UTC)
15+
- **Commit**: `b8573e9` - "Apply best practices from Comparisons.Neo4jVSDoublets"
16+
- **Issue**: This commit **deleted** the existing benchmark images and replaced them with only a `.gitkeep` file
17+
- **Git diff shows**:
18+
```
19+
diff --git a/Docs/.gitkeep b/Docs/.gitkeep
20+
new file mode 100644
21+
diff --git a/Docs/bench_rust.png b/Docs/bench_rust.png
22+
deleted file mode 100644
23+
diff --git a/Docs/bench_rust_log_scale.png b/Docs/bench_rust_log_scale.png
24+
deleted file mode 100644
25+
```
26+
27+
### Phase 3: Subsequent Commits (December 27, 2025)
28+
- `3f61b63` - Initial commit with task details
29+
- `1a6550b` - Revert "Initial commit with task details"
30+
- `e2eeed2` - Fix CodeFactor style issues in Python output scripts
31+
- `e99c5cd` - Make benchmark parameters configurable via environment variables
32+
- `04547d8` - Merge pull request #14 (current main)
33+
34+
### Phase 4: CI Runs After the Deletion
35+
- All subsequent CI runs showed warnings: "No files were found with the provided path: rust/bench_rust.png rust/bench_rust_log_scale.png"
36+
- This indicates the `out.py` script was either:
37+
1. Not running at all (due to condition `github.event_name == 'push'` not being met for PRs), OR
38+
2. Running but failing silently, OR
39+
3. Running successfully but images not being committed
40+
41+
## Root Cause Analysis
42+
43+
### Primary Root Cause
44+
**The commit `b8573e9` deleted the benchmark images without the CI having regenerated them first.**
45+
46+
When refactoring to "apply best practices from Comparisons.Neo4jVSDoublets", the commit:
47+
1. Updated the workflow to generate and commit images on push to main
48+
2. But also deleted the existing images from the `Docs/` folder
49+
3. Created only a `.gitkeep` file in their place
50+
51+
### Secondary Root Cause
52+
**The CI workflow only generates images on push to main/master, not on PR merges.**
53+
54+
Looking at the workflow condition:
55+
```yaml
56+
- name: Prepare benchmark results
57+
if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/master')
58+
```
59+
60+
This means:
61+
- PR checks: Images are NOT generated (condition not met)
62+
- Push to main: Images ARE supposed to be generated
63+
64+
### Tertiary Root Cause
65+
**The CI annotation shows "No files were found" even for push-to-main runs.**
66+
67+
CI run #15057894427 (push to main on May 16, 2025) shows:
68+
> "No files were found with the provided path: rust/bench_rust.png rust/bench_rust_log_scale.png. No artifacts will be uploaded."
69+
70+
This suggests the `out.py` script is running but the benchmark output file (`out.txt`) may be empty or malformed when the `Prepare benchmark results` step runs.
71+
72+
## Evidence
73+
74+
### 1. Git History
75+
```
76+
568275a - Has images in Docs/
77+
a67bf15 - Has images in Docs/
78+
b8573e9 - Only .gitkeep in Docs/ (IMAGES DELETED HERE)
79+
```
80+
81+
### 2. CI Run Annotations
82+
From run #15057894427:
83+
```
84+
! No files were found with the provided path: rust/bench_rust.png
85+
rust/bench_rust_log_scale.png. No artifacts will be uploaded.
86+
```
87+
88+
### 3. Local Testing
89+
The `out.py` script works correctly when given valid `out.txt` data:
90+
- Parses 16 PSQL entries
91+
- Parses 32 Doublets entries
92+
- Generates both PNG images successfully
93+
94+
### 4. Screenshot Evidence
95+
The issue screenshot shows broken image placeholders in README.md where benchmark charts should appear.
96+
97+
## Proposed Solutions
98+
99+
### Solution 1: Restore Images from Git History (Immediate Fix)
100+
```bash
101+
git checkout 568275a -- Docs/bench_rust.png Docs/bench_rust_log_scale.png
102+
git add Docs/
103+
git commit -m "Restore benchmark images from before deletion"
104+
```
105+
106+
### Solution 2: Fix CI Workflow (Long-term Fix)
107+
1. Ensure the `Prepare benchmark results` step runs correctly
108+
2. Add error handling to detect when images aren't generated
109+
3. Consider keeping images committed instead of regenerating each time
110+
111+
### Solution 3: Update README to Use Relative Paths (Alternative)
112+
If images are stored in a different location, update README.md references accordingly.
113+
114+
## Verification Steps
115+
116+
1. Restore images from git history
117+
2. Verify images appear correctly in README.md on GitHub
118+
3. Trigger a new push to main and verify CI workflow regenerates images
119+
4. Confirm no "No files were found" warnings in CI logs
120+
121+
## Lessons Learned
122+
123+
1. **Don't delete files until replacement mechanism is verified**: The commit deleted images assuming CI would regenerate them, but CI conditions prevented regeneration.
124+
125+
2. **Test workflow changes end-to-end**: The workflow refactoring should have been tested on a push to main to ensure images are actually generated.
126+
127+
3. **Add CI checks for required artifacts**: The workflow should fail (not just warn) if expected output files are missing.
128+
129+
## Related Resources
130+
131+
- Reference implementation: https://github.com/linksplatform/Comparisons.Neo4jVSDoublets
132+
- Issue: https://github.com/linksplatform/Comparisons.PostgreSQLVSDoublets/issues/15
133+
- Problematic commit: b8573e91a0651496397c44c4fd3be14ae1b79bf8
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
failed to get run log: HTTP 410: Server Error (https://api.github.com/repos/linksplatform/Comparisons.PostgreSQLVSDoublets/actions/runs/15057894427/logs)

0 commit comments

Comments
 (0)