|
| 1 | +# Case Study: Broken Image Links in README.md (Issue #15) |
| 2 | + |
| 3 | +## Summary |
| 4 | + |
| 5 | +Benchmark execution resulted in broken image links in README.md. The images `bench_rust.png` and `bench_rust_log_scale.png` were not present in the `Docs/` folder, causing broken image links on GitHub. |
| 6 | + |
| 7 | +## Timeline of Events |
| 8 | + |
| 9 | +### Phase 1: Initial State (Before December 27, 2025) |
| 10 | +- Benchmark images existed in the `Docs/` folder |
| 11 | +- README.md referenced images at `Docs/bench_rust.png` and `Docs/bench_rust_log_scale.png` |
| 12 | +- Images were being generated and committed by CI on push to main |
| 13 | + |
| 14 | +### Phase 2: The Problematic Commit (December 27, 2025, 12:15 UTC) |
| 15 | +- **Commit**: `b8573e9` - "Apply best practices from Comparisons.Neo4jVSDoublets" |
| 16 | +- **Issue**: This commit **deleted** the existing benchmark images and replaced them with only a `.gitkeep` file |
| 17 | +- **Git diff shows**: |
| 18 | + ``` |
| 19 | + diff --git a/Docs/.gitkeep b/Docs/.gitkeep |
| 20 | + new file mode 100644 |
| 21 | + diff --git a/Docs/bench_rust.png b/Docs/bench_rust.png |
| 22 | + deleted file mode 100644 |
| 23 | + diff --git a/Docs/bench_rust_log_scale.png b/Docs/bench_rust_log_scale.png |
| 24 | + deleted file mode 100644 |
| 25 | + ``` |
| 26 | + |
| 27 | +### Phase 3: Subsequent Commits (December 27, 2025) |
| 28 | +- `3f61b63` - Initial commit with task details |
| 29 | +- `1a6550b` - Revert "Initial commit with task details" |
| 30 | +- `e2eeed2` - Fix CodeFactor style issues in Python output scripts |
| 31 | +- `e99c5cd` - Make benchmark parameters configurable via environment variables |
| 32 | +- `04547d8` - Merge pull request #14 (current main) |
| 33 | + |
| 34 | +### Phase 4: CI Runs After the Deletion |
| 35 | +- All subsequent CI runs showed warnings: "No files were found with the provided path: rust/bench_rust.png rust/bench_rust_log_scale.png" |
| 36 | +- This indicates the `out.py` script was either: |
| 37 | + 1. Not running at all (due to condition `github.event_name == 'push'` not being met for PRs), OR |
| 38 | + 2. Running but failing silently, OR |
| 39 | + 3. Running successfully but images not being committed |
| 40 | + |
| 41 | +## Root Cause Analysis |
| 42 | + |
| 43 | +### Primary Root Cause |
| 44 | +**The commit `b8573e9` deleted the benchmark images without the CI having regenerated them first.** |
| 45 | + |
| 46 | +When refactoring to "apply best practices from Comparisons.Neo4jVSDoublets", the commit: |
| 47 | +1. Updated the workflow to generate and commit images on push to main |
| 48 | +2. But also deleted the existing images from the `Docs/` folder |
| 49 | +3. Created only a `.gitkeep` file in their place |
| 50 | + |
| 51 | +### Secondary Root Cause |
| 52 | +**The CI workflow only generates images on push to main/master, not on PR merges.** |
| 53 | + |
| 54 | +Looking at the workflow condition: |
| 55 | +```yaml |
| 56 | +- name: Prepare benchmark results |
| 57 | + if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/master') |
| 58 | +``` |
| 59 | +
|
| 60 | +This means: |
| 61 | +- PR checks: Images are NOT generated (condition not met) |
| 62 | +- Push to main: Images ARE supposed to be generated |
| 63 | +
|
| 64 | +### Tertiary Root Cause |
| 65 | +**The CI annotation shows "No files were found" even for push-to-main runs.** |
| 66 | +
|
| 67 | +CI run #15057894427 (push to main on May 16, 2025) shows: |
| 68 | +> "No files were found with the provided path: rust/bench_rust.png rust/bench_rust_log_scale.png. No artifacts will be uploaded." |
| 69 | +
|
| 70 | +This suggests the `out.py` script is running but the benchmark output file (`out.txt`) may be empty or malformed when the `Prepare benchmark results` step runs. |
| 71 | + |
| 72 | +## Evidence |
| 73 | + |
| 74 | +### 1. Git History |
| 75 | +``` |
| 76 | +568275a - Has images in Docs/ |
| 77 | +a67bf15 - Has images in Docs/ |
| 78 | +b8573e9 - Only .gitkeep in Docs/ (IMAGES DELETED HERE) |
| 79 | +``` |
| 80 | +
|
| 81 | +### 2. CI Run Annotations |
| 82 | +From run #15057894427: |
| 83 | +``` |
| 84 | +! No files were found with the provided path: rust/bench_rust.png |
| 85 | +rust/bench_rust_log_scale.png. No artifacts will be uploaded. |
| 86 | +``` |
| 87 | +
|
| 88 | +### 3. Local Testing |
| 89 | +The `out.py` script works correctly when given valid `out.txt` data: |
| 90 | +- Parses 16 PSQL entries |
| 91 | +- Parses 32 Doublets entries |
| 92 | +- Generates both PNG images successfully |
| 93 | +
|
| 94 | +### 4. Screenshot Evidence |
| 95 | +The issue screenshot shows broken image placeholders in README.md where benchmark charts should appear. |
| 96 | +
|
| 97 | +## Proposed Solutions |
| 98 | +
|
| 99 | +### Solution 1: Restore Images from Git History (Immediate Fix) |
| 100 | +```bash |
| 101 | +git checkout 568275a -- Docs/bench_rust.png Docs/bench_rust_log_scale.png |
| 102 | +git add Docs/ |
| 103 | +git commit -m "Restore benchmark images from before deletion" |
| 104 | +``` |
| 105 | + |
| 106 | +### Solution 2: Fix CI Workflow (Long-term Fix) |
| 107 | +1. Ensure the `Prepare benchmark results` step runs correctly |
| 108 | +2. Add error handling to detect when images aren't generated |
| 109 | +3. Consider keeping images committed instead of regenerating each time |
| 110 | + |
| 111 | +### Solution 3: Update README to Use Relative Paths (Alternative) |
| 112 | +If images are stored in a different location, update README.md references accordingly. |
| 113 | + |
| 114 | +## Verification Steps |
| 115 | + |
| 116 | +1. Restore images from git history |
| 117 | +2. Verify images appear correctly in README.md on GitHub |
| 118 | +3. Trigger a new push to main and verify CI workflow regenerates images |
| 119 | +4. Confirm no "No files were found" warnings in CI logs |
| 120 | + |
| 121 | +## Lessons Learned |
| 122 | + |
| 123 | +1. **Don't delete files until replacement mechanism is verified**: The commit deleted images assuming CI would regenerate them, but CI conditions prevented regeneration. |
| 124 | + |
| 125 | +2. **Test workflow changes end-to-end**: The workflow refactoring should have been tested on a push to main to ensure images are actually generated. |
| 126 | + |
| 127 | +3. **Add CI checks for required artifacts**: The workflow should fail (not just warn) if expected output files are missing. |
| 128 | + |
| 129 | +## Related Resources |
| 130 | + |
| 131 | +- Reference implementation: https://github.com/linksplatform/Comparisons.Neo4jVSDoublets |
| 132 | +- Issue: https://github.com/linksplatform/Comparisons.PostgreSQLVSDoublets/issues/15 |
| 133 | +- Problematic commit: b8573e91a0651496397c44c4fd3be14ae1b79bf8 |
0 commit comments