Skip to content

Commit 2700711

Browse files
LyonSsSgithub-actions[bot]
LyonSsS
and
github-actions[bot]
authored
Add CI enforced coverage improvement (#153)
* updated how autoamted baseline file for covearge is updated * chore: update coverage baseline to 96.88% Coverage improved from baseline. [automated update] * skip CI checks for automated commits * chore: retrigger CI checks * fail coverage job if test timeout after 15 seconds * updated coverage gating to remove autocommit from github bot * updated workflow to cover PR comments * fixed command for script calling --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 9cb29ba commit 2700711

File tree

5 files changed

+206
-110
lines changed

5 files changed

+206
-110
lines changed

‎.github/workflows/coverage.yml‎

Lines changed: 106 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,16 +28,117 @@ jobs:
2828
run: |
2929
# Fetch main branch
3030
git fetch origin main
31-
# Try to get baseline from main, if it doesn't exist use empty baseline
31+
# Get baseline from main (for comparison - must not decrease)
3232
git show origin/main:coverage-baseline.json > baseline-from-main.json 2>/dev/null || echo '{"lines":"0","functions":"0","branches":"0","statements":"0"}' > baseline-from-main.json
33-
echo "📊 Baseline from main:"
33+
echo "📊 Baseline from main branch:"
3434
cat baseline-from-main.json
3535
36+
- name: Get baseline from PR
37+
run: |
38+
# Get baseline from current PR branch (what developer committed)
39+
if [ -f coverage-baseline.json ]; then
40+
# Validate JSON format
41+
if jq empty coverage-baseline.json 2>/dev/null; then
42+
cp coverage-baseline.json baseline-from-pr.json
43+
echo "📊 Baseline from PR (committed by developer):"
44+
cat baseline-from-pr.json
45+
else
46+
echo "❌ ERROR: coverage-baseline.json is not valid JSON!"
47+
echo "Please run: npm run coverage:update-baseline"
48+
exit 1
49+
fi
50+
else
51+
echo "❌ ERROR: No coverage-baseline.json found in PR!"
52+
echo ""
53+
echo "You must run coverage locally and commit the baseline file."
54+
echo ""
55+
echo "📝 To fix: Run these commands locally and commit the result:"
56+
echo " npm run coverage"
57+
echo " npm run coverage:update-baseline"
58+
echo " git add coverage-baseline.json"
59+
echo " git commit -m 'chore: update coverage baseline'"
60+
echo ""
61+
exit 1
62+
fi
63+
3664
- name: Run coverage
37-
run: npm run coverage
65+
id: run_coverage
66+
timeout-minutes: 15
67+
run: |
68+
set -e # Exit immediately if coverage fails
69+
npm run coverage
70+
echo "✅ Coverage completed successfully"
71+
72+
- name: Validate coverage
73+
run: |
74+
echo "=================================================="
75+
echo "🔍 COVERAGE VALIDATION"
76+
echo "=================================================="
77+
78+
# Parse CI-generated coverage using dedicated script
79+
CI_LINES=$(npx ts-node --files scripts/get-coverage-percentage.ts)
80+
81+
# Get baselines
82+
PR_LINES=$(jq -r .lines baseline-from-pr.json)
83+
MAIN_LINES=$(jq -r .lines baseline-from-main.json)
84+
85+
echo ""
86+
echo "📊 Coverage Results:"
87+
echo " CI (actual): $CI_LINES%"
88+
echo " PR baseline: $PR_LINES%"
89+
echo " Main baseline: $MAIN_LINES%"
90+
echo ""
91+
92+
# Check 1: CI must match PR baseline (developer ran coverage correctly)
93+
echo "Check 1: Did developer run coverage locally?"
94+
if [ "$CI_LINES" = "$PR_LINES" ]; then
95+
echo " ✅ PASS - CI coverage matches PR baseline ($CI_LINES% == $PR_LINES%)"
96+
else
97+
echo " ❌ FAIL - CI coverage doesn't match PR baseline!"
98+
echo ""
99+
echo " Expected: $PR_LINES% (from your committed coverage-baseline.json)"
100+
echo " Actual: $CI_LINES% (from fresh CI coverage run)"
101+
echo ""
102+
echo "💡 This means either:"
103+
echo " 1. You forgot to run 'npm run coverage:update-baseline' locally"
104+
echo " 2. You modified coverage-baseline.json manually (cheating)"
105+
echo " 3. Your local coverage differs from CI (check .env setup)"
106+
echo ""
107+
echo "📝 To fix: Run these commands locally and commit the result:"
108+
echo " npm run coverage"
109+
echo " npm run coverage:update-baseline"
110+
echo " git add coverage-baseline.json"
111+
echo " git commit -m 'chore: update coverage baseline'"
112+
echo ""
113+
exit 1
114+
fi
115+
116+
echo ""
117+
118+
# Check 2: CI must be >= main baseline (coverage didn't decrease)
119+
echo "Check 2: Did coverage decrease?"
120+
if awk "BEGIN {exit !($CI_LINES >= $MAIN_LINES)}"; then
121+
if awk "BEGIN {exit !($CI_LINES > $MAIN_LINES)}"; then
122+
echo " ✅ PASS - Coverage improved! ($MAIN_LINES% → $CI_LINES%)"
123+
else
124+
echo " ✅ PASS - Coverage maintained ($CI_LINES%)"
125+
fi
126+
else
127+
echo " ❌ FAIL - Coverage decreased!"
128+
echo ""
129+
echo " Main baseline: $MAIN_LINES%"
130+
echo " Your PR: $CI_LINES%"
131+
echo " Decrease: $(awk "BEGIN {print $MAIN_LINES - $CI_LINES}")%"
132+
echo ""
133+
echo "💡 Please add tests to maintain or improve coverage."
134+
echo ""
135+
exit 1
136+
fi
38137
39-
- name: Check coverage against main baseline
40-
run: npm run coverage:check -- --baseline=baseline-from-main.json
138+
echo ""
139+
echo "=================================================="
140+
echo "✅ ALL CHECKS PASSED"
141+
echo "=================================================="
41142
42143
- name: Upload coverage report (optional)
43144
if: always()

‎.github/workflows/update-baseline.yml‎

Lines changed: 0 additions & 60 deletions
This file was deleted.

‎COVERAGE.md‎

Lines changed: 58 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -2,38 +2,38 @@
22

33
This project uses automated coverage checks to prevent test coverage from decreasing.
44

5-
## How It Works: Two-Phase Automated Protection
5+
## How It Works: Dual Validation
66

7-
This system uses two GitHub Actions workflows that work together to enforce coverage without manual maintenance:
7+
Developers run coverage locally and commit the baseline file. CI validates both that the developer ran coverage correctly AND that coverage didn't decrease.
88

9-
### Phase 1: PR Check (`.github/workflows/coverage.yml`)
9+
### Coverage Workflow (`.github/workflows/coverage.yml`)
1010
**Triggers:** Every pull request to main
1111

1212
**What it does:**
13-
1. Fetches `coverage-baseline.json` from **main branch** (not PR branch)
14-
2. Runs `npm run coverage` on PR code to generate fresh coverage
15-
3. Compares PR coverage against main's baseline
16-
4. **Result:**
17-
- ❌ Blocks merge if coverage drops below baseline
18-
- ✅ Passes if coverage maintained or improved
19-
- Shows detailed comparison in CI output
20-
21-
**Security:** Developers cannot cheat by modifying the baseline file in their PR because CI always fetches the baseline from the main branch.
22-
23-
### Phase 2: Auto-Update (`.github/workflows/update-baseline.yml`)
24-
**Triggers:** Every push to main (after PR merge)
25-
26-
**What it does:**
27-
1. Runs `npm run coverage` on the new main branch code
28-
2. Updates `coverage-baseline.json` with the new coverage values
29-
3. Commits the updated baseline automatically (only if coverage changed)
30-
4. Uses `github-actions[bot]` for the commit
31-
32-
**Result:** The baseline automatically tracks the current coverage on main, requiring zero manual maintenance.
13+
1. **Fetches baseline from main branch** - the current production baseline
14+
2. **Reads baseline from PR branch** - the baseline you committed
15+
3. **Runs coverage fresh in CI** - generates actual coverage from your code
16+
4. **Performs two validations:**
17+
18+
**Validation 1: Did you run coverage locally?**
19+
- ✅ **PASS** if `CI coverage === PR baseline` (you ran coverage correctly)
20+
- ❌ **FAIL** if `CI coverage !== PR baseline` (you forgot to run coverage or tampered with file)
21+
22+
**Validation 2: Did coverage decrease?**
23+
- ✅ **PASS** if `CI coverage >= main baseline` (coverage maintained or improved)
24+
- ❌ **FAIL** if `CI coverage < main baseline` (coverage decreased)
25+
26+
**Security Model:**
27+
- ✅ **Can't skip running coverage** - CI checks if your committed baseline matches actual coverage
28+
- ✅ **Can't decrease coverage** - CI checks if your coverage is below main's baseline
29+
- ✅ **Can't cheat** - CI regenerates coverage fresh and validates against both baselines
30+
- ✅ **Can't commit invalid baseline** - CI validates JSON format before processing
31+
- ✅ **Can't skip baseline file** - CI fails immediately if baseline file is missing
32+
- ✅ **Visible in PR** - Baseline changes are visible in the PR diff
3333

3434
---
3535

36-
**Combined Effect:** Coverage can only go up or stay the same, never down! This creates a "ratchet effect" where quality continuously improves. ✅
36+
**Effect:** Coverage can only go up or stay the same, never down! This creates a "ratchet effect" where quality continuously improves. ✅
3737

3838
## Commands
3939

@@ -52,25 +52,40 @@ npm run coverage:update-baseline
5252

5353
### For Developers
5454

55-
When working on a PR:
56-
1. Run `npm run coverage` to generate coverage report
57-
2. Run `npm run coverage:check` to verify you maintained coverage
58-
3. If check fails, add more tests until it passes
55+
**IMPORTANT:** You must run coverage locally and commit the baseline file with your PR.
56+
57+
**Step-by-step:**
58+
1. Make your code changes
59+
2. Run coverage locally:
60+
```bash
61+
npm run coverage
62+
```
63+
3. Update the baseline file:
64+
```bash
65+
npm run coverage:update-baseline
66+
```
67+
4. Commit the baseline file:
68+
```bash
69+
git add coverage-baseline.json
70+
git commit -m "chore: update coverage baseline"
71+
```
72+
5. Push your PR
73+
74+
**What CI validates:**
75+
- ✅ **Check 1:** Your committed baseline matches CI coverage (proves you ran coverage)
76+
- ✅ **Check 2:** Your coverage is >= main's baseline (proves coverage didn't drop)
77+
78+
**If CI fails:**
79+
- **"No coverage-baseline.json found in PR"** → You forgot to commit the baseline file. Run steps 2-4 above and push.
80+
- **"coverage-baseline.json is not valid JSON"** → The baseline file is corrupted. Run `npm run coverage:update-baseline` and commit.
81+
- **"CI coverage doesn't match PR baseline"** → You forgot to update the baseline. Run steps 2-3 above and push.
82+
- **"Coverage decreased"** → Add more tests to maintain or improve coverage.
5983

6084
### For Maintainers
6185

62-
**No manual work needed!** Phase 2 automatically:
63-
- Runs coverage after each merge to main
64-
- Updates `coverage-baseline.json`
65-
- Commits the new baseline
86+
**No special maintenance needed!** Developers commit their own baseline files.
6687

67-
You can manually update baseline if needed:
68-
```bash
69-
npm run coverage:update-baseline
70-
git add coverage-baseline.json
71-
git commit -m "chore: update coverage baseline"
72-
git push
73-
```
88+
The workflow only validates - it doesn't modify anything. When a PR merges, the updated baseline goes to main automatically.
7489

7590
## Current Coverage
7691

@@ -86,10 +101,12 @@ Current baseline (as of initial setup):
86101
- Uses Hardhat's built-in coverage tool (generates `coverage/lcov.info`)
87102
- Parses LCOV format to extract: lines, functions, branches, statements
88103
- Stores baseline in `coverage-baseline.json` at repository root
89-
- Script: `scripts/check-coverage.ts`
104+
- Scripts:
105+
- `scripts/check-coverage.ts` - Local validation (compares coverage against baseline)
106+
- `scripts/get-coverage-percentage.ts` - Extracts coverage percentage from lcov.info (used by CI)
90107

91108
### Environment Setup for CI
92-
Both workflows copy `.env.example` to `.env` to enable fork tests with public RPC endpoints during coverage runs.
109+
The workflow copies `.env.example` to `.env` to enable fork tests with public RPC endpoints during coverage runs.
93110

94111
### Branch Protection
95112
To enforce coverage checks, enable branch protection on main:

‎coverage-baseline.json‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
2-
"lines": "96.03",
3-
"functions": "98.27",
4-
"branches": "86.19",
5-
"statements": "96.03"
2+
"lines": "96.88",
3+
"functions": "98.57",
4+
"branches": "87.76",
5+
"statements": "96.88"
66
}

‎scripts/get-coverage-percentage.ts‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/usr/bin/env ts-node
2+
3+
import fs from "fs";
4+
import path from "path";
5+
6+
/**
7+
* Extracts line coverage percentage from lcov.info
8+
* Outputs just the percentage number (e.g., "96.03")
9+
*/
10+
11+
const lcovPath = path.join(__dirname, "..", "coverage", "lcov.info");
12+
13+
// Check if file exists
14+
if (!fs.existsSync(lcovPath)) {
15+
console.error("Error: coverage/lcov.info not found");
16+
process.exit(1);
17+
}
18+
19+
// Read and parse lcov file
20+
const content = fs.readFileSync(lcovPath, "utf8");
21+
const lines = content.split("\n");
22+
23+
let linesFound = 0;
24+
let linesHit = 0;
25+
26+
for (const line of lines) {
27+
if (line.startsWith("LF:")) {
28+
linesFound += parseInt(line.substring(3), 10);
29+
} else if (line.startsWith("LH:")) {
30+
linesHit += parseInt(line.substring(3), 10);
31+
}
32+
}
33+
34+
// Calculate percentage
35+
const percentage = linesFound > 0 ? (linesHit / linesFound * 100).toFixed(2) : "0";
36+
37+
// Output only the percentage (no extra text)
38+
console.log(percentage);

0 commit comments

Comments
 (0)