Skip to content

Commit 694f2b5

Browse files
committed
fix: add min-threshold to baseline comparison CI test
The baseline comparison test was failing intermittently because: - Fast policies (~200µs) showed large percentage swings (72%) - Absolute differences were small (~170µs) - normal CI variance - 50% threshold alone couldn't account for this Adding --min-threshold 500µs ensures regressions only fail when BOTH: 1. Percentage exceeds threshold (50%), AND 2. Absolute time exceeds min-threshold (500µs) This is exactly the scenario min-threshold was designed to handle. Signed-off-by: Sertac Ozercan <[email protected]>
1 parent 40cd80d commit 694f2b5

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

.github/workflows/test-gator.yaml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -123,12 +123,14 @@ jobs:
123123
--iterations 30 \
124124
--save /tmp/baseline.json
125125
126-
# Compare against baseline (should pass with 50% threshold)
126+
# Compare against baseline with min-threshold to handle CI variance
127+
# Fast policies (~200µs) can show large % swings from small absolute changes
127128
./bin/gator bench \
128129
--filename test/gator/bench/basic/ \
129130
--iterations 30 \
130131
--compare /tmp/baseline.json \
131-
--threshold 50
132+
--threshold 50 \
133+
--min-threshold 500µs
132134
133135
- name: Test min-threshold
134136
run: |

0 commit comments

Comments
 (0)