Skip to content

Commit 2e9a2a5

Browse files
authored
feat: enable all validators and center SVG chart (#61)
* fix: create PR instead of direct push, test with tjs+schemasafe only * fix: add ajv to test validators (required for README generation) * fix: use PAT_TOKEN for PR creation to trigger required checks * fix: use PAT for git push to trigger PR workflows * fix: use peter-evans/create-pull-request to trigger PR workflows * feat: enable all validators and center SVG chart * fix: remove 'returning early' text from SVG * fix: use head-to-head comparison in SVG chart Update update-svg.ts to use the same comparison logic as the PR comment: only tests where BOTH validators pass are compared (apples-to-apples). This ensures the SVG chart shows consistent performance ratios with the PR comment benchmarks. * fix: use adjusted ops/sec in SVG based on head-to-head ratios Calculate each validator's "adjusted" ops/sec from the head-to-head ratio comparison. This ensures apples-to-apples comparison where each bar represents performance on only the tests that both validators pass. - tjs baseline is taken from the comparison with most shared tests - Other validators' ops are calculated as: tjs_baseline_ops / ratio - Maintains familiar ops/sec format while being truly comparable
1 parent a82255f commit 2e9a2a5

File tree

3 files changed

+166
-206
lines changed

3 files changed

+166
-206
lines changed

.github/workflows/update-benchmarks.yml

Lines changed: 21 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -5,38 +5,16 @@ on:
55
branches: [main]
66
workflow_dispatch:
77

8-
# Runs full benchmarks on main and commits updated results
8+
# Runs full benchmarks on main and opens a PR with updated results
99

1010
permissions:
1111
contents: write
12+
pull-requests: write
1213

1314
jobs:
14-
# Check if this is a benchmark commit to prevent infinite loop
15-
check-skip:
16-
runs-on: ubuntu-latest
17-
outputs:
18-
should_skip: ${{ steps.check.outputs.should_skip }}
19-
steps:
20-
- uses: actions/checkout@v4
21-
with:
22-
fetch-depth: 1
23-
- name: Check if benchmark commit
24-
id: check
25-
run: |
26-
COMMIT_MSG=$(git log -1 --format=%s)
27-
if [[ "$COMMIT_MSG" == "chore: update benchmark results" ]]; then
28-
echo "should_skip=true" >> $GITHUB_OUTPUT
29-
echo "Skipping: benchmark commit detected"
30-
else
31-
echo "should_skip=false" >> $GITHUB_OUTPUT
32-
echo "Running: not a benchmark commit"
33-
fi
34-
3515
# Run benchmarks for all validators (except jsonschema)
3616
benchmark:
3717
runs-on: ubuntu-latest
38-
needs: check-skip
39-
if: needs.check-skip.outputs.should_skip != 'true'
4018
strategy:
4119
matrix:
4220
draft: [draft4, draft6, draft7, draft2019-09, draft2020-12]
@@ -71,17 +49,15 @@ jobs:
7149
name: benchmark-${{ matrix.validator }}-${{ matrix.draft }}
7250
path: benchmarks/results/${{ matrix.validator }}-${{ matrix.draft }}.json
7351

74-
# Merge results, update files, and commit
52+
# Merge results, update files, and open PR
7553
finalize:
7654
runs-on: ubuntu-latest
77-
needs: [check-skip, benchmark]
78-
if: needs.check-skip.outputs.should_skip != 'true'
55+
needs: [benchmark]
7956

8057
steps:
8158
- uses: actions/checkout@v4
8259
with:
8360
submodules: true
84-
token: ${{ secrets.PAT_TOKEN }}
8561

8662
- name: Use Node.js 22
8763
uses: actions/setup-node@v4
@@ -123,35 +99,28 @@ jobs:
12399
# Generate BENCHMARKS.md
124100
npx tsx benchmarks/generate-summary.ts
125101
126-
# Update README.md
127-
npx tsx benchmarks/update-readme.ts
102+
# Update README.md (requires ajv.json)
103+
if [ -f benchmarks/results/ajv.json ]; then
104+
npx tsx benchmarks/update-readme.ts
105+
else
106+
echo "Skipping README update (ajv.json not found)"
107+
fi
128108
129109
# Update SVG chart
130110
npx tsx benchmarks/update-svg.ts
131111
132-
# Generate detailed markdown reports for each validator
133-
for v in ajv zod joi is-my-json-valid z-schema djv jsen schemasafe; do
112+
# Generate detailed markdown reports for each validator that has results
113+
for v in $(ls benchmarks/results/*.json 2>/dev/null | xargs -n1 basename | sed 's/.json//' | grep -v tjs); do
134114
UPPER_NAME=$(echo "$v" | tr '[:lower:]-' '[:upper:]_')
135115
npx tsx benchmarks/generate-report.ts "$v" -o "benchmarks/results/BENCHMARK_${UPPER_NAME}.md"
136116
done
137117
138-
- name: Check for changes
139-
id: check_changes
140-
run: |
141-
git add -A
142-
if git diff --staged --quiet; then
143-
echo "has_changes=false" >> $GITHUB_OUTPUT
144-
echo "No changes detected"
145-
else
146-
echo "has_changes=true" >> $GITHUB_OUTPUT
147-
echo "Changes detected:"
148-
git diff --staged --stat
149-
fi
150-
151-
- name: Commit benchmark updates
152-
if: steps.check_changes.outputs.has_changes == 'true'
153-
run: |
154-
git config user.name "github-actions[bot]"
155-
git config user.email "github-actions[bot]@users.noreply.github.com"
156-
git commit -m "chore: update benchmark results"
157-
git push
118+
- name: Create PR with benchmark updates
119+
uses: peter-evans/create-pull-request@v7
120+
with:
121+
token: ${{ secrets.PAT_TOKEN }}
122+
commit-message: "chore: update benchmark results"
123+
title: "chore: update benchmark results"
124+
body: "Automated benchmark update from main branch."
125+
branch: chore/update-benchmarks
126+
delete-branch: true

assets/benchmark.svg

Lines changed: 22 additions & 100 deletions
Loading

0 commit comments

Comments
 (0)