Skip to content

Commit 55fd016

Browse files
yug490xNeshi
yug49
and
0xNeshi
authored
feat(benches): add benchmarks for historic and latest events scanning (#254)
<!-- Append the issue number --> Related to #229 Hey! This PR introduces a benchmarking system for Event Scanner using **Criterion.rs** to measure performance impact of changes to the scanner. Currently drafting, Bencher CI integration coming in follow-up. --- ### What's Included #### New benches Crate Structure ``` benches/ ├── Cargo.toml # Benchmark crate config ├── src/ │ └── lib.rs # Shared utilities (Anvil setup, contract deployment, event generation) └── benches/ ├── historic_scanning.rs # Historic mode benchmarks └── latest_events_scanning.rs # Latest events mode benchmarks ``` #### Benchmarks Implemented | Mode | Event Counts | What It Measures | |------|--------------|------------------| | **Historic** | 10K, 50K, 100K | Time to scan all events from block 0 to latest | | **Latest Events** | 100, 1K, 10K, 50K | Time to fetch the N most recent events from a 100K event pool | example of Historic: <img width="1186" height="623" alt="Screenshot 2025-12-13 at 2 58 25 PM" src="https://github.com/user-attachments/assets/3fdcb55b-65bf-423a-8e02-d8005c943b51" /> --- ### How Regression Testing Works Criterion stores baseline results in `target/criterion/<benchmark>/base/`. On subsequent runs, it compares new measurements against this baseline and reports: ``` historic_scanning/events/10000 time: [30.963 ms 36.506 ms 40.598 ms] thrpt: [246.32 Kelem/s 273.93 Kelem/s 322.97 Kelem/s] change: [-2.12% -1.01% +0.12%] (p = 0.12 > 0.05) No change in performance detected. ``` If a change introduces a regression, you'll see something like: ``` change: [+15.2% +18.4% +21.1%] (p = 0.00 < 0.05) Performance has regressed. ``` This makes it easy to catch slowdowns before merging --- ### Running Benchmarks ```bash # All benchmarks cargo bench --manifest-path benches/Cargo.toml # Specific benchmark cargo bench --manifest-path benches/Cargo.toml --bench historic_scanning cargo bench --manifest-path benches/Cargo.toml --bench latest_events_scanning # Filter by event count cargo bench --manifest-path benches/Cargo.toml -- "historic_scanning/events/10000" ``` --- ### Next Steps - **Bencher CI integration**: Add GitHub Actions workflow for on-demand benchmarking with historical tracking via [Bencher](https://bencher.dev/) --------- Signed-off-by: yug49 <148035793+yug49@users.noreply.github.com> Co-authored-by: Nenad <xinef.it@gmail.com>
1 parent 9c8eff3 commit 55fd016

File tree

11 files changed

+1369
-20
lines changed

11 files changed

+1369
-20
lines changed

‎.github/workflows/benchmarks.yml‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
name: Benchmarks
2+
3+
# Runs benchmarks to track performance and detect regressions.
4+
# - On push to main: automatically updates baseline in Bencher
5+
# - On manual dispatch: run specific benchmarks on-demand
6+
7+
on:
8+
push:
9+
branches: [main]
10+
paths:
11+
# Only run benchmarks if relevant code changes
12+
- "src/**"
13+
- "benches/**"
14+
- "Cargo.toml"
15+
- "Cargo.lock"
16+
workflow_dispatch:
17+
inputs:
18+
benchmark:
19+
description: "Benchmark to run"
20+
required: true
21+
default: "all"
22+
type: choice
23+
options:
24+
- all
25+
- historic_scanning
26+
- latest_events_scanning
27+
28+
permissions:
29+
contents: read
30+
checks: write
31+
32+
env:
33+
CARGO_TERM_COLOR: always
34+
35+
jobs:
36+
# Run historic benchmarks
37+
benchmark_historic:
38+
name: Run Historic Benchmarks
39+
runs-on: ubuntu-latest
40+
# Run on push to main, or when manually selecting 'all' or 'historic_scanning'
41+
if: ${{ github.event_name == 'push' || inputs.benchmark == 'all' || inputs.benchmark == 'historic_scanning' }}
42+
43+
steps:
44+
- name: Harden the runner (Audit all outbound calls)
45+
uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0
46+
with:
47+
egress-policy: audit
48+
49+
- name: Checkout repository
50+
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
51+
52+
- name: Install Rust toolchain
53+
uses: actions-rust-lang/setup-rust-toolchain@1780873c7b576612439a134613cc4cc74ce5538c # v1.15.2
54+
55+
- name: Install Foundry (Anvil)
56+
uses: foundry-rs/foundry-toolchain@8b0419c685ef46cb79ec93fbdc131174afceb730 # v1.6.0
57+
58+
- name: Install Bencher CLI
59+
uses: bencherdev/bencher@2f1532643adc0e69e52acaec936d227ff14da24f # v0.5.9
60+
61+
- name: Decompress benchmark state dumps
62+
run: |
63+
echo "Decompressing Anvil state dumps..."
64+
gunzip -k benches/dumps/*.json.gz
65+
ls -lh benches/dumps/
66+
67+
- name: Run historic benchmarks and track with Bencher
68+
env:
69+
BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
70+
BENCHER_PROJECT: ${{ vars.BENCHER_PROJECT }}
71+
run: |
72+
bencher run \
73+
--project "$BENCHER_PROJECT" \
74+
--token "$BENCHER_API_TOKEN" \
75+
--branch main \
76+
--testbed ubuntu-latest \
77+
--threshold-measure latency \
78+
--threshold-test t_test \
79+
--threshold-max-sample-size 64 \
80+
--threshold-upper-boundary 0.99 \
81+
--thresholds-reset \
82+
--adapter rust_criterion \
83+
--github-actions "${{ secrets.GITHUB_TOKEN }}" \
84+
"cargo bench --manifest-path benches/Cargo.toml --bench historic_scanning"
85+
86+
# Run latest events benchmarks
87+
benchmark_latest:
88+
name: Run Latest Events Benchmarks
89+
runs-on: ubuntu-latest
90+
# Run on push to main, or when manually selecting 'all' or 'latest_events_scanning'
91+
if: ${{ github.event_name == 'push' || inputs.benchmark == 'all' || inputs.benchmark == 'latest_events_scanning' }}
92+
93+
steps:
94+
- name: Harden the runner (Audit all outbound calls)
95+
uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0
96+
with:
97+
egress-policy: audit
98+
99+
- name: Checkout repository
100+
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
101+
102+
- name: Install Rust toolchain
103+
uses: actions-rust-lang/setup-rust-toolchain@1780873c7b576612439a134613cc4cc74ce5538c # v1.15.2
104+
105+
- name: Install Foundry (Anvil)
106+
uses: foundry-rs/foundry-toolchain@8b0419c685ef46cb79ec93fbdc131174afceb730 # v1.6.0
107+
108+
- name: Install Bencher CLI
109+
uses: bencherdev/bencher@2f1532643adc0e69e52acaec936d227ff14da24f # v0.5.9
110+
111+
- name: Decompress benchmark state dumps
112+
run: |
113+
echo "Decompressing Anvil state dumps..."
114+
gunzip -k benches/dumps/*.json.gz
115+
ls -lh benches/dumps/
116+
117+
- name: Run latest events benchmarks and track with Bencher
118+
env:
119+
BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
120+
BENCHER_PROJECT: ${{ vars.BENCHER_PROJECT }}
121+
run: |
122+
bencher run \
123+
--project "$BENCHER_PROJECT" \
124+
--token "$BENCHER_API_TOKEN" \
125+
--branch main \
126+
--testbed ubuntu-latest \
127+
--threshold-measure latency \
128+
--threshold-test t_test \
129+
--threshold-max-sample-size 64 \
130+
--threshold-upper-boundary 0.99 \
131+
--thresholds-reset \
132+
--adapter rust_criterion \
133+
--github-actions "${{ secrets.GITHUB_TOKEN }}" \
134+
"cargo bench --manifest-path benches/Cargo.toml --bench latest_events_scanning"

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
11
/target
22
/examples/**/target
33
.DS_Store
4+
5+
# Benchmark dumps - only commit compressed files
6+
benches/dumps/*.json
7+
!benches/dumps/*.metadata.json

0 commit comments

Comments
 (0)