Skip to content

Conversation

@spencer-tb
Copy link
Contributor

🗒️ Description

This PR is a WIP. It updates the benchmark CI workflow to add sanity check jobs that validate benchmark tests and the framework, before building artifacts. These checks will also make sure we never have issues with future benchmark test releases.

Key Changes

  • Renamed artifact_benchmark_fast.yaml to benchmark.yaml
  • Added three parallel sanity check jobs that run on PRs and pushes:
    a. Fixed Opcode Count Tests: fill --fixed-opcode-count 1 -m repricing
    b. Fixed Opcode Count Config Tests: Runs benchmark_parser to generate config, then fill --fixed-opcode-count -m repricing
    c. Gas Benchmark Tests: fill --gas-benchmark-values 1 --generate-pre-alloc-groups -m "benchmark and not state_test"
  • Artifact build job now depends on all sanity checks passing
  • All jobs use evmone for filling

Remaining Todos

  • Add execute remote tests with --gas-benchmark-values
  • Add execute remote tests with --fixed-opcode-count
  • Add all possible ways teams are using execute remote for benchmark tests
  • Remove any redundancies and optimize.

🔗 Related Issues or PRs

Requires #1790

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx tox -e static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

@spencer-tb spencer-tb added C-enhance Category: an improvement or new feature A-test-benchmark Area: Tests Benchmarks—Performance measurement (eg. `tests/benchmark/*`, `p/t/s/e/benchmark/*`) A-ci Area: Continuous Integration labels Dec 5, 2025
@codecov
Copy link

codecov bot commented Dec 5, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.31%. Comparing base (843178f) to head (7c9856c).

Additional details and impacted files
@@             Coverage Diff              @@
##           forks/osaka    #1853   +/-   ##
============================================
  Coverage        87.31%   87.31%           
============================================
  Files              541      541           
  Lines            32832    32832           
  Branches          3015     3015           
============================================
  Hits             28668    28668           
  Misses            3557     3557           
  Partials           607      607           
Flag Coverage Δ
unittests 87.31% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci Area: Continuous Integration A-test-benchmark Area: Tests Benchmarks—Performance measurement (eg. `tests/benchmark/*`, `p/t/s/e/benchmark/*`) C-enhance Category: an improvement or new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant