zkEVM Benchmarking Workload

This workspace contains code for executing Ethereum block validation logic within different Zero-Knowledge Virtual Machines (zkVMs) where the input is generated by the execution spec tests.

Goal

The primary goal is to measure and compare the performance (currently in cycle counts) of running a standardized benchmarking workload (including known worse-cases) with the same stateless validation logic* across various zkVM platforms.

*We cannot guarantee the logic will be the same for targets that are not supported by the Rust toolchain. An example of this currently is Cairo.

Workspace Structure

The workspace is organized into several key components:

• crates/metrics: Defines common data structures (BenchmarkRun<Metadata>) for storing and serializing benchmark results with generic metadata support.
• crates/witness-generator: A library that provides functionality for generating benchmark fixture files (BlockAndWitness: individual block + witness pairs) required for stateless block validation by processing standard Ethereum test fixtures or RPC endpoints.
• crates/witness-generator-cli: A standalone binary that uses the witness-generator library to generate fixture files. These are saved in the zkevm-fixtures-input folder. The crate includes Docker support for containerized deployment.
• crates/ere-hosts: A standalone binary that runs benchmarks across different zkVM platforms using pre-generated fixture files from zkevm-fixtures-input.
• crates/benchmark-runner: Provides a unified framework for running benchmarks across different zkVM implementations, including guest program input generation and execution orchestration.
• ere-guests/: Directory containing guest program implementations organized by program type, with each type containing implementations for different zkVM platforms. See the Guest Program Types section for detailed information about each type.
• zkevm-fixtures: (Git submodule) Contains the Ethereum execution layer test fixtures used by witness-generator-cli.
• zkevm-fixtures-input: Default directory where witness-generator-cli saves individual fixture files (.json) that are consumed by ere-hosts.
• zkevm-metrics: Directory where benchmark results (cycle counts) are stored by the host programs, organized by zkVM type.
• scripts: Contains helper scripts (e.g., fetching fixtures).
• xtask: Cargo xtask runner for automating tasks.

Workflow Overview

The benchmarking process is decoupled into two distinct phases:

1. Fixture Generation (witness-generator-cli): Processes Ethereum test fixtures (EEST) or RPC data to generate individual BlockAndWitness fixtures as JSON files saved in zkevm-fixtures-input/.
2. Benchmark Execution (ere-hosts): Reads from zkevm-fixtures-input/ and runs performance benchmarks across different zkVM platforms.

This decoupling provides several benefits:

• Independent fixture generation and benchmark execution
• Reuse of generated fixtures across multiple benchmark runs

Core Concepts

Each zkVM benchmark implementation follows a common pattern:

1. Guest Program:

   • Located within the ere-guests/ directory, organized by guest program type (e.g., ere-guests/stateless-validator/sp1/, ere-guests/empty-program/sp1/).
   • Each guest program type contains implementations for specific zkVMs.
   • For stateless validator programs: Contains the Rust code that executes the Ethereum state transition function (reth_stateless::validation::stateless_validation).
   • For empty programs: Contains minimal code to benchmark zkVM overhead.
   • This code is compiled specifically for the target zkVM's architecture (e.g., RISC-V for SP1, MIPS for zkMIPS).
   • It reads block/witness data from its zkVM environment's standard input.
   • Uses platform-specific mechanisms (often println! markers) to delineate code regions for cycle counting.

2. Host Program:

   • Located within crates/ere-hosts/ with shared host logic across zkVM platforms.
   • A standalone Rust binary that orchestrates the benchmarking for different guest program types.
   • Uses the benchmark-runner crate to generate guest program inputs from pre-generated fixture files.
   • Supports multiple guest program types via command-line arguments (e.g., stateless-validator, empty-program).
   • Invokes the corresponding zkVM SDK to execute the compiled Guest program ELF with the necessary inputs.
   • Collects cycle count metrics reported by the zkVM SDK.
   • Saves the results using the metrics crate with flexible metadata support into the appropriate subdirectory within zkevm-metrics/.

3. Automatic Patch Application:

   • The benchmark runner includes functionality to automatically apply precompile patches for each zkVM.

Prerequisites

1. Rust Toolchain: A standard Rust installation managed by rustup.
2. zkVM-Specific Toolchains: Each zkVM requires its own SDK and potentially a custom Rust toolchain/target. Please refer to the specific zkVM guest directory in ere-guests/ (e.g., ere-guests/stateless-validator/sp1/README.md, ere-guests/stateless-validator/openvm/README.md) for detailed setup instructions for that platform.
3. Git: Required for cloning the repository :)
4. Common Shell Utilities: The scripts in the ./scripts directory require a bash-compatible shell and standard utilities like curl, jq, and tar.

Setup

1. Clone the Repository:

   git clone <repository-url>
   cd zkevm-benchmark-workload

2. Fetch/Update Benchmark Fixtures:

   ./scripts/download-and-extract-fixtures.sh

3. Generate Benchmark Input Files:

   Navigate to crates/witness-generator-cli/ and generate fixture files:

   cd crates/witness-generator-cli
   cargo run --release -- tests --include Prague --include cold
   
   # Or generate from local EEST fixtures
   cargo run --release -- tests --eest-fixtures-path /path/to/local/eest/fixtures
   
   # Or generate from RPC
   cargo run --release -- rpc --last-n-blocks 2 --rpc-url <your-rpc-url>
   
   # Or listen for new blocks continuously
   cargo run --release -- rpc --follow --rpc-url <your-rpc-url>

   This creates individual .json files in the zkevm-fixtures-input/ directory that will be consumed by the benchmark runner.

4. (Optionally) Patching Precompiles: Each zkVM, for efficiency purposes requires particular dependencies to be patched. This repository contains an xtask that will automate this process by calling cargo <zkvm-name>. See .config/cargo.toml for how this is setup and precompile-patches for the patches that each zkVM requires.

Note: This is optional as it should be automatically done when the benchmarks are ran.

5. Run Benchmarks:

   Navigate to crates/ere-hosts/ and run benchmarks using the generated fixture files. You must specify which guest program type to benchmark:

   cd crates/ere-hosts
   
   # Run Ethereum stateless validator benchmarks
   cargo run --release --features sp1 -- stateless-validator
   cargo run --release --features "sp1,risc0" -- stateless-validator
   
   # Run empty program benchmarks (for measuring zkVM overhead)
   cargo run --release --features sp1 -- empty-program
   
   # Run block encoding length benchmarks (with RLP encoding format)
   cargo run --release --features sp1 -- block-encoding-length --loop-count 100 --format rlp
   
   # Run block encoding length benchmarks (with SSZ encoding format)
   cargo run --release --features sp1 -- block-encoding-length --loop-count 100 --format ssz
   
   # Use custom input folder for stateless validator benchmarks
   cargo run --release --features sp1 -- stateless-validator --input-folder my-fixtures

   See the respective README files in each crate for detailed usage instructions.

Guest Program Types

This repository supports multiple guest program types for comprehensive zkVM benchmarking across different computational workloads. Each guest program type is designed to measure specific aspects of zkVM performance:

Available Guest Program Types

1. stateless-validator - The primary benchmarking workload that executes Ethereum stateless block validation logic

   • Purpose: Measures zkVM performance on realistic Ethereum state transition computations
   • Input: Requires BlockAndWitness fixture files generated by witness-generator-cli
   • Computation: Uses reth_stateless::validation::stateless_validation to validate Ethereum blocks
   • zkVM Support: SP1, RISC0, OpenVM, Pico, Zisk
   • Usage: cargo run --features sp1 -- stateless-validator

2. empty-program - Minimal program for measuring zkVM overhead

   • Purpose: Measures the baseline overhead of zkVM execution without computational workload
   • Input: No input files required
   • Computation: Minimal operations to establish zkVM baseline performance
   • zkVM Support: SP1 (other zkVMs not yet implemented)
   • Usage: cargo run --features sp1 -- empty-program

3. block-encoding-length - Measures block encoding performance with support for both RLP and SSZ formats

   • Purpose: Benchmarks the performance of calculating encoded length of Ethereum blocks using different encoding formats
   • Input: Uses the same BlockAndWitness fixture files as stateless-validator
   • Computation: Repeatedly calls encoding length calculation functions on block headers and bodies
   • Encoding Formats: Supports both RLP (--format rlp) and SSZ (--format ssz) encoding
   • zkVM Support: SP1 (other zkVMs not yet implemented)
   • Usage: cargo run --features sp1 -- block-encoding-length --loop-count 100 --format rlp

Guest Program Architecture

Each guest program type follows a consistent structure across different zkVM platforms:

ere-guests/<program-type>/
├── sp1/          # SP1 implementation
│   ├── Cargo.toml
│   └── src/main.rs
├── risc0/        # RISC0 implementation (if available)
│   ├── Cargo.toml
│   └── src/main.rs
├── openvm/       # OpenVM implementation (if available)
├── pico/         # Pico implementation (if available)
└── zisk/         # Zisk implementation (if available)

Adding New Guest Program Types

The architecture is designed to be extensible. To add a new guest program type:

1. Create the guest program directory structure:

   ere-guests/your-new-program-type/
   ├── sp1/
   │   ├── Cargo.toml
   │   └── src/main.rs
   ├── risc0/
   │   ├── Cargo.toml
   │   └── src/main.rs
   └── ... (other zkVMs)
   

2. Add the new command to ere-hosts:

   • Add a new variant to the GuestProgramCommand enum in crates/ere-hosts/src/main.rs
   • Add corresponding logic to handle the new guest program type
   • Update the get_zkvm_instances function to point to the correct subdirectory

3. Update documentation to include the new guest program type in the supported benchmarks table and usage examples

CI/CD and Docker Support

The repository includes automated Docker image building through GitHub Actions:

• Automated Builds: Docker images are automatically built and pushed to GitHub Container Registry (ghcr.io) on pushes to the master branch
• Manual Builds: Docker builds can be triggered manually via GitHub Actions workflow dispatch
• Container Registry: Pre-built images are available at ghcr.io/<repository>/witness-generator-cli
• Multi-platform Support: Images are built for multiple architectures and cached for efficient builds

The Docker workflow (.github/workflows/docker-build.yml) provides:

• Automatic tagging with git SHA and latest for the default branch
• Docker layer caching for faster subsequent builds
• Integration with GitHub's container registry with proper authentication

The Docker image provides a self-contained environment for running the witness generator without local Rust toolchain setup.