feat(tests): Initial implementation for generating client test vectors #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

tcoratger merged 18 commits into leanEthereum:main from fselmo:feat/test-vectors

Oct 28, 2025

.github/workflows/ci.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -57,3 +57,23 @@ jobs: @@
           - name: Run tests via tox
             run: uvx tox -e pytest
+      fill-tests:
+        name: Fill test fixtures - Python 3.14
+        runs-on: ubuntu-latest
+        steps:
+          - name: Checkout leanSpec
+            uses: actions/checkout@v4
+          - name: Install uv and Python 3.14
+            uses: astral-sh/setup-uv@v4
+            with:
+              enable-cache: true
+              cache-dependency-glob: "pyproject.toml"
+              python-version: "3.14"
+          - name: Sync dependencies
+            run: uv sync --all-packages --no-progress
+          - name: Fill test fixtures
+            run: uv run fill --fork=Devnet --clean

CLAUDE.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -17,89 +17,96 @@ subspecifications that the Lean Ethereum protocol relies on.
  
    ### Running Tests

    ```bash

    # Install and sync project and dev dependencies

    uv sync

    # Run all tests

    uv run pytest

    # Run tests with coverage

    uv run pytest --cov=src/lean_spec --cov-report=html

    uv sync --all-packages            # Install dependencies

    uv run pytest                     # Run unit tests

    uv run fill --fork=devnet --clean # Generate test vectors

    # Note: execution layer support is planned for future, infrastructure is ready

    # for now, `--layer=consensus` is default and the only value used.

    ```

    ### Code Quality Checks

    ### Code Quality

    ```bash

    # Format code

    uv run ruff format src tests

    # Check linting

    uv run ruff check src tests

    # Fix fixable linting errors

    uv run ruff check --fix src tests

    # Type checking

    uv run mypy src tests

    # Run all quality checks (lint, typecheck, spellcheck)

    uvx tox -e all-checks

    # Run everything (all checks + tests + docs)

    uvx tox

    uv run ruff format src tests               # Format code

    uv run ruff check --fix src tests packages # Lint and fix

    uvx tox -e typecheck                       # Type check

    uvx tox -e all-checks                      # All quality checks

    uvx tox                                    # Everything (checks + tests + docs)

    ```

    ### Common Tasks

    - **Main specs**: `src/lean_spec/`

    - **Subspecs**: `src/lean_spec/subspecs/{subspec}/`

    - **Unit tests**: `tests/lean_spec/` (mirrors source structure)

    - **Consensus spec tests**: `tests/consensus/` (generates test vectors)

    - **Execution spec tests**: `tests/execution/` (future - infrastructure ready)

    1. **Adding to main specs**: Located in `src/lean_spec/`

    2. **Adding to subspecs**: Located in `src/lean_spec/subspecs/`

       - Create a new subdirectory for each subspec (e.g., `src/lean_spec/subspecs/poseidon2/`)

       - Tests for subspecs should be in `tests/subspecs/{subspec}/`, mirroring the source structure

    ## Code Style

    - Line length: 100 characters, type hints everywhere

    - Google docstring style (no docstrings for `__init__`)

    - Test files/functions must start with `test_`

    ## Important Patterns

    ## Test Framework Structure

    ### Test Patterns

    - Tests should be placed in `tests/` and follow the same structure as the source code.

    - Use `pytest.fixture`, in `conftest.py` or test files, for reusable test setup.

    - Use `pytest.mark.parametrize` to parametrize tests with multiple inputs

    - Use `pytest.raises(...)` with specific exceptions to test error cases

    - Use `@pytest.mark.slow` for long-running tests

    **Two types of tests:**

    ## Code Style

    1. **Unit tests** (`tests/lean_spec/`) - Standard pytest tests for implementation

    2. **Spec tests** (`tests/consensus/`) - Generate JSON test vectors via fillers

       - *Note: `tests/execution/` infrastructure is ready for future execution layer work*

    **Test Filling Framework:**

    - Layer-agnostic pytest plugin in `packages/testing/src/framework/pytest_plugins/filler.py`

    - Layer-specific packages: `consensus_testing` (active) and `execution_testing` (future)

    - Write consensus spec tests using `state_transition_test` or `fork_choice_test` fixtures

    - These fixtures are type aliases that create test vectors when called

    - Run `uv run fill --fork=Devnet --clean` to generate consensus fixtures

    - Use `--layer=execution` flag when execution layer is implemented

    - Output goes to `fixtures/{layer}/{format}/{test_path}/...`

    **Example spec test:**

    ```python

    def test_block(state_transition_test: StateTransitionTestFiller) -> None:

        state_transition_test(

            pre=genesis_state,

            blocks=[block],

            post=StateExpectation(slot=Slot(1))  # Only check what matters

        )

    ```

    - Line length: 79 characters

    - Use type hints everywhere

    - Follow Google docstring style

    - No docstrings needed for `__init__` methods

    - Imports are automatically sorted by `isort` and `ruff`

    ## Testing Philosophy

    - Tests should be simple and clear

    - Test file names must start with `test_`

    - Test function names must start with `test_`

    - Use descriptive test names that explain what's being tested

    ## Common Commands Reference

    | Task                                          | Command                          |

    |-----------------------------------------------|----------------------------------|

    | Install and sync project and dev dependencies | `uv sync`                        |

    | Run tests                                     | `uv run pytest`                  |

    | Format code                                   | `uv run ruff format src tests`   |

    | Lint code                                     | `uv run ruff check src tests`    |

    | Fix lint errors                               | `uv run ruff check --fix src tests` |

    | Type check                                    | `uv run mypy src tests`          |

    | Build docs                                    | `uv run mkdocs build`            |

    | Serve docs                                    | `uv run mkdocs serve`            |

    | Run all quality checks (no tests/docs)        | `uvx tox -e all-checks`          |

    | Run everything (checks + tests + docs)        | `uvx tox`                        |

    **How it works:**

    1. Test function receives a fixture class (not instance) as parameter

    2. Calling it creates a `FixtureWrapper` that runs `make_fixture()`

    3. `make_fixture()` executes the spec code (state transitions, fork choice steps)

    4. Validates output against expectations (`StateExpectation`, `StoreChecks`)

    5. Serializes to JSON via Pydantic's `model_dump(mode="json")`

    6. Writes fixtures at session end to `fixtures/{layer}/{format}/{test_path}/...`

    **Layer-specific architecture:**

    - `framework/` - Shared infrastructure (base classes, pytest plugin, CLI)

    - `consensus_testing/` - Consensus layer fixtures, forks, builders

    - `execution_testing/` - Execution layer fixtures, forks, builders

    - Regular pytest runs (`uv run pytest`) ignore spec tests - they only run via `fill` command

    **Serialization requirements:**

    - All spec types (State, Block, Uint64, etc.) must be Pydantic models

    - Custom types need `@field_serializer` or `model_serializer` for JSON output

    - SSZ types typically serialize to hex strings (e.g., `"0x1234..."`)

    - Fixture models inherit from layer-specific base classes:

      - Consensus: `BaseConsensusFixture` (in `consensus_testing/test_fixtures/base.py`)

      - Execution: `BaseExecutionFixture` (in `execution_testing/test_fixtures/base.py`)

      - Both use `CamelModel` for camelCase JSON output

    - Test the serialization: `fixture.model_dump(mode="json")` must produce valid JSON

    **Key fixture types:**

    - `StateTransitionTest` - Tests state transitions with blocks

    - `ForkChoiceTest` - Tests fork choice with steps (tick/block/attestation)

    - Selective validation via `StateExpectation` and `StoreChecks` (only validates fields you specify)

    ## Important Notes

    1. This repository uses Python 3.12+ features

    2. All models should use Pydantic for automatic validation.

    3. Keep things simple, readable, and clear. These are meant to be clear specifications.

    4. The repository is `leanSpec` not `lean-spec`.

    - Python 3.12+ required

    - Use Pydantic models for validation

    - Keep specs simple, readable, and clear

    - Repository is `leanSpec` not `lean-spec`

    ## SSZ Type Design Patterns

CONTRIBUTING.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -3,10 +3,10 @@ @@
     ## Quick Start
 . Fork and clone the repository
-. Install dependencies: `uv sync`
+. Install dependencies: `uv sync --all-packages`
 . Make your changes
 . Run checks: `uvx tox -e all-checks`
-. Run tests: `uv run pytest`
+. Run tests: `uvx pytest`
 . Submit a pull request
     ## Pull Request Guidelines
@@ Expand Down @@

README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -177,17 +177,16 @@ def test_withdrawal_amount_above_uint64_max():
  
    | Task                                          | Command                            |

    |-----------------------------------------------|------------------------------------|

    | Install and sync project and dev dependencies | `uv sync`                          |

    | Run tests                                     | `uv run pytest`                    |

    | Install and sync project and dev dependencies | `uv sync --all-packages`           |

    | Run tests                                     | `uv run pytest ...`                |

    | Format code                                   | `uv run ruff format src tests`     |

    | Lint code                                     | `uv run ruff check src tests`      |

    | Fix lint errors                               | `uv run ruff check --fix src tests` |

    | Type check                                    | `uv run mypy src tests`            |

    | Build docs                                    | `uv run mkdocs build`              |

    | Serve docs                                    | `uv run mkdocs serve`              |

    | Run all quality checks (no tests/docs)        | `uvx tox -e all-checks`            |

    | Run everything (checks + tests + docs)        | `uvx tox`                          |

    | Run specific tox environment                  | `uvx tox -e lint`                  |

    | Run all quality checks (no tests/docs)        | `uvx tox -e all-checks`            |

    ## Contributing

packages/testing/README.md

-Original file line number
+Diff line change
@@ -0,0 +1,29 @@
+    # Lean Ethereum Specification Testing Framework
+    Testing framework for generating and running Lean Ethereum specification tests.
+    This package provides tools for generating consensus test fixtures, including:
+    - Pytest plugins for fixture generation
+    - Base fixture types and serialization
+    - CLI tools for test management
+    ## Installation
+    This package is part of the lean-spec workspace and is automatically installed when you
+    sync the parent project with `--all-packages`.
+    ```bash
+    # from `leanSpec/` (root of workspace)
+    uv sync --all-packages
+    ```
+    ## Usage
+    Generate test fixtures using the `fill` command:
+    ```bash
+    # from `leanSpec/` (root of workspace)
+    uv run fill --clean --fork=devnet
+    ```
+    See the main project documentation for more details.

packages/testing/pyproject.toml

-Original file line number
+Diff line change
@@ -0,0 +1,47 @@
+    [build-system]
+    requires = ["setuptools>=77.0.3", "wheel"]
+    build-backend = "setuptools.build_meta"
+    [project]
+    name = "lean-ethereum-testing"
+    version = "0.0.1"
+    description = "Lean Ethereum client test generation and runner framework"
+    readme = "README.md"
+    authors = [
+        { name = "Ethereum Foundation", email = "thomas.coratger@ethereum.org" },
+    ]
+    keywords = ["ethereum", "testing", "consensus", "lean"]
+    classifiers = [
+        "License :: OSI Approved :: MIT License",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.12",
+        "Programming Language :: Python :: 3.13",
+        "Programming Language :: Python :: 3.14",
+    ]
+    requires-python = ">=3.12"
+    dependencies = [
+        "lean-spec",
+        "pydantic>=2.12.0,<3",
+        "pytest>=8.3.3,<9",
+        "click>=8.1.0,<9",
+    ]
+    license = {text = "MIT"}
+    [project.optional-dependencies]
+    test = ["pytest-cov>=6.0.0,<7"]
+    lint = ["ruff>=0.11.8,<1", "mypy>=1.15.0,<1.16"]
+    [project.urls]
+    Homepage = "https://github.com/leanEthereum/lean-spec"
+    Source = "https://github.com/leanEthereum/lean-spec"
+    Issues = "https://github.com/leanEthereum/lean-spec/issues"
+    [project.scripts]
+    fill = "framework.cli.fill:fill"
+    [tool.setuptools.packages.find]
+    where = ["src"]
+    [tool.uv.sources]
+    lean-spec = { workspace = true }

packages/testing/src/consensus_testing/__init__.py

-Original file line number
+Diff line change
@@ -0,0 +1,50 @@
+    """Test tools for generating and consuming leanSpec consensus test vectors."""
+    from typing import Type
+    from framework.base_types import CamelModel
+    from . import forks
+    from .block_spec import BlockSpec
+    from .genesis import generate_pre_state
+    from .test_fixtures import (
+        BaseConsensusFixture,
+        ForkChoiceTest,
+        StateTransitionTest,
+    )
+    from .test_types import (
+        AttestationStep,
+        BaseForkChoiceStep,
+        BlockStep,
+        ForkChoiceStep,
+        StateExpectation,
+        StoreChecks,
+        TickStep,
+    )
+    StateTransitionTestFiller = Type[StateTransitionTest]
+    ForkChoiceTestFiller = Type[ForkChoiceTest]
+    __all__ = [
+        # Public API
+        "BlockSpec",
+        "forks",
+        "generate_pre_state",
+        # Base types
+        "CamelModel",
+        # Fixture classes
+        "BaseConsensusFixture",
+        "StateTransitionTest",
+        "ForkChoiceTest",
+        # Test types
+        "BaseForkChoiceStep",
+        "TickStep",
+        "BlockStep",
+        "AttestationStep",
+        "ForkChoiceStep",
+        "StateExpectation",
+        "StoreChecks",
+        # Type aliases for test function signatures
+        "StateTransitionTestFiller",
+        "ForkChoiceTestFiller",
+    ]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tests): Initial implementation for generating client test vectors #76

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!