Skip to content

Commit b091ef7

Browse files
fselmofselmo
authored
feat(tests): Initial implementation for generating client test vectors (#76)
* feat: first pass at filling test vectors * Add more _info to test vectors * Add proper fork support * fix: updates to parametrize fork for tests * chore: move framework tests into appropriate parent * fix: fork system hierarchy * fix: hex serialization issues for test vectors * refactor: Refactor fillers; design fork_choice_test; remove genesis tests - Remove genesis tests for now since we don't have the need yet - Refactor fillers and add a ``fork_choice_test`` type * refactor: some fork choice logic to help advance chain with blocks - forkchoice was not advancing chain with blocks, made some minor changes as compared to consensus-specs that may not be correct but help show how to write forkchoice tests for vector generation. * fix: cleanups from comments on PR #76 * Add missing STF test post checks + confirm state validation works * chore: Update CLAUDE.md; make more succinct to reduce context * chore: Add official Python 3.14 support to lean-spec-tests * refactor: rename package + generalize framework + use consensus scope - Rename the package to ``lean-ethereum-testing`` - Rename `uv` workspace package to ``packages/testing`` - Refactor the package into a generalized framework component and ``consensus_testing`` module that focuses on testing the consensus protocol. This opens up the possibility of adding execution-specific testing in the future. - Use `tests/consensus/{fork}` as consensus tests that generate vectors. - Uses a ``--layer`` flag for ``fill`` that defaults to ``consensus`` scope we don't even have to think about this separation for now. * chore: Update CLAUDE.md based on last commit's refactor * fix: reconcile differences with main after rebase + cleanup impl - Reconcile differences with ``main`` after rebase. - Clean up the implementation of the test fixtures + remove clunky builder for now. * chore: add simple fill-devnet-tests run to CI for sanity checks * fix: updates from comments on #76 - Remove comment that snuck into spec docstring - Remove ``Union`` for ``BlockSpec | Block`` for fixtures. Instead, use ``@field_serializer`` to serialize the filled block as the ``Block`` in the fixture. - Add a TODO to configure appropriate default values for validator number and pubkeys for fixtures.
1 parent 9930041 commit b091ef7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+2891
-141
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,23 @@ jobs:
5757

5858
- name: Run tests via tox
5959
run: uvx tox -e pytest
60+
61+
fill-tests:
62+
name: Fill test fixtures - Python 3.14
63+
runs-on: ubuntu-latest
64+
steps:
65+
- name: Checkout leanSpec
66+
uses: actions/checkout@v4
67+
68+
- name: Install uv and Python 3.14
69+
uses: astral-sh/setup-uv@v4
70+
with:
71+
enable-cache: true
72+
cache-dependency-glob: "pyproject.toml"
73+
python-version: "3.14"
74+
75+
- name: Sync dependencies
76+
run: uv sync --all-packages --no-progress
77+
78+
- name: Fill test fixtures
79+
run: uv run fill --fork=Devnet --clean

‎CLAUDE.md‎

Lines changed: 76 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -17,89 +17,96 @@ subspecifications that the Lean Ethereum protocol relies on.
1717

1818
### Running Tests
1919
```bash
20-
# Install and sync project and dev dependencies
21-
uv sync
22-
23-
# Run all tests
24-
uv run pytest
25-
26-
# Run tests with coverage
27-
uv run pytest --cov=src/lean_spec --cov-report=html
20+
uv sync --all-packages # Install dependencies
21+
uv run pytest # Run unit tests
22+
uv run fill --fork=devnet --clean # Generate test vectors
23+
# Note: execution layer support is planned for future, infrastructure is ready
24+
# for now, `--layer=consensus` is default and the only value used.
2825
```
2926

30-
### Code Quality Checks
27+
### Code Quality
3128
```bash
32-
# Format code
33-
uv run ruff format src tests
34-
35-
# Check linting
36-
uv run ruff check src tests
37-
38-
# Fix fixable linting errors
39-
uv run ruff check --fix src tests
40-
41-
# Type checking
42-
uv run mypy src tests
43-
44-
# Run all quality checks (lint, typecheck, spellcheck)
45-
uvx tox -e all-checks
46-
47-
# Run everything (all checks + tests + docs)
48-
uvx tox
29+
uv run ruff format src tests # Format code
30+
uv run ruff check --fix src tests packages # Lint and fix
31+
uvx tox -e typecheck # Type check
32+
uvx tox -e all-checks # All quality checks
33+
uvx tox # Everything (checks + tests + docs)
4934
```
5035

5136
### Common Tasks
37+
- **Main specs**: `src/lean_spec/`
38+
- **Subspecs**: `src/lean_spec/subspecs/{subspec}/`
39+
- **Unit tests**: `tests/lean_spec/` (mirrors source structure)
40+
- **Consensus spec tests**: `tests/consensus/` (generates test vectors)
41+
- **Execution spec tests**: `tests/execution/` (future - infrastructure ready)
5242

53-
1. **Adding to main specs**: Located in `src/lean_spec/`
54-
2. **Adding to subspecs**: Located in `src/lean_spec/subspecs/`
55-
- Create a new subdirectory for each subspec (e.g., `src/lean_spec/subspecs/poseidon2/`)
56-
- Tests for subspecs should be in `tests/subspecs/{subspec}/`, mirroring the source structure
43+
## Code Style
44+
- Line length: 100 characters, type hints everywhere
45+
- Google docstring style (no docstrings for `__init__`)
46+
- Test files/functions must start with `test_`
5747

58-
## Important Patterns
48+
## Test Framework Structure
5949

60-
### Test Patterns
61-
- Tests should be placed in `tests/` and follow the same structure as the source code.
62-
- Use `pytest.fixture`, in `conftest.py` or test files, for reusable test setup.
63-
- Use `pytest.mark.parametrize` to parametrize tests with multiple inputs
64-
- Use `pytest.raises(...)` with specific exceptions to test error cases
65-
- Use `@pytest.mark.slow` for long-running tests
50+
**Two types of tests:**
6651

67-
## Code Style
52+
1. **Unit tests** (`tests/lean_spec/`) - Standard pytest tests for implementation
53+
2. **Spec tests** (`tests/consensus/`) - Generate JSON test vectors via fillers
54+
- *Note: `tests/execution/` infrastructure is ready for future execution layer work*
55+
56+
**Test Filling Framework:**
57+
- Layer-agnostic pytest plugin in `packages/testing/src/framework/pytest_plugins/filler.py`
58+
- Layer-specific packages: `consensus_testing` (active) and `execution_testing` (future)
59+
- Write consensus spec tests using `state_transition_test` or `fork_choice_test` fixtures
60+
- These fixtures are type aliases that create test vectors when called
61+
- Run `uv run fill --fork=Devnet --clean` to generate consensus fixtures
62+
- Use `--layer=execution` flag when execution layer is implemented
63+
- Output goes to `fixtures/{layer}/{format}/{test_path}/...`
64+
65+
**Example spec test:**
66+
```python
67+
def test_block(state_transition_test: StateTransitionTestFiller) -> None:
68+
state_transition_test(
69+
pre=genesis_state,
70+
blocks=[block],
71+
post=StateExpectation(slot=Slot(1)) # Only check what matters
72+
)
73+
```
6874

69-
- Line length: 79 characters
70-
- Use type hints everywhere
71-
- Follow Google docstring style
72-
- No docstrings needed for `__init__` methods
73-
- Imports are automatically sorted by `isort` and `ruff`
74-
75-
## Testing Philosophy
76-
77-
- Tests should be simple and clear
78-
- Test file names must start with `test_`
79-
- Test function names must start with `test_`
80-
- Use descriptive test names that explain what's being tested
81-
82-
## Common Commands Reference
83-
84-
| Task | Command |
85-
|-----------------------------------------------|----------------------------------|
86-
| Install and sync project and dev dependencies | `uv sync` |
87-
| Run tests | `uv run pytest` |
88-
| Format code | `uv run ruff format src tests` |
89-
| Lint code | `uv run ruff check src tests` |
90-
| Fix lint errors | `uv run ruff check --fix src tests` |
91-
| Type check | `uv run mypy src tests` |
92-
| Build docs | `uv run mkdocs build` |
93-
| Serve docs | `uv run mkdocs serve` |
94-
| Run all quality checks (no tests/docs) | `uvx tox -e all-checks` |
95-
| Run everything (checks + tests + docs) | `uvx tox` |
75+
**How it works:**
76+
1. Test function receives a fixture class (not instance) as parameter
77+
2. Calling it creates a `FixtureWrapper` that runs `make_fixture()`
78+
3. `make_fixture()` executes the spec code (state transitions, fork choice steps)
79+
4. Validates output against expectations (`StateExpectation`, `StoreChecks`)
80+
5. Serializes to JSON via Pydantic's `model_dump(mode="json")`
81+
6. Writes fixtures at session end to `fixtures/{layer}/{format}/{test_path}/...`
82+
83+
**Layer-specific architecture:**
84+
- `framework/` - Shared infrastructure (base classes, pytest plugin, CLI)
85+
- `consensus_testing/` - Consensus layer fixtures, forks, builders
86+
- `execution_testing/` - Execution layer fixtures, forks, builders
87+
- Regular pytest runs (`uv run pytest`) ignore spec tests - they only run via `fill` command
88+
89+
**Serialization requirements:**
90+
- All spec types (State, Block, Uint64, etc.) must be Pydantic models
91+
- Custom types need `@field_serializer` or `model_serializer` for JSON output
92+
- SSZ types typically serialize to hex strings (e.g., `"0x1234..."`)
93+
- Fixture models inherit from layer-specific base classes:
94+
- Consensus: `BaseConsensusFixture` (in `consensus_testing/test_fixtures/base.py`)
95+
- Execution: `BaseExecutionFixture` (in `execution_testing/test_fixtures/base.py`)
96+
- Both use `CamelModel` for camelCase JSON output
97+
- Test the serialization: `fixture.model_dump(mode="json")` must produce valid JSON
98+
99+
**Key fixture types:**
100+
- `StateTransitionTest` - Tests state transitions with blocks
101+
- `ForkChoiceTest` - Tests fork choice with steps (tick/block/attestation)
102+
- Selective validation via `StateExpectation` and `StoreChecks` (only validates fields you specify)
96103

97104
## Important Notes
98105

99-
1. This repository uses Python 3.12+ features
100-
2. All models should use Pydantic for automatic validation.
101-
3. Keep things simple, readable, and clear. These are meant to be clear specifications.
102-
4. The repository is `leanSpec` not `lean-spec`.
106+
- Python 3.12+ required
107+
- Use Pydantic models for validation
108+
- Keep specs simple, readable, and clear
109+
- Repository is `leanSpec` not `lean-spec`
103110

104111
## SSZ Type Design Patterns
105112

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,10 @@
33
## Quick Start
44

55
1. Fork and clone the repository
6-
2. Install dependencies: `uv sync`
6+
2. Install dependencies: `uv sync --all-packages`
77
3. Make your changes
88
4. Run checks: `uvx tox -e all-checks`
9-
5. Run tests: `uv run pytest`
9+
5. Run tests: `uvx pytest`
1010
6. Submit a pull request
1111

1212
## Pull Request Guidelines

‎README.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -177,17 +177,16 @@ def test_withdrawal_amount_above_uint64_max():
177177

178178
| Task | Command |
179179
|-----------------------------------------------|------------------------------------|
180-
| Install and sync project and dev dependencies | `uv sync` |
181-
| Run tests | `uv run pytest` |
180+
| Install and sync project and dev dependencies | `uv sync --all-packages` |
181+
| Run tests | `uv run pytest ...` |
182182
| Format code | `uv run ruff format src tests` |
183183
| Lint code | `uv run ruff check src tests` |
184184
| Fix lint errors | `uv run ruff check --fix src tests` |
185185
| Type check | `uv run mypy src tests` |
186186
| Build docs | `uv run mkdocs build` |
187187
| Serve docs | `uv run mkdocs serve` |
188-
| Run all quality checks (no tests/docs) | `uvx tox -e all-checks` |
189188
| Run everything (checks + tests + docs) | `uvx tox` |
190-
| Run specific tox environment | `uvx tox -e lint` |
189+
| Run all quality checks (no tests/docs) | `uvx tox -e all-checks` |
191190

192191

193192
## Contributing

‎packages/testing/README.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Lean Ethereum Specification Testing Framework
2+
3+
Testing framework for generating and running Lean Ethereum specification tests.
4+
5+
This package provides tools for generating consensus test fixtures, including:
6+
- Pytest plugins for fixture generation
7+
- Base fixture types and serialization
8+
- CLI tools for test management
9+
10+
## Installation
11+
12+
This package is part of the lean-spec workspace and is automatically installed when you
13+
sync the parent project with `--all-packages`.
14+
15+
```bash
16+
# from `leanSpec/` (root of workspace)
17+
uv sync --all-packages
18+
```
19+
20+
## Usage
21+
22+
Generate test fixtures using the `fill` command:
23+
24+
```bash
25+
# from `leanSpec/` (root of workspace)
26+
uv run fill --clean --fork=devnet
27+
```
28+
29+
See the main project documentation for more details.

‎packages/testing/pyproject.toml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
[build-system]
2+
requires = ["setuptools>=77.0.3", "wheel"]
3+
build-backend = "setuptools.build_meta"
4+
5+
[project]
6+
name = "lean-ethereum-testing"
7+
version = "0.0.1"
8+
description = "Lean Ethereum client test generation and runner framework"
9+
readme = "README.md"
10+
authors = [
11+
{ name = "Ethereum Foundation", email = "thomas.coratger@ethereum.org" },
12+
]
13+
keywords = ["ethereum", "testing", "consensus", "lean"]
14+
classifiers = [
15+
"License :: OSI Approved :: MIT License",
16+
"Programming Language :: Python :: 3",
17+
"Programming Language :: Python :: 3.12",
18+
"Programming Language :: Python :: 3.13",
19+
"Programming Language :: Python :: 3.14",
20+
]
21+
requires-python = ">=3.12"
22+
dependencies = [
23+
"lean-spec",
24+
"pydantic>=2.12.0,<3",
25+
"pytest>=8.3.3,<9",
26+
"click>=8.1.0,<9",
27+
]
28+
29+
license = {text = "MIT"}
30+
31+
[project.optional-dependencies]
32+
test = ["pytest-cov>=6.0.0,<7"]
33+
lint = ["ruff>=0.11.8,<1", "mypy>=1.15.0,<1.16"]
34+
35+
[project.urls]
36+
Homepage = "https://github.com/leanEthereum/lean-spec"
37+
Source = "https://github.com/leanEthereum/lean-spec"
38+
Issues = "https://github.com/leanEthereum/lean-spec/issues"
39+
40+
[project.scripts]
41+
fill = "framework.cli.fill:fill"
42+
43+
[tool.setuptools.packages.find]
44+
where = ["src"]
45+
46+
[tool.uv.sources]
47+
lean-spec = { workspace = true }

‎packages/testing/src/consensus_testing/__init__.py‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
"""Test tools for generating and consuming leanSpec consensus test vectors."""
2+
3+
from typing import Type
4+
5+
from framework.base_types import CamelModel
6+
7+
from . import forks
8+
from .block_spec import BlockSpec
9+
from .genesis import generate_pre_state
10+
from .test_fixtures import (
11+
BaseConsensusFixture,
12+
ForkChoiceTest,
13+
StateTransitionTest,
14+
)
15+
from .test_types import (
16+
AttestationStep,
17+
BaseForkChoiceStep,
18+
BlockStep,
19+
ForkChoiceStep,
20+
StateExpectation,
21+
StoreChecks,
22+
TickStep,
23+
)
24+
25+
StateTransitionTestFiller = Type[StateTransitionTest]
26+
ForkChoiceTestFiller = Type[ForkChoiceTest]
27+
28+
__all__ = [
29+
# Public API
30+
"BlockSpec",
31+
"forks",
32+
"generate_pre_state",
33+
# Base types
34+
"CamelModel",
35+
# Fixture classes
36+
"BaseConsensusFixture",
37+
"StateTransitionTest",
38+
"ForkChoiceTest",
39+
# Test types
40+
"BaseForkChoiceStep",
41+
"TickStep",
42+
"BlockStep",
43+
"AttestationStep",
44+
"ForkChoiceStep",
45+
"StateExpectation",
46+
"StoreChecks",
47+
# Type aliases for test function signatures
48+
"StateTransitionTestFiller",
49+
"ForkChoiceTestFiller",
50+
]

0 commit comments

Comments
 (0)