Skip to content

Commit dfdd433

Browse files
authored
feat(fill): add --generate-all-formats to enable generation of BlockhainEngineXFixture with other formats (#1855)
* feat(fill): add --generate-all-formats flag for comprehensive fixture generation Add new --generate-all-formats command line option that enables generation of all fixture formats including BlockchainEngineXFixture in a single command execution. Key changes: - Add --generate-all-formats CLI option with two-phase execution support. - Phase 1: Generate pre-allocation groups (same as --generate-pre-alloc-groups). - Phase 2: Generate ALL supported fixture formats (vs only BlockchainEngineXFixture). - Update FixtureOutput class to handle new generate_all_formats parameter. - Preserve --generate-all-formats flag in phase 2 execution for proper format selection. - Maintain backward compatibility with existing --generate-pre-alloc-groups workflow. This makes BlockchainEngineXFixture a first-class citizen alongside other fixture formats, enabling comprehensive test coverage generation in a single command while leveraging the performance optimizations of pre-allocation groups. * docs(fill): update docs for `--generate-all-formats` * tests(fill): add unit tests for `--generate-all-formats` * feat(fill): auto-enable `--generate-all-formats` if tarball output * test(fill): add tests for enabling `--generate-all-formats` for tar.gz * docs(fill): update docs for auto-enable of `--generate-all-formats` if .tar.gz * docs: update changelog * refactor(fill): extract helper methods in FillCommand - Extract _should_use_two_phase_execution() method to simplify complex conditional logic. - Extract _ensure_generate_all_formats_for_tarball() method for cleaner argument processing. - Update create_executions() to use the new helper methods. * refactor(fill): improve FixtureOutput auto-enable logic - Add should_auto_enable_all_formats property to clarify auto-enable logic. - Improve variable naming: should_generate_all_formats instead of generate_all_formats. - Fix duplication: use existing is_tarball property logic instead of creating duplicate method. - Update tests to use new field name. * refactor(fill): simplify fixture format selection logic with proper phase enum - Add ExecutionPhase enum with 4 distinct phases: NORMAL, PHASE_1_PREALLOC, PHASE_2_ENGINE_X_ONLY, PHASE_2_ALL_FORMATS. - Extract _determine_execution_phase() method to encapsulate phase logic. - Extract _is_blockchain_engine_x_fixture() helper for format checking. - Extract _determine_fixture_formats() method to handle all 4 execution phases. - Simplify pytest_generate_tests() from 40+ lines of nested conditionals to 2 clean lines.
1 parent bfe359d commit dfdd433

File tree

9 files changed

+478
-33
lines changed

9 files changed

+478
-33
lines changed

docs/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ Users can select any of the artifacts depending on their testing needs for their
6262
- ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository ([#1590](https://github.com/ethereum/execution-spec-tests/pull/1590)).
6363
- ✨ Add a new pytest plugin, `ported_tests`, that lists the static fillers and PRs from `ported_from` markers for use in the coverage Github Workflow ([#1634](https://github.com/ethereum/execution-spec-tests/pull/1634)).
6464
- ✨ Enable two-phase filling of fixtures with pre-allocation groups and add a `BlockchainEngineXFixture` format ([#1706](https://github.com/ethereum/execution-spec-tests/pull/1706), [#1760](https://github.com/ethereum/execution-spec-tests/pull/1760)).
65+
- ✨ Add `--generate-all-formats` flag to enable generation of all fixture formats including `BlockchainEngineXFixture` in a single command; enable `--generate-all-formats` automatically for tarball output, `--output=fixtures.tar.gz`, [#1855](https://github.com/ethereum/execution-spec-tests/pull/1855).
6566
- 🔀 Refactor: Encapsulate `fill`'s fixture output options (`--output`, `--flat-output`, `--single-fixture-per-file`) into a `FixtureOutput` class ([#1471](https://github.com/ethereum/execution-spec-tests/pull/1471),[#1612](https://github.com/ethereum/execution-spec-tests/pull/1612)).
6667
- ✨ Don't warn about a "high Transaction gas_limit" for `zkevm` tests ([#1598](https://github.com/ethereum/execution-spec-tests/pull/1598)).
6768
- 🐞 `fill` no longer writes generated fixtures into an existing, non-empty output directory; it must now be empty or `--clean` must be used to delete it first ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)).

docs/filling_tests/filling_tests_command_line.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,37 @@ uv run fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_war
8888

8989
See: [Filling Tests for Features under Development](./filling_tests_dev_fork.md).
9090

91+
## Generating All Fixture Formats
92+
93+
The `--generate-all-formats` flag enables generation of all fixture formats including the optimized `BlockchainEngineXFixture` in a single command:
94+
95+
```console
96+
uv run fill --generate-all-formats tests/shanghai/
97+
```
98+
99+
This flag automatically performs a two-phase execution:
100+
101+
1. **Phase 1**: Generates pre-allocation groups for optimization.
102+
2. **Phase 2**: Generates all supported fixture formats (`StateFixture`, `BlockchainFixture`, `BlockchainEngineFixture`, `BlockchainEngineXFixture`, etc.).
103+
104+
!!! tip "Automatic enabling with tarball output"
105+
When using tarball output (`.tar.gz` files), the `--generate-all-formats` flag is automatically enabled:
106+
```console
107+
# Automatically enables --generate-all-formats due to .tar.gz output
108+
uv run fill --output=fixtures.tar.gz tests/shanghai/
109+
110+
# Equivalent to:
111+
uv run fill --generate-all-formats --output=fixtures.tar.gz tests/shanghai/
112+
```
113+
114+
!!! note "Alternative approach"
115+
You can still use the legacy approach, but this will only generate the `BlockchainEngineXFixture` format:
116+
```console
117+
# Single command that automatically does 2-phase execution
118+
# but only generates BlockchainEngineXFixture
119+
uv run fill --generate-pre-alloc-groups tests/shanghai/
120+
```
121+
91122
## Debugging the `t8n` Command
92123

93124
The `--evm-dump-dir` flag can be used to dump the inputs and outputs of every call made to the `t8n` command for debugging purposes, see [Debugging Transition Tools](./debugging_t8n_tools.md).

docs/running_tests/releases.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,14 @@ For standard releases, two tarballs are available:
3737

3838
I.e., `fixtures_develop` are a superset of `fixtures_stable`.
3939

40+
!!! tip "Generating tarballs directly via `--output` includes all fixture formats"
41+
When generating fixtures for release, specifying tarball output automatically enables all fixture formats:
42+
```console
43+
# Automatically enables --generate-all-formats due to .tar.gz output
44+
uv run fill --output=fixtures_stable.tar.gz tests/
45+
```
46+
This ensures that all fixture formats are included in the tarball release.
47+
4048
### Pre-Release and Devnet Releases
4149

4250
Intermediate releases that target specific subsets of features or tests under active development are published at @ethereum/execution-spec-tests [releases](https://github.com/ethereum/execution-spec-tests/releases).

docs/running_tests/test_formats/blockchain_test_engine_x.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
The Blockchain Engine X Test fixture format tests are included in the fixtures subdirectory `blockchain_tests_engine_x`, and use Engine API directives with optimized pre-allocation groups for improved execution performance.
44

5-
These are produced by the `StateTest` and `BlockchainTest` test specs when using the `--generate-pre-alloc-groups` and `--use-pre-alloc-groups` flags.
5+
These are produced by the `StateTest` and `BlockchainTest` test specs when using the `--generate-pre-alloc-groups` and `--use-pre-alloc-groups` flags, or by using the `--generate-all-formats` flag which generates all fixture formats including `BlockchainEngineXFixture` in a single command.
66

77
## Description
88

@@ -138,7 +138,9 @@ Engine API payload structure identical to the one defined in [Blockchain Engine
138138

139139
## Usage Notes
140140

141-
- This format is only generated when using `--generate-pre-alloc-groups` and `--use-pre-alloc-groups` flags
141+
- This format is generated when using:
142+
- `--generate-pre-alloc-groups` flag (automatically triggers 2-phase execution, generates only `BlockchainEngineXFixture`)
143+
- `--generate-all-formats` flag (automatically triggers 2-phase execution, generates all fixture formats)
142144
- The `pre_alloc` folder is essential and must be distributed with the test fixtures
143145
- Tests are grouped by identical (fork + environment + pre-allocation) combinations
144146
- The format is optimized for Engine API testing (post-Paris forks)

src/cli/pytest_commands/fill.py

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,13 @@ def create_executions(self, pytest_args: List[str]) -> List[PytestExecution]:
2727
Create execution plan that supports two-phase pre-allocation group generation.
2828
2929
Returns single execution for normal filling, or two-phase execution
30-
when --generate-pre-alloc-groups is specified.
30+
when --generate-pre-alloc-groups or --generate-all-formats is specified.
3131
"""
3232
processed_args = self.process_arguments(pytest_args)
3333

3434
# Check if we need two-phase execution
35-
if "--generate-pre-alloc-groups" in processed_args:
35+
if self._should_use_two_phase_execution(processed_args):
36+
processed_args = self._ensure_generate_all_formats_for_tarball(processed_args)
3637
return self._create_two_phase_executions(processed_args)
3738
elif "--use-pre-alloc-groups" in processed_args:
3839
# Only phase 2: using existing pre-allocation groups
@@ -110,6 +111,7 @@ def _remove_unwanted_phase1_args(self, args: List[str]) -> List[str]:
110111
# Pre-allocation group flags (we'll add our own)
111112
"--generate-pre-alloc-groups",
112113
"--use-pre-alloc-groups",
114+
"--generate-all-formats",
113115
}
114116

115117
filtered_args = []
@@ -134,7 +136,7 @@ def _remove_unwanted_phase1_args(self, args: List[str]) -> List[str]:
134136
return filtered_args
135137

136138
def _remove_generate_pre_alloc_groups_flag(self, args: List[str]) -> List[str]:
137-
"""Remove --generate-pre-alloc-groups flag from argument list."""
139+
"""Remove --generate-pre-alloc-groups flag but keep --generate-all-formats for phase 2."""
138140
return [arg for arg in args if arg != "--generate-pre-alloc-groups"]
139141

140142
def _remove_clean_flag(self, args: List[str]) -> List[str]:
@@ -145,6 +147,33 @@ def _add_use_pre_alloc_groups_flag(self, args: List[str]) -> List[str]:
145147
"""Add --use-pre-alloc-groups flag to argument list."""
146148
return args + ["--use-pre-alloc-groups"]
147149

150+
def _should_use_two_phase_execution(self, args: List[str]) -> bool:
151+
"""Determine if two-phase execution is needed."""
152+
return (
153+
"--generate-pre-alloc-groups" in args
154+
or "--generate-all-formats" in args
155+
or self._is_tarball_output(args)
156+
)
157+
158+
def _ensure_generate_all_formats_for_tarball(self, args: List[str]) -> List[str]:
159+
"""Auto-add --generate-all-formats for tarball output."""
160+
if self._is_tarball_output(args) and "--generate-all-formats" not in args:
161+
return args + ["--generate-all-formats"]
162+
return args
163+
164+
def _is_tarball_output(self, args: List[str]) -> bool:
165+
"""Check if output argument specifies a tarball (.tar.gz) path."""
166+
from pathlib import Path
167+
168+
for i, arg in enumerate(args):
169+
if arg.startswith("--output="):
170+
output_path = Path(arg.split("=", 1)[1])
171+
return str(output_path).endswith(".tar.gz")
172+
elif arg == "--output" and i + 1 < len(args):
173+
output_path = Path(args[i + 1])
174+
return str(output_path).endswith(".tar.gz")
175+
return False
176+
148177

149178
class PhilCommand(FillCommand):
150179
"""Friendly fill command with emoji reporting."""
Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
"""Test the --generate-all-formats CLI flag functionality."""
2+
3+
from unittest.mock import patch
4+
5+
from cli.pytest_commands.fill import FillCommand
6+
7+
8+
def test_generate_all_formats_creates_two_phase_execution():
9+
"""Test that --generate-all-formats triggers two-phase execution."""
10+
command = FillCommand()
11+
12+
# Mock the argument processing to bypass click context requirements
13+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
14+
# Test that --generate-all-formats triggers two-phase execution
15+
pytest_args = ["--generate-all-formats", "tests/somedir/"]
16+
executions = command.create_executions(pytest_args)
17+
18+
assert len(executions) == 2, "Expected two-phase execution"
19+
20+
# Phase 1: Should have --generate-pre-alloc-groups
21+
phase1_args = executions[0].args
22+
assert "--generate-pre-alloc-groups" in phase1_args
23+
assert "--generate-all-formats" not in phase1_args
24+
25+
# Phase 2: Should have --use-pre-alloc-groups and --generate-all-formats
26+
phase2_args = executions[1].args
27+
assert "--use-pre-alloc-groups" in phase2_args
28+
assert "--generate-all-formats" in phase2_args
29+
assert "--generate-pre-alloc-groups" not in phase2_args
30+
31+
32+
def test_generate_all_formats_preserves_other_args():
33+
"""Test that --generate-all-formats preserves other command line arguments."""
34+
command = FillCommand()
35+
36+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
37+
pytest_args = [
38+
"--generate-all-formats",
39+
"--output=custom-output",
40+
"--fork=Paris",
41+
"-v",
42+
"tests/somedir/",
43+
]
44+
executions = command.create_executions(pytest_args)
45+
46+
assert len(executions) == 2
47+
48+
# Both phases should preserve most args
49+
for execution in executions:
50+
assert "--output=custom-output" in execution.args
51+
assert "--fork=Paris" in execution.args
52+
assert "-v" in execution.args
53+
assert "tests/somedir/" in execution.args
54+
55+
56+
def test_generate_all_formats_removes_clean_from_phase2():
57+
"""Test that --clean is removed from phase 2."""
58+
command = FillCommand()
59+
60+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
61+
pytest_args = ["--generate-all-formats", "--clean", "tests/somedir/"]
62+
executions = command.create_executions(pytest_args)
63+
64+
assert len(executions) == 2
65+
66+
# Phase 1: Actually keeps --clean (it's needed for cleaning before phase 1)
67+
# Note: --clean actually remains in phase 1 args but gets filtered out
68+
# in _remove_unwanted_phase1_args
69+
70+
# Phase 2: Should not have --clean (gets removed)
71+
phase2_args = executions[1].args
72+
assert "--clean" not in phase2_args
73+
74+
75+
def test_legacy_generate_pre_alloc_groups_still_works():
76+
"""Test that the legacy --generate-pre-alloc-groups flag still works."""
77+
command = FillCommand()
78+
79+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
80+
pytest_args = ["--generate-pre-alloc-groups", "tests/somedir/"]
81+
executions = command.create_executions(pytest_args)
82+
83+
assert len(executions) == 2
84+
85+
# Phase 1: Should have --generate-pre-alloc-groups
86+
phase1_args = executions[0].args
87+
assert "--generate-pre-alloc-groups" in phase1_args
88+
89+
# Phase 2: Should have --use-pre-alloc-groups but NOT --generate-all-formats
90+
phase2_args = executions[1].args
91+
assert "--use-pre-alloc-groups" in phase2_args
92+
assert "--generate-all-formats" not in phase2_args
93+
assert "--generate-pre-alloc-groups" not in phase2_args
94+
95+
96+
def test_single_phase_without_flags():
97+
"""Test that normal execution without flags creates single phase."""
98+
command = FillCommand()
99+
100+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
101+
pytest_args = ["tests/somedir/"]
102+
executions = command.create_executions(pytest_args)
103+
104+
assert len(executions) == 1
105+
execution = executions[0]
106+
107+
assert "--generate-pre-alloc-groups" not in execution.args
108+
assert "--use-pre-alloc-groups" not in execution.args
109+
assert "--generate-all-formats" not in execution.args
110+
111+
112+
def test_tarball_output_auto_enables_generate_all_formats():
113+
"""Test that tarball output automatically enables --generate-all-formats."""
114+
command = FillCommand()
115+
116+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
117+
pytest_args = ["--output=fixtures.tar.gz", "tests/somedir/"]
118+
executions = command.create_executions(pytest_args)
119+
120+
# Should trigger two-phase execution due to tarball output
121+
assert len(executions) == 2
122+
123+
# Phase 1: Should have --generate-pre-alloc-groups
124+
phase1_args = executions[0].args
125+
assert "--generate-pre-alloc-groups" in phase1_args
126+
127+
# Phase 2: Should have --generate-all-formats (auto-added) and --use-pre-alloc-groups
128+
phase2_args = executions[1].args
129+
assert "--generate-all-formats" in phase2_args
130+
assert "--use-pre-alloc-groups" in phase2_args
131+
assert "--output=fixtures.tar.gz" in phase2_args
132+
133+
134+
def test_tarball_output_with_explicit_generate_all_formats():
135+
"""Test that explicit --generate-all-formats with tarball output works correctly."""
136+
command = FillCommand()
137+
138+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
139+
pytest_args = ["--output=fixtures.tar.gz", "--generate-all-formats", "tests/somedir/"]
140+
executions = command.create_executions(pytest_args)
141+
142+
# Should trigger two-phase execution
143+
assert len(executions) == 2
144+
145+
# Phase 2: Should have --generate-all-formats (explicit, not duplicated)
146+
phase2_args = executions[1].args
147+
assert "--generate-all-formats" in phase2_args
148+
# Ensure no duplicate flags
149+
assert phase2_args.count("--generate-all-formats") == 1
150+
151+
152+
def test_regular_output_does_not_auto_trigger_two_phase():
153+
"""Test that regular directory output doesn't auto-trigger two-phase execution."""
154+
command = FillCommand()
155+
156+
with patch.object(command, "process_arguments", side_effect=lambda x: x):
157+
pytest_args = ["--output=fixtures/", "tests/somedir/"]
158+
executions = command.create_executions(pytest_args)
159+
160+
# Should remain single-phase execution
161+
assert len(executions) == 1
162+
execution = executions[0]
163+
164+
assert "--generate-pre-alloc-groups" not in execution.args
165+
assert "--use-pre-alloc-groups" not in execution.args
166+
assert "--generate-all-formats" not in execution.args
167+
168+
169+
def test_tarball_output_detection_various_formats():
170+
"""Test tarball output detection with various argument formats."""
171+
command = FillCommand()
172+
173+
# Test --output=file.tar.gz format
174+
args1 = ["--output=test.tar.gz", "tests/somedir/"]
175+
assert command._is_tarball_output(args1) is True
176+
177+
# Test --output file.tar.gz format
178+
args2 = ["--output", "test.tar.gz", "tests/somedir/"]
179+
assert command._is_tarball_output(args2) is True
180+
181+
# Test regular directory
182+
args3 = ["--output=test/", "tests/somedir/"]
183+
assert command._is_tarball_output(args3) is False
184+
185+
# Test no output argument
186+
args4 = ["tests/somedir/"]
187+
assert command._is_tarball_output(args4) is False

0 commit comments

Comments
 (0)