feat(fixtures,specs,tests): Transaction tests, EIP-7702 convert invalid tx tests (#933)

* new(fixtures): TransactionFixture format

* new(specs): TransactionTest format

* fix(forks): rename intrinsic cost calc parameter name

* fix(tests): rename intrinsic cost calc parameter name

* fix(specs): use `fork` intrinsic gas calculator

* fix(fixtures): Add `TransactionFixture` to file

* fix(fixtures): transaction type optional fields

* feat(fixtures): Add fixture format to `_info` for easier parsing

* fix(cli/check_fixtures): Allow a single fixture check

* fix(types): Address(0) == ""

* fix(fixtures,specs): fixes

* feat(types): Allow nonce list in auth tuple, for testing purposes

* fix(tests): EIP-7702: Convert invalid tx tests

* docs: update, changelog

* fix(docs): tox

* fix(tests): EIP-7702: Move invalid tx tests to its own file

* new(docs): Add Transaction Test to `consuming_tests`

* fix(fixtures): fix `fixture_type_discriminator`

* Update docs/consuming_tests/index.md

Co-authored-by: danceratopz <[email protected]>

* Apply suggestions from code review (fixture_type_discriminator)

Co-authored-by: danceratopz <[email protected]>

* nit

---------

Co-authored-by: danceratopz <[email protected]>

Author: marioevz

docs/CHANGELOG.md

@@ -60,6 +60,7 @@ Test fixtures for use by clients are available for each release on the [Github r
 - ✨ Add the `eest clean` command that helps delete generated files and directories from the repository ([#980](https://github.com/ethereum/execution-spec-tests/pull/980)).
 - ✨ Add framework changes for EIP-7742, required for Prague devnet-5 ([#931](https://github.com/ethereum/execution-spec-tests/pull/931)).
 - ✨ Add the `eest make env` command that generates a default env file (`env.yaml`)([#996](https://github.com/ethereum/execution-spec-tests/pull/996)).
+- ✨ Generate Transaction Test type ([#933](https://github.com/ethereum/execution-spec-tests/pull/933)).
 
 ### 🔧 EVM Tools
 

docs/consuming_tests/index.md

@@ -7,6 +7,7 @@
 | [State Tests](./state_test.md) | directly via a `statetest`-like command<br/> (e.g., [go-ethereum/cmd/evm/staterunner.go](https://github.com/ethereum/go-ethereum/blob/509a64ffb9405942396276ae111d06f9bded9221/cmd/evm/staterunner.go#L35)) | `./fixtures/state_tests/` |
 | [Blockchain Tests](./blockchain_test.md) | directly via a `blocktest`-like command<br/> (e.g., [go-ethereum/cmd/evm/blockrunner.go](https://github.com/ethereum/go-ethereum/blob/509a64ffb9405942396276ae111d06f9bded9221/cmd/evm/blockrunner.go#L39)) | `./fixtures/blockchain_tests/` |
 | [Blockchain Engine Tests](./blockchain_test_engine.md) | in the [Hive `pyspec` simulator](https://github.com/ethereum/hive/tree/master/simulators/ethereum/pyspec#readme) via the Engine API and other RPC endpoints  | `./fixtures/blockchain_tests_engine/` |
+| [Transaction Tests](./transaction_test.md) | directly via a `t9`-like command<br/> (e.g., [go-ethereum's `evm t9`](https://github.com/ethereum/go-ethereum/tree/67a3b087951a3f3a8e341ae32b6ec18f3553e5cc/cmd/evm#transaction-tool)) | `./fixtures/transaction_tests/` |
 
 Here's a top-level comparison of the different methods of consuming tests:
 

docs/consuming_tests/transaction_test.md

@@ -0,0 +1,67 @@
+# Transaction Tests <!-- markdownlint-disable MD051 (MD051=link-fragments "Link fragments should be valid") -->
+
+The Transaction Test fixture format tests are included in the fixtures subdirectory `transaction_tests`.
+
+These are produced by the `TransactionTest` test spec.
+
+## Description
+
+The transaction test fixture format is used to test client's transaction RLP parsing without executing the transaction on the EVM.
+
+It does so by defining a transaction binary RLP representation, and whether the transaction should be accepted or rejected by the client in each fork.
+
+A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`Fixture`](#fixture) test object, with the key string representing the test name.
+
+The JSON file path plus the test name are used as the unique test identifier.
+
+The transaction test fixture format could contain multiple test vectors per test object, each represented by an element in the mapping of lists of the `result` field.
+
+However tests generated by the `execution-spec-tests` repository do **not** use this feature, as every single test object contains only a single test vector.
+
+## Consumption
+
+For each [`Fixture`](#fixture) test object in the JSON fixture file, perform the following steps:
+
+1. Obtain the [`txbytes`](#-txbytes-bytes) serialized bytes of the transaction to be parsed.
+2. For each [`Fork`](./common_types.md#fork) key of [`result`](#-result-mappingforkfixtureresult) in the test:
+
+    1. Assume the fork schedule according to the current [`Fork`](./common_types.md#fork) key.
+    2. Using the [`txbytes`](#-txbytes-bytes), attempt to decode the transaction.
+    3. If the transaction could not be decoded:
+        - If the [`hash`](#-hash-hash-none) field is present, fail the test.
+        - Compare the exception thrown with the expected exception contained in the [`exception`](#-exception-transactionexception) field, and fail the test if they do not match.
+        - Proceed to the next fork.
+    4. If the transaction could be decoded:
+        - Compare the calculated hash with the expected hash contained in the [`hash`](#-hash-hash-none) field, and fail the test if they do not match.
+        - Compare the calculated intrinsic gas with the expected intrinsic gas contained in the [`intrinsicGas`](#-intrinsicgas-zeropaddedhexnumber) field, and fail the test if they do not match.
+        - Compare the calculated sender with the expected sender contained in the [`sender`](#-sender-address) field, and fail the test if they do not match.
+
+## Structures
+
+### `Fixture`
+
+#### - `txbytes`: [`Bytes`](./common_types.md#bytes)
+
+Serialized bytes of the transaction under test.
+
+#### - `result`: [`Mapping`](./common_types.md#mapping)`[`[`Fork`](./common_types.md#fork)`,`[`FixtureResult`](#fixtureresult) `]`
+
+Mapping of results for verification per fork, where each key-value represents a single possible outcome of the transaction parsed in the given fork.
+
+### `FixtureResult`
+
+#### - `hash`: [`Hash`](./common_types.md#hash) `| None`
+
+Calculated hash of the transaction (Field is missing if the transaction is expected to fail).
+
+#### - `intrinsicGas`: [`ZeroPaddedHexNumber`](./common_types.md#zeropaddedhexnumber)
+
+Total intrinsic gas cost of the transaction (Field is missing if the transaction is expected to fail).
+
+#### - `sender`: [`Address`](./common_types.md#address)
+
+Sender address of the transaction (Field is missing if the transaction is expected to fail).
+
+#### - `exception`: [`TransactionException`](./exceptions.md#transactionexception)
+
+Exception that is expected to be thrown by the transaction parsing (Field is missing if the transaction is expected to succeed).

docs/navigation.md

@@ -28,6 +28,7 @@
     * [Blockchain Tests](consuming_tests/blockchain_test.md)
     * [Blockchain Engine Tests](consuming_tests/blockchain_test_engine.md)
     * [EOF Tests](consuming_tests/eof_test.md)
+    * [Transaction Tests](consuming_tests/transaction_test.md)
     * [Common Types](consuming_tests/common_types.md)
     * [Exceptions](consuming_tests/exceptions.md)
   * [Executing Tests](executing_tests/index.md)

docs/writing_tests/types_of_tests.md

@@ -1,9 +1,10 @@
 # Types of tests
 
-There are currently two types of tests that can be produced by a test spec:
+There are currently three types of tests that can be produced by a test spec:
 
 1. State Tests
 2. Blockchain Tests
+3. Transaction Tests
 
 ## State Tests
 
@@ -51,6 +52,21 @@ def test_blob_type_tx_pre_fork(
     """
 ```
 
+## Transaction Tests
+
+### Purpose
+
+Test correct transaction rejection/acceptance of a serialized transaction (currently RLP only).
+
+### Use cases
+
+- Verify that a badly formatted transaction is correctly rejected by the client.
+- Verify that a transaction with an invalid value in one of its fields is correctly rejected by the client.
+
+!!! info
+
+    Using the `execute` command, transaction tests can be sent to clients in a live network using the `eth_sendRawTransaction` endpoint.
+
 ## Deciding on a test type
 
 ### Prefer `state_test` for single transactions

src/cli/check_fixtures.py

@@ -4,6 +4,7 @@
 """
 
 from pathlib import Path
+from typing import Generator
 
 import click
 from rich.progress import BarColumn, Progress, TaskProgressColumn, TextColumn, TimeElapsedColumn
@@ -59,10 +60,10 @@ def check_json(json_file_path: Path):
 @click.option(
     "--input",
     "-i",
-    "input_dir",
-    type=click.Path(exists=True, file_okay=False, dir_okay=True, readable=True),
+    "input_str",
+    type=click.Path(exists=True, file_okay=True, dir_okay=True, readable=True),
     required=True,
-    help="The input directory containing json fixture files",
+    help="The input json file or directory containing json fixture files",
 )
 @click.option(
     "--quiet",
@@ -83,17 +84,25 @@ def check_json(json_file_path: Path):
     expose_value=True,
     help="Stop and raise any exceptions encountered while checking fixtures.",
 )
-def check_fixtures(input_dir: str, quiet_mode: bool, stop_on_error: bool):
+def check_fixtures(input_str: str, quiet_mode: bool, stop_on_error: bool):
     """
     Perform some checks on the fixtures contained in the specified directory.
     """
-    input_path = Path(input_dir)
+    input_path = Path(input_str)
     success = True
     file_count = 0
     filename_display_width = 25
-    if not quiet_mode:
+    if input_path.is_file():
+        file_count = 1
+    elif not quiet_mode:
         file_count = count_json_files_exclude_index(input_path)
 
+    def get_input_files() -> Generator[Path, None, None]:
+        if input_path.is_file():
+            yield input_path
+        else:
+            yield from input_path.rglob("*.json")
+
     with Progress(
         TextColumn(
             f"[bold cyan]{{task.fields[filename]:<{filename_display_width}}}[/]", justify="left"
@@ -106,7 +115,7 @@ def check_fixtures(input_dir: str, quiet_mode: bool, stop_on_error: bool):
     ) as progress:
 
         task_id = progress.add_task("Checking fixtures", total=file_count, filename="...")
-        for json_file_path in input_path.rglob("*.json"):
+        for json_file_path in get_input_files():
             if json_file_path.name == "index.json":
                 continue
 

src/ethereum_test_fixtures/__init__.py

@@ -11,6 +11,7 @@
 from .collector import FixtureCollector, TestInfo
 from .eof import Fixture as EOFFixture
 from .state import Fixture as StateFixture
+from .transaction import Fixture as TransactionFixture
 from .verify import FixtureVerifier
 
 FIXTURE_FORMATS: Dict[str, FixtureFormat] = {
@@ -20,6 +21,7 @@
         BlockchainEngineFixture,
         EOFFixture,
         StateFixture,
+        TransactionFixture,
     ]
 }
 __all__ = [
@@ -34,4 +36,5 @@
     "FixtureVerifier",
     "StateFixture",
     "TestInfo",
+    "TransactionFixture",
 ]

src/ethereum_test_fixtures/base.py

@@ -71,6 +71,7 @@ def fill_info(
         self.info["filling-transition-tool"] = t8n_version
         self.info["description"] = test_case_description
         self.info["url"] = fixture_source_url
+        self.info["fixture_format"] = self.fixture_format_name
         if ref_spec is not None:
             ref_spec.write_info(self.info)
 

src/ethereum_test_fixtures/file.py

@@ -4,7 +4,9 @@
 
 import json
 from pathlib import Path
-from typing import Any, Dict, Optional
+from typing import Annotated, Any, Dict, Optional
+
+from pydantic import Discriminator, Tag
 
 from ethereum_test_base_types import EthereumTestRootModel
 
@@ -13,8 +15,11 @@
 from .blockchain import Fixture as BlockchainFixture
 from .eof import Fixture as EOFFixture
 from .state import Fixture as StateFixture
+from .transaction import Fixture as TransactionFixture
 
-FixtureModel = BlockchainFixture | BlockchainEngineFixture | StateFixture | EOFFixture
+FixtureModel = (
+    BlockchainFixture | BlockchainEngineFixture | StateFixture | EOFFixture | TransactionFixture
+)
 
 
 class BaseFixturesRootModel(EthereumTestRootModel):
@@ -101,6 +106,7 @@ def from_json_data(
             BlockchainFixture: BlockchainFixtures,
             BlockchainEngineFixture: BlockchainEngineFixtures,
             StateFixture: StateFixtures,
+            TransactionFixture: TransactionFixtures,
             EOFFixture: EOFFixtures,
         }
 
@@ -114,12 +120,34 @@ def from_json_data(
         return model_class(root=json_data)
 
 
+def fixture_format_discriminator(v: Any) -> str | None:
+    """
+    A discriminator function that returns the model type as a string.
+    """
+    if v is None:
+        return None
+    if isinstance(v, dict):
+        info_dict = v["_info"]
+    elif hasattr(v, "info"):
+        info_dict = v.info
+    return info_dict.get("fixture_format")
+
+
 class Fixtures(BaseFixturesRootModel):
     """
     A model that can contain any fixture type.
     """
 
-    root: Dict[str, BlockchainFixture | BlockchainEngineFixture | StateFixture]
+    root: Dict[
+        str,
+        Annotated[
+            Annotated[BlockchainFixture, Tag(BlockchainFixture.fixture_format_name)]
+            | Annotated[BlockchainEngineFixture, Tag(BlockchainEngineFixture.fixture_format_name)]
+            | Annotated[StateFixture, Tag(StateFixture.fixture_format_name)]
+            | Annotated[TransactionFixture, Tag(TransactionFixture.fixture_format_name)],
+            Discriminator(fixture_format_discriminator),
+        ],
+    ]
 
 
 class BlockchainFixtures(BaseFixturesRootModel):
@@ -152,6 +180,16 @@ class StateFixtures(BaseFixturesRootModel):
     root: Dict[str, StateFixture]
 
 
+class TransactionFixtures(BaseFixturesRootModel):
+    """
+    Defines a top-level model containing multiple transaction test fixtures in a
+    dictionary of (fixture-name, fixture) pairs. This is the format used in JSON
+    fixture files for transaction tests.
+    """
+
+    root: Dict[str, TransactionFixture]
+
+
 class EOFFixtures(BaseFixturesRootModel):
     """
     Defines a top-level model containing multiple state test fixtures in a

src/ethereum_test_fixtures/transaction.py

@@ -0,0 +1,44 @@
+"""
+TransactionTest types
+"""
+
+from typing import ClassVar, Mapping
+
+from pydantic import Field
+
+from ethereum_test_base_types import Address, Bytes, Hash, ZeroPaddedHexNumber
+from ethereum_test_exceptions import TransactionExceptionInstanceOrList
+from ethereum_test_types.types import CamelModel
+
+from .base import BaseFixture
+
+
+class FixtureResult(CamelModel):
+    """
+    The per-network (fork) result structure.
+    """
+
+    hash: Hash | None = None
+    intrinsic_gas: ZeroPaddedHexNumber
+    sender: Address | None = None
+    exception: TransactionExceptionInstanceOrList | None = None
+
+
+class Fixture(BaseFixture):
+    """
+    Fixture for a single TransactionTest.
+    """
+
+    fixture_format_name: ClassVar[str] = "transaction_test"
+    description: ClassVar[str] = "Tests that generate a transaction test fixture."
+
+    result: Mapping[str, FixtureResult]
+    transaction: Bytes = Field(..., alias="txbytes")
+
+    def get_fork(self) -> str | None:
+        """
+        Returns the fork of the fixture as a string.
+        """
+        forks = list(self.result.keys())
+        assert len(forks) == 1, "Expected transaction test fixture with single fork"
+        return forks[0]