tools(feat): Add defined exceptions (#384)

* tools(feat): Add exception type

* tests(fix): Use exception type

* docs: Add exception tests description

* docs(fix): tox fixes

* docs(fix): more tox fixes

* docs: Typed exceptions to consuming tests section

* docs: remove link

* changelog

* feat(docs): add links to test examples using each exception type

* Update docs/writing_tests/exception_tests.md

Co-authored-by: danceratopz <[email protected]>

---------

Co-authored-by: danceratopz <[email protected]>

Author: marioevz

docs/CHANGELOG.md

@@ -54,6 +54,7 @@ Test fixtures for use by clients are available for each release on the [Github r
 - 🔀 Updates fork name from Merge to Paris ([#363](https://github.com/ethereum/execution-spec-tests/pull/363)).
 - ✨ Add framework unit tests for post state exception verification ([#350](https://github.com/ethereum/execution-spec-tests/pull/350)).
 - ✨ Add [solc 0.8.23](https://github.com/ethereum/solidity/releases/tag/v0.8.23) support ([#373](https://github.com/ethereum/execution-spec-tests/pull/373)).
+- 💥 Tests must now use `BlockException` and `TransactionException` to define the expected exception of a given test, which can be used to test whether the client is hitting the proper exception when processing the block or transaction ([#384](https://github.com/ethereum/execution-spec-tests/pull/384)).
 
 ### 🔧 EVM Tools
 
@@ -81,6 +82,33 @@ Test fixtures for use by clients are available for each release on the [Github r
    Fixture name example:
    - Previous fixture name: `fork=Frontier`
    - New fixture name: `fork_Frontier`
+4. Produced `blockchain_tests` fixtures and their corresponding `blockchain_tests_hive` fixtures now contain the named exceptions `BlockException` and `TransactionException` as strings in the `expectException` and `validationError` fields, respectively. These exceptions can be used to test whether the client is hitting the proper exception when processing an invalid block.
+
+Blockchain test:
+
+```json
+"blocks": [
+      {
+         ...
+         "expectException": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS",
+         ...
+      }
+      ...
+]
+```
+
+Blockchain hive test:
+
+```json
+"engineNewPayloads": [
+      {
+         ...
+         "validationError": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS",
+         ...
+      }
+      ...
+]
+```
 
 ## [v1.0.6](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.6) - 2023-10-19: 🐍🏖️ Cancun Devnet 10
 

docs/consuming_tests/blockchain_test.md

@@ -197,9 +197,9 @@ Optional list of withdrawals included in the block RLP.
 
 ### `InvalidFixtureBlock`
 
-#### - `expectException`: `str`
+#### - `expectException`: [`TransactionException`](./exceptions.md#transactionexception)` | `[`BlockException`](./exceptions.md#blockexception)
 
-Expected exception message that invalidates the block.
+Expected exception that invalidates the block.
 
 #### - `rlp`: [`Bytes`](./common_types.md#bytes)
 

docs/consuming_tests/blockchain_test_hive.md

@@ -82,9 +82,17 @@ They can mismatch the hashes of the versioned blobs in the execution payload, fo
 
 Hash of the parent beacon block root.
 
-#### - `valid`: `bool`
+#### - `validationError`: [`TransactionException`](./exceptions.md#transactionexception)` | `[`BlockException`](./exceptions.md#blockexception)
 
-To be deprecated: Whether the execution payload is valid or not. Expectation is `VALID` if `true`, `INVALID` if `false`, in the `status` field of [PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1).
+Validation error expected when executing the payload.
+
+When the payload is valid, this field is not present, and a `VALID` status is
+expected in the `status` field of
+[PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1).
+
+If this field is present, the `status` field of
+[PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1)
+is expected to be `INVALID`.
 
 #### - `version`: [`Number`](./common_types.md#number)
 

docs/consuming_tests/exceptions.md

@@ -0,0 +1,22 @@
+# Exceptions
+
+Exception types are represented as a JSON string in the test fixtures.
+
+The exception converted into a string is composed of the exception type name,
+followed by a period, followed by the specific exception name.
+
+For example, the exception `INSUFFICIENT_ACCOUNT_FUNDS` of type
+`TransactionException` is represented as
+`"TransactionException.INSUFFICIENT_ACCOUNT_FUNDS"`.
+
+The JSON string can contain multiple exception types, separated by the `|`
+character, denoting that the transaction or block can throw either one of
+the exceptions.
+
+## `TransactionException`
+
+::: ethereum_test_tools.TransactionException
+
+## `BlockException`
+
+::: ethereum_test_tools.BlockException

docs/consuming_tests/state_test.md

@@ -167,7 +167,7 @@ Expected state root value that results of applying the transaction to the pre-st
 Hash of the RLP representation of the state logs result of applying the transaction to the pre-state
 (TODO: double-check this.)
 
-#### - `expectException`: `str`
+#### - `expectException`: [`TransactionException`](./exceptions.md#transactionexception)
 
 Exception that is expected to be thrown by the transaction execution (Field is missing if the transaction is expected to succeed)
 

docs/navigation.md

@@ -15,13 +15,15 @@
     * [Writing a New Test](writing_tests/writing_a_new_test.md)
     * [Referencing an EIP Spec Version](writing_tests/reference_specification.md)
     * [Verifying Changes Locally](writing_tests/verifying_changes.md)
+    * [Exception Tests](writing_tests/exception_tests.md)
   * Tutorials
     * [State Transition Tests](tutorials/state_transition.md)
   * [Consuming Tests](consuming_tests/index.md)
     * [State Tests](consuming_tests/state_test.md)
     * [Blockchain Tests](consuming_tests/blockchain_test.md)
     * [Blockchain Hive Tests](consuming_tests/blockchain_test_hive.md)
     * [Common Types](consuming_tests/common_types.md)
+    * [Exceptions](consuming_tests/exceptions.md)
   * [Getting Help](getting_help/index.md)
   * [Developer Doc](dev/index.md)
     * [Documentation](dev/docs.md)

docs/writing_tests/exception_tests.md

@@ -0,0 +1,34 @@
+# Exception Tests
+
+Exception tests are a special type of test which verify that an invalid transaction or an invalid block are correctly rejected with the expected error.
+
+## Creating an Exception Test
+
+To test for an exception, the test can use either of the following types from `ethereum_test_tools` library:
+
+1. [`TransactionException`](../consuming_tests/exceptions.md#transactionexception): To be added to the `error` field of the `Transaction` object, and to the `exception` field of the `Block` object that includes the transaction; this exception type is used when a transaction is invalid, and therefore when included in a block, the block is expected to be invalid too. This is different from valid transactions where an exception during EVM execution is expected (e.g. a revert, or out-of-gas), which can be included in valid blocks.
+
+    For an example, see [`eip3860_initcode.test_initcode.test_contract_creating_tx`](../tests/shanghai/eip3860_initcode/test_initcode/index.md#tests.shanghai.eip3860_initcode.test_initcode.test_contract_creating_tx) which raises `TransactionException.INITCODE_SIZE_EXCEEDED` in the case that the initcode size exceeds the maximum allowed size.
+
+2. [`BlockException`](../consuming_tests/exceptions.md#blockexception): To be added to the `exception` field of the `Block` object; this exception type is used when a block is expected to be invalid, but the exception is related to a block property, e.g. an invalid value of the block header.
+
+    For an example, see [`eip4844_blobs.test_excess_blob_gas.test_invalid_static_excess_blob_gas`](../tests/cancun/eip4844_blobs/test_excess_blob_gas/index.md#tests.cancun.eip4844_blobs.test_excess_blob_gas.test_invalid_static_excess_blob_gas) which raises `BlockException.INCORRECT_EXCESS_BLOB_GAS` in the case that the the `excessBlobGas` remains unchanged
+    but the parent blobs included are not `TARGET_BLOBS_PER_BLOCK`.
+
+Although exceptions can be combined with the `|` operator to indicate that a test vector can throw either one of multiple exceptions, ideally the tester should aim to use only one exception per test vector, and only use multiple exceptions on the rare instance when it is not possible to know which exception will be thrown because it depends on client implementation.
+
+## Adding a new exception
+
+If a test requires a new exception, because none of the existing ones is suitable for the test, a new exception can be added to either [`TransactionException`](../consuming_tests/exceptions.md#transactionexception) or [`BlockException`](../consuming_tests/exceptions.md#blockexception) classes.
+
+The new exception should be added as a new enum value, and the docstring of the attribute should be a string that describes the exception.
+
+The name of the exception should be unique, and should not be used by any other exception.
+
+## Test runner behavior on exception tests
+
+When an exception is added to a test vector, the test runner must check that the transaction or block is rejected with the expected exception.
+
+The test runner must map the exception key to the corresponding error string that is expected to be returned by the client.
+
+Exception mapping are particularly important in blockchain tests because the block can be invalid for multiple reasons, and the client returning a different error can mean that a verification in the client is faulty.

src/ethereum_test_tools/__init__.py

@@ -43,6 +43,7 @@
     to_hash_bytes,
     transaction_list_root,
 )
+from .exceptions import BlockException, ExceptionList, ExceptionType, TransactionException
 from .reference_spec import ReferenceSpec, ReferenceSpecTypes
 from .spec import (
     SPEC_TYPES,
@@ -68,13 +69,16 @@
     "Block",
     "BlockchainTest",
     "BlockchainTestFiller",
-    "Case",
+    "BlockException",
     "CalldataCase",
+    "Case",
     "Code",
     "CodeGasMeasure",
     "Conditional",
     "EngineAPIError",
     "Environment",
+    "ExceptionList",
+    "ExceptionType",
     "FixtureCollector",
     "Header",
     "HistoryStorageAddress",
@@ -97,6 +101,7 @@
     "TestPrivateKey",
     "TestPrivateKey2",
     "Transaction",
+    "TransactionException",
     "Withdrawal",
     "Yul",
     "YulCompiler",

src/ethereum_test_tools/common/types.py

@@ -27,6 +27,7 @@
 
 from ethereum_test_forks import Fork
 
+from ..exceptions import ExceptionList, TransactionException
 from .constants import AddrAA, TestPrivateKey
 from .conversions import (
     BytesConvertible,
@@ -1264,7 +1265,7 @@ class Transaction:
             skip=True,
         ),
     )
-    error: Optional[str] = field(
+    error: Optional[TransactionException | ExceptionList] = field(
         default=None,
         json_encoder=JSONEncoder.Field(
             skip=True,
@@ -1339,7 +1340,7 @@ def __post_init__(self) -> None:
         if self.ty >= 2 and self.max_priority_fee_per_gas is None:
             self.max_priority_fee_per_gas = 0
 
-    def with_error(self, error: str) -> "Transaction":
+    def with_error(self, error: TransactionException | ExceptionList) -> "Transaction":
         """
         Create a copy of the transaction with an added error.
         """

src/ethereum_test_tools/exceptions/__init__.py

@@ -0,0 +1,7 @@
+"""
+Exceptions for invalid execution.
+"""
+
+from .exceptions import BlockException, ExceptionList, ExceptionType, TransactionException
+
+__all__ = ["BlockException", "ExceptionType", "ExceptionList", "TransactionException"]