feat(consume): add ExceptionMapper support to the consume-engine simulator (#1416)

* fix(exceptions): UndefinedException return on new

* feat(rpc,plugins/consume): Support exception mapper

* fix(exceptions): Allow undefined mapper

* feat(plugins/consume): Add more mappers

* fix(logging): don't use newlines within log messages

* refactor(exceptions,specs): ExceptionWithMessage contains a list

* new(exceptions): Add `INVALID_BLOCK_HASH` exception

* feat(clis): Add ethjs exceptions

* feat(clis): Add geth exceptions

* feat(plugins/consume): Add besu exceptions

* feat(plugins/consume): Add nethermind exceptions

* feat(plugins/consume): Add reth exceptions

* feat(consume): add `--strict-exception-matching` parameter

* more reth exceptions

* refactor mappers

* add ethjs exception

* fix(tests): EIP-7685: Add expected exception to `test_invalid_multi_type_requests_engine`

* add reth exception

* fix(plugins/consume): log failure

* feat(clis): Add Erigon Exception Mapper

* feat(specs): Allow setting expected block exception for state tests (only affects generated block tests)

* fix(tests): Add block exceltion to test_blob_type_tx_pre_fork

* fix(clis): Exceptions in mappers

* change(plugins/consume): Enable strict exception matching by default

* fix(specs): Exception message check on strict match

* docs: changelog

* chore(clis): update exception mappers for TYPE_4_TX_PRE_FORK

* refactor(consume): cache exception mappers in session-scoped fixture

* refactor(consume): make string match more robust by applying `lower()`

* feat(consume): allow disabling of strict exception checking per fork

* refactor(consume/hive): create `strict_exception_matching` fixture

* fix: whitelist

* changelog: Add exception

* changelog: Invalid versioned hashes exception

* refactor(consume/engine): Better error logging

* fix(clis/reth): Exception mapper

* fix(clis/besu): Exception mapper

* fix(clis/geth): Exception mapper

* fix(clis/ethereumjs): Exception mapper

* fix(clis/erigon): Exception mapper

---------

Co-authored-by: danceratopz <[email protected]>

Author: marioevz

docs/CHANGELOG.md

@@ -8,6 +8,14 @@ Test fixtures for use by clients are available for each release on the [Github r
 
 ### 💥 Breaking Change
 
+#### Consume engine strict exception checking
+
+`consume engine` now checks exceptions returned by the execution clients in their Engine API responses, specifically in the `validationError`field of the `engine_newPayloadVX` method.
+
+While not strictly a breaking change since tests will continue to run normally, failures are expected if a client modifies their exception messages.
+
+This feature can be disabled by using `--disable-strict-exception-matching` for specific clients or forks.
+
 ### 🛠️ Framework
 
 #### `fill`
@@ -19,6 +27,7 @@ Test fixtures for use by clients are available for each release on the [Github r
 #### `consume`
 
 - 🐞 Fix fixture tarball downloading with regular, non-Github release URLS and with numerical versions in regular release specs, e.g., `[email protected]` ([#1437](https://github.com/ethereum/execution-spec-tests/pull/1437)).
+- ✨ `consume engine` now has strict exception mapping enabled by default ([#1416](https://github.com/ethereum/execution-spec-tests/pull/1416)).
 
 #### Tools
 
@@ -27,7 +36,10 @@ Test fixtures for use by clients are available for each release on the [Github r
 
 #### Exceptions
 
-- ✨ New exceptions `BlockException.SYSTEM_CONTRACT_EMPTY` and `BlockException.SYSTEM_CONTRACT_CALL_FAILED` to handle EIP updates [#9508](https://github.com/ethereum/EIPs/pull/9508) and [#9582](https://github.com/ethereum/EIPs/pull/9582) ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)).
+- ✨ `BlockException.SYSTEM_CONTRACT_EMPTY`: Raised when a required system contract was not found in the state by the time it was due to execution with a system transaction call ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)).
+- ✨ `BlockException.SYSTEM_CONTRACT_CALL_FAILED`: Raised when a system contract call made by a system transaction fails ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)).
+- ✨ `BlockException.INVALID_BLOCK_HASH`: Raised when the calculated block hash does not match the expectation (Currently only during Engine API calls) ([#1416](https://github.com/ethereum/execution-spec-tests/pull/1416)).
+- ✨ `BlockException.INVALID_VERSIONED_HASHES`: Raised when a discrepancy is found between versioned hashes in the payload and the ones found in the transactions ([#1416](https://github.com/ethereum/execution-spec-tests/pull/1416)).
 
 ### 🧪 Test Cases
 
@@ -37,6 +49,7 @@ Test fixtures for use by clients are available for each release on the [Github r
 - ✨ [EIP-7623](https://eips.ethereum.org/EIPS/eip-7623): Additionally parametrize transaction validity tests with the `to` set to an EOA account (previously only contracts) ([#1422](https://github.com/ethereum/execution-spec-tests/pull/1422)).
 - ✨ [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Add EIP-7251 test cases for modified consolidations contract that allows more consolidations ([#1465](https://github.com/ethereum/execution-spec-tests/pull/1465)).
 - ✨ [EIP-6110](https://eips.ethereum.org/EIPS/eip-6110): Add extra deposit request edge cases, sending eth to the deposit contract while sending a deposit request ([#1467](https://github.com/ethereum/execution-spec-tests/pull/1467)).
+- ✨ [EIP-6110](https://eips.ethereum.org/EIPS/eip-6110): Add cases for deposit log layout and other topics (ERC-20) transfer ([#1371](https://github.com/ethereum/execution-spec-tests/pull/1371)).
 - ✨ [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Remove pytest skips for consolidation request cases ([#1449](https://github.com/ethereum/execution-spec-tests/pull/1449)).
 - ✨ [EIP-7002](https://eips.ethereum.org/EIPS/eip-7002), [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Add cases to verify behavior of contracts missing at fork ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)).
 - ✨ [EIP-7002](https://eips.ethereum.org/EIPS/eip-7002), [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Add cases to verify behavior of system contract errors invalidating a block ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)).

src/ethereum_clis/clis/besu.py

@@ -13,6 +13,7 @@
 
 from ethereum_test_base_types import BlobSchedule
 from ethereum_test_exceptions import (
+    BlockException,
     EOFException,
     ExceptionBase,
     ExceptionMapper,
@@ -212,36 +213,53 @@ class BesuExceptionMapper(ExceptionMapper):
     """Translate between EEST exceptions and error strings returned by Besu."""
 
     mapping_substring: ClassVar[Dict[ExceptionBase, str]] = {
-        TransactionException.TYPE_4_TX_CONTRACT_CREATION: (
-            "transaction code delegation transactions must have a non-empty code delegation list"
+        TransactionException.INSUFFICIENT_MAX_FEE_PER_BLOB_GAS: (
+            "transaction invalid tx max fee per blob gas less than block blob gas fee"
         ),
-        TransactionException.INSUFFICIENT_ACCOUNT_FUNDS: (
-            "exceeds transaction sender account balance"
+        TransactionException.INSUFFICIENT_MAX_FEE_PER_GAS: (
+            "transaction invalid gasPrice is less than the current BaseFee"
         ),
-        TransactionException.TYPE_3_TX_MAX_BLOB_GAS_ALLOWANCE_EXCEEDED: (
-            "would exceed block maximum"
+        TransactionException.PRIORITY_GREATER_THAN_MAX_FEE_PER_GAS: (
+            "transaction invalid max priority fee per gas cannot be greater than max fee per gas"
         ),
-        TransactionException.INSUFFICIENT_MAX_FEE_PER_BLOB_GAS: (
-            "max fee per blob gas less than block blob gas fee"
+        TransactionException.TYPE_3_TX_INVALID_BLOB_VERSIONED_HASH: "Invalid versionedHash",
+        TransactionException.TYPE_3_TX_CONTRACT_CREATION: (
+            "transaction invalid transaction blob transactions must have a to address"
         ),
-        TransactionException.INSUFFICIENT_MAX_FEE_PER_GAS: (
-            "gasPrice is less than the current BaseFee"
+        TransactionException.TYPE_3_TX_WITH_FULL_BLOBS: (
+            "Failed to decode transactions from block parameter"
         ),
+        TransactionException.TYPE_3_TX_ZERO_BLOBS: (
+            "Failed to decode transactions from block parameter"
+        ),
+        TransactionException.TYPE_3_TX_MAX_BLOB_GAS_ALLOWANCE_EXCEEDED: "Invalid Blob Count",
+        TransactionException.TYPE_3_TX_BLOB_COUNT_EXCEEDED: "Invalid Blob Count",
         TransactionException.TYPE_3_TX_PRE_FORK: (
-            "Transaction type BLOB is invalid, accepted transaction types are "
-            "[EIP1559, ACCESS_LIST, FRONTIER]"
+            "Transaction type BLOB is invalid, accepted transaction types are"
         ),
-        TransactionException.TYPE_3_TX_INVALID_BLOB_VERSIONED_HASH: (
-            "Only supported hash version is 0x01, sha256 hash."
+        TransactionException.TYPE_4_EMPTY_AUTHORIZATION_LIST: (
+            "transaction invalid transaction code delegation transactions must have a "
+            "non-empty code delegation list"
         ),
-        # This message is the same as TYPE_3_TX_MAX_BLOB_GAS_ALLOWANCE_EXCEEDED
-        TransactionException.TYPE_3_TX_BLOB_COUNT_EXCEEDED: "exceed block maximum",
-        TransactionException.TYPE_3_TX_ZERO_BLOBS: (
-            "Blob transaction must have at least one versioned hash"
+        TransactionException.TYPE_4_TX_CONTRACT_CREATION: (
+            "transaction invalid transaction code delegation transactions must have a to address"
+        ),
+        TransactionException.TYPE_4_TX_PRE_FORK: (
+            "transaction invalid Transaction type DELEGATE_CODE is invalid"
+        ),
+        TransactionException.INVALID_DEPOSIT_EVENT_LAYOUT: "Invalid deposit log",
+        BlockException.RLP_STRUCTURES_ENCODING: (
+            "Failed to decode transactions from block parameter"
+        ),
+        BlockException.INCORRECT_EXCESS_BLOB_GAS: (
+            "Payload excessBlobGas does not match calculated excessBlobGas"
+        ),
+        BlockException.BLOB_GAS_USED_ABOVE_LIMIT: (
+            "Payload BlobGasUsed does not match calculated BlobGasUsed"
+        ),
+        BlockException.INCORRECT_BLOB_GAS_USED: (
+            "Payload BlobGasUsed does not match calculated BlobGasUsed"
         ),
-        TransactionException.INTRINSIC_GAS_TOO_LOW: "intrinsic gas",
-        TransactionException.INITCODE_SIZE_EXCEEDED: "exceeds maximum size",
-        TransactionException.NONCE_MISMATCH_TOO_LOW: "below sender account nonce",
         # TODO EVMONE needs to differentiate when the section is missing in the header or body
         EOFException.MISSING_STOP_OPCODE: "err: no_terminating_instruction",
         EOFException.MISSING_CODE_HEADER: "err: code_section_missing",
@@ -283,4 +301,32 @@ class BesuExceptionMapper(ExceptionMapper):
         EOFException.TOO_MANY_CONTAINERS: "err: too_many_container_sections",
         EOFException.INVALID_CODE_SECTION_INDEX: "err: invalid_code_section_index",
     }
-    mapping_regex: ClassVar[Dict[ExceptionBase, str]] = {}
+    mapping_regex = {
+        BlockException.INVALID_REQUESTS: (
+            r"Invalid execution requests|Requests hash mismatch, calculated: 0x[0-9a-f]+ header: "
+            r"0x[0-9a-f]+"
+        ),
+        BlockException.INVALID_BLOCK_HASH: (
+            r"Computed block hash 0x[0-9a-f]+ does not match block hash parameter 0x[0-9a-f]+"
+        ),
+        BlockException.SYSTEM_CONTRACT_CALL_FAILED: (
+            r"System call halted|System call did not execute to completion"
+        ),
+        TransactionException.INITCODE_SIZE_EXCEEDED: (
+            r"transaction invalid Initcode size of \d+ exceeds maximum size of \d+"
+        ),
+        TransactionException.INSUFFICIENT_ACCOUNT_FUNDS: (
+            r"transaction invalid transaction up-front cost 0x[0-9a-f]+ exceeds transaction "
+            r"sender account balance 0x[0-9a-f]+"
+        ),
+        TransactionException.INTRINSIC_GAS_TOO_LOW: (
+            r"transaction invalid intrinsic gas cost \d+ exceeds gas limit \d+"
+        ),
+        TransactionException.SENDER_NOT_EOA: (
+            r"transaction invalid Sender 0x[0-9a-f]+ has deployed code and so is not authorized "
+            r"to send transactions"
+        ),
+        TransactionException.NONCE_MISMATCH_TOO_LOW: (
+            r"transaction invalid transaction nonce \d+ below sender account nonce \d+"
+        ),
+    }

src/ethereum_clis/clis/erigon.py

@@ -0,0 +1,42 @@
+"""Erigon execution client transition tool."""
+
+from ethereum_test_exceptions import BlockException, ExceptionMapper, TransactionException
+
+
+class ErigonExceptionMapper(ExceptionMapper):
+    """Erigon exception mapper."""
+
+    mapping_substring = {
+        TransactionException.SENDER_NOT_EOA: "sender not an eoa",
+        TransactionException.INITCODE_SIZE_EXCEEDED: "max initcode size exceeded",
+        TransactionException.INSUFFICIENT_ACCOUNT_FUNDS: (
+            "insufficient funds for gas * price + value"
+        ),
+        TransactionException.INTRINSIC_GAS_TOO_LOW: "intrinsic gas too low",
+        TransactionException.INSUFFICIENT_MAX_FEE_PER_GAS: "fee cap less than block base fee",
+        TransactionException.PRIORITY_GREATER_THAN_MAX_FEE_PER_GAS: "tip higher than fee cap",
+        TransactionException.INSUFFICIENT_MAX_FEE_PER_BLOB_GAS: "max fee per blob gas too low",
+        TransactionException.NONCE_MISMATCH_TOO_LOW: "nonce too low",
+        TransactionException.TYPE_3_TX_PRE_FORK: "blob txn is not supported by signer",
+        TransactionException.TYPE_3_TX_INVALID_BLOB_VERSIONED_HASH: (
+            "invalid blob versioned hash, must start with VERSIONED_HASH_VERSION_KZG"
+        ),
+        TransactionException.TYPE_3_TX_ZERO_BLOBS: "a blob stx must contain at least one blob",
+        TransactionException.TYPE_3_TX_WITH_FULL_BLOBS: "rlp: expected String or Byte",
+        TransactionException.TYPE_3_TX_CONTRACT_CREATION: "wrong size for To: 0",
+        TransactionException.TYPE_3_TX_MAX_BLOB_GAS_ALLOWANCE_EXCEEDED: (
+            "blobs/blobgas exceeds max"
+        ),
+        TransactionException.TYPE_4_EMPTY_AUTHORIZATION_LIST: (
+            "SetCodeTransaction without authorizations is invalid"
+        ),
+        TransactionException.TYPE_4_TX_CONTRACT_CREATION: "wrong size for To: 0",
+        TransactionException.TYPE_4_TX_PRE_FORK: "setCode tx is not supported by signer",
+        TransactionException.INVALID_DEPOSIT_EVENT_LAYOUT: "could not parse requests logs",
+        BlockException.INVALID_REQUESTS: "invalid requests root hash in header",
+        BlockException.INVALID_BLOCK_HASH: "invalid block hash",
+    }
+    mapping_regex = {
+        BlockException.INCORRECT_BLOB_GAS_USED: r"blobGasUsed by execution: \d+, in header: \d+",
+        BlockException.INCORRECT_EXCESS_BLOB_GAS: r"invalid excessBlobGas: have \d+, want \d+",
+    }

src/ethereum_clis/clis/ethereumjs.py

@@ -5,6 +5,7 @@
 from typing import ClassVar, Dict, Optional
 
 from ethereum_test_exceptions import (
+    BlockException,
     EOFException,
     ExceptionBase,
     ExceptionMapper,
@@ -48,30 +49,45 @@ class EthereumJSExceptionMapper(ExceptionMapper):
     """Translate between EEST exceptions and error strings returned by EthereumJS."""
 
     mapping_substring: ClassVar[Dict[ExceptionBase, str]] = {
-        TransactionException.TYPE_4_TX_CONTRACT_CREATION: (
-            "set code transaction must not be a create transaction"
-        ),
-        TransactionException.INSUFFICIENT_ACCOUNT_FUNDS: (
-            "insufficient funds for gas * price + value"
-        ),
         TransactionException.TYPE_3_TX_MAX_BLOB_GAS_ALLOWANCE_EXCEEDED: (
             "would exceed maximum allowance"
         ),
         TransactionException.INSUFFICIENT_MAX_FEE_PER_BLOB_GAS: (
-            "max fee per blob gas less than block blob gas fee"
+            "Invalid 4844 transactions: undefined"
         ),
         TransactionException.INSUFFICIENT_MAX_FEE_PER_GAS: (
-            "max fee per gas less than block base fee"
+            "tx unable to pay base fee (EIP-1559 tx)"
         ),
-        TransactionException.TYPE_3_TX_PRE_FORK: (
-            "blob tx used but field env.ExcessBlobGas missing"
+        TransactionException.PRIORITY_GREATER_THAN_MAX_FEE_PER_GAS: (
+            "maxFeePerGas cannot be less than maxPriorityFeePerGas"
+        ),
+        TransactionException.TYPE_3_TX_INVALID_BLOB_VERSIONED_HASH: (
+            "versioned hash does not start with KZG commitment version"
         ),
-        TransactionException.TYPE_3_TX_INVALID_BLOB_VERSIONED_HASH: "has invalid hash version",
         # This message is the same as TYPE_3_TX_MAX_BLOB_GAS_ALLOWANCE_EXCEEDED
         TransactionException.TYPE_3_TX_BLOB_COUNT_EXCEEDED: "exceed maximum allowance",
-        TransactionException.TYPE_3_TX_ZERO_BLOBS: "blob transaction missing blob hashes",
+        TransactionException.TYPE_3_TX_ZERO_BLOBS: "tx should contain at least one blob",
+        TransactionException.TYPE_3_TX_WITH_FULL_BLOBS: "Invalid EIP-4844 transaction",
+        TransactionException.TYPE_3_TX_CONTRACT_CREATION: (
+            'tx should have a "to" field and cannot be used to create contracts'
+        ),
+        TransactionException.TYPE_4_EMPTY_AUTHORIZATION_LIST: (
+            "Invalid EIP-7702 transaction: authorization list is empty"
+        ),
         TransactionException.INTRINSIC_GAS_TOO_LOW: "is lower than the minimum gas limit of",
-        TransactionException.INITCODE_SIZE_EXCEEDED: "max initcode size exceeded",
+        TransactionException.INITCODE_SIZE_EXCEEDED: (
+            "the initcode size of this transaction is too large"
+        ),
+        TransactionException.TYPE_4_TX_CONTRACT_CREATION: (
+            'tx should have a "to" field and cannot be used to create contracts'
+        ),
+        TransactionException.INSUFFICIENT_ACCOUNT_FUNDS: (
+            "sender doesn't have enough funds to send tx"
+        ),
+        TransactionException.NONCE_MISMATCH_TOO_LOW: "the tx doesn't have the correct nonce",
+        TransactionException.INVALID_DEPOSIT_EVENT_LAYOUT: "Error verifying block while running",
+        BlockException.INCORRECT_EXCESS_BLOB_GAS: "Invalid 4844 transactions",
+        BlockException.INVALID_RECEIPTS_ROOT: "invalid receipttrie",
         # TODO EVMONE needs to differentiate when the section is missing in the header or body
         EOFException.MISSING_STOP_OPCODE: "err: no_terminating_instruction",
         EOFException.MISSING_CODE_HEADER: "err: code_section_missing",
@@ -113,4 +129,22 @@ class EthereumJSExceptionMapper(ExceptionMapper):
         EOFException.TOO_MANY_CONTAINERS: "err: too_many_container_sections",
         EOFException.INVALID_CODE_SECTION_INDEX: "err: invalid_code_section_index",
     }
-    mapping_regex: ClassVar[Dict[ExceptionBase, str]] = {}
+    mapping_regex: ClassVar[Dict[ExceptionBase, str]] = {
+        TransactionException.TYPE_3_TX_MAX_BLOB_GAS_ALLOWANCE_EXCEEDED: (
+            r"tx causes total blob gas of \d+ to exceed maximum blob gas per block of \d+|"
+            r"tx can contain at most \d+ blobs"
+        ),
+        TransactionException.TYPE_3_TX_BLOB_COUNT_EXCEEDED: (
+            r"tx causes total blob gas of \d+ to exceed maximum blob gas per block of \d+|"
+            r"tx can contain at most \d+ blobs"
+        ),
+        TransactionException.TYPE_3_TX_PRE_FORK: (
+            r"blob tx used but field env.ExcessBlobGas missing|EIP-4844 not enabled on Common"
+        ),
+        BlockException.BLOB_GAS_USED_ABOVE_LIMIT: r"invalid blobGasUsed expected=\d+ actual=\d+",
+        BlockException.INCORRECT_BLOB_GAS_USED: r"invalid blobGasUsed expected=\d+ actual=\d+",
+        BlockException.INVALID_BLOCK_HASH: (
+            r"Invalid blockHash, expected: 0x[0-9a-f]+, received: 0x[0-9a-f]+"
+        ),
+        BlockException.INVALID_REQUESTS: r"Unknown request identifier|invalid requestshash",
+    }