Releases · ethereum/execution-spec-tests

27 Aug 19:27

1d040a0

[email protected] Pre-release

Pre-release

Third release for Pectra Devnet-3, filled again using EthereumJS transition tool implementation which includes fixes to the state root expected in state tests (thanks again @jochem-brouwer!).

Fixes

In previous release, in the state tests, the system contracts' addresses were touched even though the contracts were not present in the pre-alloc.

Important Notes

EIP-2935 slow tests (256+ blocks) have been skipped.

Included EIPs

Missing EIPs

EIP-7692: EVM Object Format (EOFv1) Meta

Full Changelog: https://github.com/ethereum/execution-spec-tests/compare/[email protected]@v1.2.0

Contributors

jochem-brouwer

Assets 3

26 Aug 14:29

github-actions

[email protected]

afe5dd6

[email protected] Pre-release

Pre-release

⚠️ Note these tests are up to date with the latest devnet-7 spec!

This release is equivalent to [email protected] but filled with the latest geth gballet/kaustinen-with-shapella branch aligned with devnet-7.

From the VIC no 23 testing slides, a reasonable order of steps for client teams to start running tests can be found below, starting with the genesis tests from: fixtures_verkle-genesis.tar.gz

Run EIP-6800 tests (~23) for a good first check, fixtures/verkle/eip6800_genesis_verkle_tree/ within the extracted the fixture genesis tar.
Run "backfilled" tests (~300), all tests excludingfixtures/verkle/*.
Wait for the next release to run additional tests after framework witness assertions verify the remaining test cases.

Full Changelog: https://github.com/ethereum/execution-spec-tests/compare/[email protected]@v0.0.3

🌪️ Fixture Format Changes

All fixtures now contain a block witness although currently without the parent state root.

class Witness(CamelModel):
    state_diff: StateDiff
    verkle_proof: VerkleProof

Client test consumers can now utilize this to compare there computed block witness against the witness contained within the fixtures (computed from geth's t8n).

For more information on our witness definition please adhere to src/ethereum_test_types/verkle/types.py.

Additionally the post state is removed. Future fixture releases will contain the post state as a VKT.

🐘 Verkle Genesis Test Fixtures

Contains verkle specific test vectors from #659 including all existing EEST test cases filled for a verkle configured fork. Note these tests assume the MPT to VKT conversion has completed where we start at the Verkle fork.

Please use fixtures_verkle-genesis.tar.gz!

Generating Genesis Fixtures

Using the geth evm binary from this commit, fill with the following command:

fill --fork Verkle --evm-bin=<path_to_geth_evm> -n auto -m blockchain_test

🔁 Verkle Conversion Test Fixtures - 0 Stride

Contains an improvement to the initial set of transition tests.

These aim to verify a basic fork transition from Shanghai to Verkle. 0 stride denotes that the initial MPT remains frozen. Thus the MPT is not being converted to a VKT within these tests. The intention is to check that only blocks after the transition update the VKT, isolating VKT fork transition issues without touching MPT stride conversion logic.

Test cases additionally include Shanghai genesis tests to assert that the fork before Verkle is not broken.

The next release will contain conversion tests with some stride enabled to dynamically validate the MPT conversion.

Please use fixtures_verkle-conversion-stride-0.tar.gz!

Generating Conversion Fixtures

Using the geth evm binary from this commit, fill with the following command:

fill --from Shanghai --until EIP6800Transition --evm-bin=<path_to_geth_evm> -n auto -m blockchain_test

Assets 4

23 Aug 17:49

github-actions

[email protected]

7a1777e

[email protected] Pre-release

Pre-release

Second release for Pectra Devnet-3, filled again using EthereumJS transition tool implementation which included gas accounting fixes (thanks again @jochem-brouwer!).

Fixes

Added re-authorization gas costs tests, i.e. existing set-code in account receives a new delegation (c27e010)
Fixed warm account costs to consider cost to access the delegated-to account (59ec505)

Important Notes

EIP-2935 slow tests (256+ blocks) have been skipped.

Included EIPs

Missing EIPs

EIP-7692: EVM Object Format (EOFv1) Meta

Full Changelog: https://github.com/ethereum/execution-spec-tests/compare/[email protected]@v1.1.0

Contributors

jochem-brouwer

Assets 3

21 Aug 17:01

github-actions

[email protected]

0711643

[email protected] Pre-release

Pre-release

First release for Pectra Devnet-3, filled using EthereumJS transition tool implementation (thanks @jochem-brouwer!).

Important Notes

EIP-2537 tests have been skipped for this release but will be available in a later release.
EIP-2935 slow tests (256+ blocks) have been skipped too.

Included EIPs

~~EIP-2537: Precompile for BLS12-381 curve operations~~ TESTS SKIPPED ONLY FOR THIS RELEASE, EIP STILL ENABLED IN DEVNET-3
EIP-2935: Save historical block hashes in state
EIP-6110: Supply validator deposits on chain
EIP-7002: Execution layer triggerable exits
EIP-7251: Increase the MAX_EFFECTIVE_BALANCE
EIP-7685: General purpose execution layer requests
EIP-7702: Set EOA account code for one transaction

Missing EIPs

EIP-7692: EVM Object Format (EOFv1) Meta

Full Changelog: https://github.com/ethereum/execution-spec-tests/compare/[email protected]

Contributors

jochem-brouwer

Assets 3

15 Aug 16:17

github-actions

[email protected]

0e9b3d9

[email protected] Pre-release

Pre-release

What's Changed

new(tests): EOF - EIP-3540: Expand section size testing by @pdobacz in #705
new(tests) Deep and wide EOF subcontainers by @shemnon in #718
fix(docs): Add more cases to EOF tracker by @gumb0 in #723
new(tests): EOF - EIP-7069: Add tests by @pdobacz in #722
fix(fixtures): Fix index generation for EOF tests by @marioevz in #728
fix(docs): Add some execution cases to EOF test tracker by @gumb0 in #727
feat(fw,forks,tests): Add EVM code type marker by @marioevz in #610
new(tests): EOF - EIP-7069: Add tests, part 2. by @pdobacz in #730
feat(fw): add optional Container.expected_bytecode by @chfast in #737
new(tests): migrate "valid" EOFCREATE validation by @chfast in #738
fix(docs): Add stack validation cases to EOF tracker by @gumb0 in #735
new(tests): EOF - EIP-3540: migrate tests for truncated sections by @chfast in #740
feat(plugins,forks,github): Allow dual-feature (Prague+Cancun) build on EOF releases by @marioevz in #743
fix(docs): Add execution cases from evmone-generated tests to EOF tracker by @gumb0 in #742

Full Changelog: https://github.com/ethereum/execution-spec-tests/compare/[email protected]

Contributors

shemnon, chfast, and 3 other contributors

Assets 4

10 Aug 19:21

github-actions

[email protected]

4a3f829

[email protected] Pre-release

Pre-release

⚠️ Note these tests are up to date with the devnet-6 spec!

This release is equivalent to [email protected] but with the addition of Shanghai genesis tests within the conversion fixture set.

Full Changelog: https://github.com/ethereum/execution-spec-tests/compare/[email protected]@v0.0.2

🌪️ Fixture Format Changes

All fixtures now contain a block witness although currently without the parent state root.

class Witness(CamelModel):
    state_diff: StateDiff
    verkle_proof: VerkleProof

Client test consumers can now utilize this to compare there computed block witness against the witness present within the fixtures (computed from geth's t8n).

For more information on our witness definition please adhere to src/ethereum_test_types/verkle/types.py.

Additionally the post state is removed. Future fixture releases will contain the post state as a VKT.

🐘 Verkle Genesis Test Fixtures

Please use fixtures_verkle-genesis.tar.gz!

Generating Genesis Fixtures

Using the geth evm binary from this commit, fill with the following command:

fill --fork Verkle --evm-bin=<path_to_geth_evm>/evm -n auto -m blockchain_test

🔁 Verkle Conversion Test Fixtures - 0 Stride

Contains an improvement to the initial set of transition tests.

Test cases additionally include Shanghai genesis tests to assert that the fork before Verkle is not broken.

The next release will contain conversion tests with some stride enabled to dynamically validate the MPT conversion.

Please use fixtures_verkle-conversion-stride-0.tar.gz!

Generating Conversion Fixtures

Using the geth evm binary from this commit, fill with the following command:

fill --from Shanghai --until EIP6800Transition --evm-bin=<path_to_geth_evm>/evm -n auto -m blockchain_test

Assets 4

08 Aug 23:32

github-actions

[email protected]

fba0400

[email protected] Pre-release

Pre-release

⚠️ Note these tests are up to date with the devnet-6 spec!

🌪️ Fixture Format Changes

All fixtures now contain a block witness although currently without the parent state root.

class Witness(CamelModel):
    state_diff: StateDiff
    verkle_proof: VerkleProof

Client test consumers can now utilize this to compare there computed block witness against the witness present within the fixtures (computed from geth's t8n).

For more information on our witness definition please adhere to src/ethereum_test_types/verkle/types.py.

Additionally the post state is removed. Future fixture releases will contain the post state as a VKT.

🐘 Verkle Genesis Test Fixtures

Please use fixtures_verkle-genesis.tar.gz!

Generating Genesis Fixtures

Using the geth evm binary from this commit, fill with the following command:

fill --fork Verkle --evm-bin=<path_to_geth_evm>/evm -n auto -m blockchain_test

🔁 Verkle Conversion Test Fixtures - 0 Stride

Contains an improvement to the initial set of transition tests.

The next release will contain conversion tests with some stride enabled to dynamically validate the MPT conversion.

Please use fixtures_verkle-conversion-stride-0.tar.gz!

Generating Conversion Fixtures

Using the geth evm binary from this commit, fill with the following command:

fill --fork EIP6800Transition --evm-bin=<path_to_geth_evm>/evm -n auto -m blockchain_test

Assets 4

23 Jul 23:43

github-actions

v3.0.0

1e1e3cf

Petřín (v3.0.0)

EEST's Petřín release adds many improvements and additions. Please read the breaking changes!

A noteable key package ethereum_test_tools is now fragmented into several fine grained packages that are suitable for use within other Python based repositories.

The consume simulator pytest plugin now contains 4 fixture runners:

consume direct: provides a pytest wrapper to execute multiple fixtures against client evm statetest/blocktest runners.
consume rlp: using hive as a back-end, executes fixture block rlps against fully instantiated clients, verifying the last block hash is expected.
consume engine: with a hive back-end, sends engine_newPayloadVX calls against fully instantiated clients validating each call response.
consume all: provides a wrapper surrounding all commands to execute consume direct, rlp and engine at once for the specified fixtures.

With a refined consume suite, testers can now fill and consume the generated fixtures extremely quickly to validate both the generated fixture and client implementation. This removes the cumbersome requirement of updating hive when verifying new fixtures.

To align the execution-apis engine specification with the consume engine plugin, the blockchain_test_hive fixture is renamed to blockchain_test_engine to align more with the sentiment of the spec. Similarly, the fixture format of the latter is changed to match the engine new payload "params" field defined in the spec.

💥 Breaking Changes

Cancun is now the latest deployed fork, and the development fork is now Prague (#489).
Stable fixtures artifact fixtures.tar.gz has been renamed to fixtures_stable.tar.gz (#573)
The "Blockchain Test Hive" fixture format has been renamed to "Blockchain Test Engine" and updated to more closely resemble the engine_newPayload format in the execution-apis specification (https://github.com/ethereum/execution-apis/blob/main/src/engine/prague.md#request) and now contains a single "params" field instead of multiple fields for each parameter (#687).
Output folder for fixtures has been renamed from "blockchain_tests_hive" to "blockchain_tests_engine" (#687).

🧪 Test Cases

✨ Add tests for eof container's section bytes position smart fuzzing (#592).
✨ Add test_create_selfdestruct_same_tx_increased_nonce which tests self-destructing a contract with a nonce > 1 (#478).
✨ Add test_double_kill and test_recreate which test resurrection of accounts killed with SELFDESTRUCT (#488).
✨ Add eof example valid invalid tests from ori, fetch EOF Container implementation (#535).
✨ Add tests for EIP-2537: Precompile for BLS12-381 curve operations (#499).
✨ EIP-663: Add test_dupn.py and test_swapn.py (#502).
✨ Add tests for EIP-6110: Supply validator deposits on chain (#530).
✨ Add tests for EIP-7002: Execution layer triggerable withdrawals (#530).
✨ Add tests for EIP-7685: General purpose execution layer requests (#530).
✨ Add tests for EIP-2935: Serve historical block hashes from state (#564, #585).
✨ Add tests for EIP-4200: EOF - Static relative jumps (#581, #666).
✨ Add tests for EIP-7069: EOF - Revamped CALL instructions (#595).
🐞 Fix typos in self-destruct collision test from erroneous pytest parametrization (#608).
✨ Add tests for EIP-3540: EOF - EVM Object Format v1 (#634, #668).
🔀 Update EIP-7002 tests to match spec changes in ethereum/execution-apis#549 (#600)
✨ Convert a few eip1153 tests from ethereum/tests repo into .py (#440).
✨ Add tests for EIP-7480: EOF - Data section access instructions (#518, #664).
✨ Add tests for subcontainer kind validation from EIP-7620: EOF Contract Creation for the cases with deeply nested containers and non-first code sections (#676).
✨ Add tests for runtime stack overflow at CALLF instruction from EIP-4750: EOF - Functions (#678).
✨ Add tests for runtime stack overflow at JUMPF instruction from EIP-6206: EOF - JUMPF and non-returning functions (#690).
✨ Add tests for EIP-7251: Increase the MAX_EFFECTIVE_BALANCE (#642)
✨ Add tests for Devnet-1 version of EIP-7702: Set EOA account code (#621)

🛠️ Framework

🐞 Fix incorrect != operator for FixedSizeBytes (#477).
✨ Add Macro enum that represents byte sequence of Op instructions (#457)
✨ Number of parameters used to call opcodes (to generate bytecode) is now checked (#492).
✨ Libraries have been refactored to use pydantic for type checking in most test types (#486, #501, #508).
✨ Opcodes are now subscriptable and it's used to define the data portion of the opcode: Op.PUSH1(1) == Op.PUSH1[1] == b"\x60\x01" (#513)
✨ Added EOF fixture format (#512).
✨ Verify filled EOF fixtures using evmone-eofparse during fill execution (#519).
✨ Added --traces support when running with Hyperledger Besu (#511).
✨ Use pytest's "short" traceback style (--tb=short) for failure summaries in the test report for more compact terminal output (#542).
✨ The fill command now generates HTML test reports with links to the JSON fixtures and debug information (#537).
✨ Add an Ethereum RPC client class for use with consume commands (#556).
✨ Add a "slow" pytest marker, in order to be able to limit the filled tests until release (#562).
✨ Add a CLI tool that generates blockchain tests as Python from a transaction hash (#470, #576).
✨ Add more Transaction and Block exceptions from existing ethereum/tests repo (#572).
✨ Add "description" and "url" fields containing test case documentation and a source code permalink to fixtures during fill and use them in consume-generated Hive test reports (#579).
✨ Add git workflow evmone coverage script for any new lines mentioned in converted_ethereum_tests.txt (#503).
✨ Add a new covariant marker with_all_contract_creating_tx_types that allows automatic parametrization of a test with all contract-creating transaction types at the current executing fork (#602).
✨ Tests are now encouraged to declare a pre: Alloc parameter to get the pre-allocation object for the test, and use pre.deploy_contract and pre.fund_eoa to deploy contracts and fu...