v1.0.1

🐍 Cancun Devnet-8 Pre-Release

This pre-release holds our initial set of tests for the current Devnet-8 Cancun specification.

🧪 Tests include the following EIPs

EIP-4844: Shard Blob Transactions
EIP-5656: MCOPY - Memory copying instruction
EIP-6780: SELFDESTRUCT only in same transaction
EIP-4788: Beacon block root in the EVM

Our full release will include tests for EIP-1153: Transient storage opcodes including additional coverage for the above EIPs.

⚠️ Note
Beacon block root pre-compile tests are to be updated following its change to a system call:
https://ethereum-magicians.org/t/eip-4788-beacon-root-in-evm/8281/33

⚙️ Test Generation

Fixtures for these tests can be generated on main with the following geth branch:
spencer-tb/go-ethereum@4844-devnet-6-t8n-update

fill --from Cancun --until Cancun

📄 Documentation

Further information on each test can be found within the docs:
https://ethereum.github.io/execution-spec-tests/main/tests/cancun/

v1.0.0

🧪 Welcome to the PyTest Era

We are extremely happy to announce this major new release as it marks a huge milestone for execution-spec-tests.

With this release, adding tests has become easier than ever, so we invite and encourage all client developers to seize this opportunity and contribute to the continuous enhancement of the ethereum testing ecosystem.

🗝️ Key Changes

• Pytest takes the stage: Introducing the upgrade to pytest, now used to collect and execute tests, including generation of fixtures.
• Shiny new documentation: A full revamp of the docs which is now available at ethereum.github.io/execution-spec-tests.
• Tests by fork: Tests (previously named fillers) and fixtures, are now re-organized by fork name for improved clarity and usage.
• Commands with comfort: Tests are now filled with a new user friendly and readable CLI, including a new additional option to specify the solc binary.
• Write tests like a pro: the repo now includes recommended extensions & settings for a seamless new developer start.

Full Changelog: v0.2.5...v1.0.0

🐍 Tests

Updates to Existing Tests

• fixtures.tar.gz includes a refresh of previous State & Blockchain tests, up to and including Shanghai:
  • Each test has been refactored and now utilizes Pytest parameterization. These can be viewed within the test case reference section of the docs.

New EIP-4844 Tests

• fixtures-cancun.tar.gz includes a full set of Blockchain tests for EIP-4844, up to date with the latest Devnet 6 & 7 changes

🛠️ Usage

All tests can be filled with the new CLI using the fill command. (Note: tf is deprecated and will be removed in the next release).

By default fill generates fixtures for all tests up to the current mainnet fork - Shanghai. Detailed information on the new CLI can be found within the docs.

Generating EIP-4844 Tests

To only generate the new EIP-4844 tests the following command can be run:

fill -v tests/cancun --until=Cancun

Versions Used

Main fixtures: ethereum/go-ethereum @ c7b099b go-ethereum branch.

Cancun fixtures: marioevz/go-ethereum @ d36b63d go-ethereum branch.

Additional dependency versions used (others may work):

• Solidity @ 0.8.20
• Python @ 3.10.6

🐝 Pyspec Hive Simulator

Each EIP-4844 test case within fixtures-cancun.tar.gz can additionally be run within the updated pyspec hive simulator. It sends each block within a test fixture as a payload to the execution client using the Engine API.

💬 Contact Us

We are here to help!

If you stumble over any issues with this release please reach out to us.

You can find our contact details within the docs!

v0.2.5

Tests

Cancun

• Separate fixtures to test for Cancun: includes the first version of the 4844 fixtures. (#63)
• These currently test for valid/invalid blob transactions, excess data gas calculations and the DATAHASH opcode.
• Additional tests for kzg pre-compiles and ssz will be added in a future release.
• Test documentation can be found -> here.
• These will be run within the hive pyspec simulator on this branch.

Important: These tests can only be filled using the geth version listed below, alongside the --latest-fork Cancun flag. As an example, you can run the 4844 tests with the following command: tf --force-refill --latest-fork Cancun --test-categories eips.eip4844.

Pre-Cancun

• New filler category added for security tests, used to verify previously identified client vulnerabilities. (#112)
• Access list test case example filler added. (#83)

Changes

• Reference spec feature added to fillers & fixtures. (#81)
• Fork object feature added to fillers. (#91)
  • Can now write tests for fork transitions.
• Callable Op-codes feature added reducing the dependency of Yul code (#102)
  • Most filler modules are now entirely python.
• Access list type support added. (#83)
• Documentation updated to include 4884 test descriptions -> here.

Fixes

• Small bug fixes on the fixture format (#80 #92 #109)
• Correctly handle the processing of negative integer indexes in storage (#106)

Versions Used

go-ethereum: mdehoog/go-ethereum@ac64c44

v0.2.4

Changes

• Additional checks added to the Tox pipeline. Python file and function names now checked in accordance with PEP8. (PR: #59)
• Naming style structure changed in the fixture format. withdrawals_ head removed from name of all withdrawals fixtures. Fixture test case names now like 000/shanghai instead of 000_shanghai. Tags added to specific test case names i.e. 001/multiple_blocks/shanghai. (PR: #69)
• Updates and additions to the test filler tf: (PRs: #66 & #72)
  • Refactor of ethereum_test_filling_tool.
  • Default tf execution is now concurrent/parallel. So far ~65% increase in speed which will scale as more tests are added.
  • Filling is skipped for tests that don't need to be filled by default. Improvement to tf execution speed.
  • tf --benchmark. Adds timing of filling to the console logs.
  • tf --max-workers. Allows the user to set the number of workers used for concurrent tf execution. For serial execution (the previous way) use tf --max-workers 1.
  • tf --force-refill. Force tests to be filled. Works with --test-module and --test-categories.
  • tf --no-output-structure. When enabled all tests are added to a single equivalent output folder.

Versions Used

go-ethereum: https://github.com/ethereum/go-ethereum/tree/v1.11.3

v0.2.3

Changes

• Remove withdrawals overflow test (PR: #68)

Fixes

• Fixture format now better matches original BlockchainTest JSON format as produced by ethereum/retesteth. (PR: #47)

Versions Used

go-ethereum: https://github.com/ethereum/go-ethereum/tree/v1.11.2

v0.2.2

Changes

• Test for withdrawals with a large gwei amount (PR: #44)

Fixes

• All fixtures now have withdrawals in their candidate block RLP. (PR: #37)

Versions Used

go-ethereum: lightclient/go-ethereum@b49a883

v0.2.1

Changes

• Test fixtures now include genesisRLP field (PR: #35)

Fixes

• Post-Merge test vectors now use genesis.difficulty=0 (PR: #35)

Versions Used

go-ethereum: lightclient/go-ethereum@b49a883

v0.2.0

Changes

• EIP-3860: Changes failure mode to Out-of-Gas when calling CREATE/CREATE2 when using initcode that exceeds MAX_INITCODE_SIZE: ethereum/EIPs#6249
• EIP-4895: Updates the withdrawal amount to Gwei instead of Wei: ethereum/EIPs#6325
• Fillers/Fixtures: Merged network is renamed to Merge

Versions Used

go-ethereum: lightclient/go-ethereum@6ab6d7388

v0.1.0

Contains Shanghai targeted tests:

• EIP-3651: Warm COINBASE
• EIP-3855: PUSH0 instruction
• EIP-3860: Limit and meter initcode
• EIP-4895: Beacon chain push withdrawals as operations

Fixtures generated using multiple go-ethereum commits:

• lightclient/go-ethereum@955c423
• marioevz/go-ethereum@0e83633