Skip to content

Letná (v4.0.0)

Choose a tag to compare

View all tags
@github-actions github-actions released this 17 Feb 13:05
· 715 commits to main since this release
v4.0.0
2e1d3a0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Summary

EESTs Letná release contains many framework improvements, new commands and additional tests that focus on the next hard forks Prague and Osaka (EOF). Please read the breaking changes!

Improved User & Test Developer Experience

Astral python tooling is now utilized within the the repo inclusive of uv and ruff:

  • uv is a python package and project manger primarily replacing our pip entry point for repo setup and installation.
  • ruff is an extremely fast python linter and code formatter, that now replaces the black, isort & flake8 within the repo.

EEST installation and usage is now significantly faster, although requires every command to be pre-pended with uv run <command>, more information can be found within the updated documentation.

EELS Transition Tool

As we move towards a world where EIPs are developed within EELS for hard fork inclusion, EEST now utilizes the EELS transition tool as the default fixture filler. The EELS transition tool, along with the EELS resolver, are included as packages within EEST, allowing repository users to fill tests out of the box upon repo installation. Previously, users were required to install an additional client’s transition tool as a prerequisite for filling the latest tests.

Support for filling tests in EEST using other transition tools will remain available, although the primary focus will now be directed towards EELS. Currently, EELS serves as the main filler for all forks up to Prague. For Osaka (EOF), tests are being filled with evmone - with a nearly complete working version in EELS, while Stateless is being filled with geth - with a movement towards EELS.

Execute Command

A new command called execute is now introduced that allows existing python tests (not fixtures generated by fill) to be ran within live networks including devnets. execute collects transactions written within existing python tests, sends them to a client connected to a network, wait for the network to include them in a block and, finally, check the resulting state of the involved smart-contracts against the expected state to validate the behavior of the clients. This is a new way of testing that was not possible until now. Please check the documentation for more details.

Consume Command Enhancements

The Petřín release introduced the consume command to provide a unified way to run test fixtures against clients directly within EEST. In this release, we have added required fixes and tweaks to consume, and now fixture .tar releases can be ran directly using the syntax --input <RELEASE_NAME>@<RELEASE_VERSION>, making it easier to run any set of fixture releases.

An additional sub-command, consume cache, has been added to allow fixture releases to be downloaded with a single command. When running consume within hive, this enhancement enables Docker to download fixture releases as a cached step, thereby improving run time and overall user experience.

EEST Contributions

We are pleased to announce that @felix314159 has joined EEST as an active full-time maintainer, joining the ranks of previous maintainers @marioevz, @danceratopz, @winsvega, and @spencer-tb.

Since the last release, many members of the Ethereum community have contributed significantly to EEST. A special thanks goes to the @ipsilon team for their efforts on EOF, to @jsign for contributions to Stateless, and to @raxhvl for improvements to the framework and documentation. We also extend our thanks to the @snake-charmers for their ongoing support and collaboration for their test porting efforts from ethereum/tests.

What's Changed

📁 Fixture Releases

  • 🔀 Initially we moved old fork configured tests within stable and develop fixture releases to a separate legacy release (#788).
  • 🔀 This was later reverted after some client teams preferred to keep them in all in the same releases (#1053).

💥 Breaking Change

  • ✨ Use uv for package management replacing pip (#777).
  • ✨ Ruff now replaces Flake8, Isort and Black resulting in significant changes to the entire code base including its usage (#922).
  • 🔀 Fill test fixtures using EELS by default. EEST now uses the ethereum-specs-evm-resolver with the EELS daemon (#792).
  • 🔀 The EOF fixture format contained in eof_tests may now contain multiple exceptions in the "exception" field in the form of a pipe (|) separated string (#759).
  • 🔀 state_test, blockchain_test and blockchain_test_engine fixtures now contain a config field, which contains an object that contains a blobSchedule field. On the blockchain_test and blockchain_test_engine fixtures, the object also contains a duplicate of the network root field. The root's network field will be eventually deprecated (#1040).
  • 🔀 latest-stable-release and latest-develop-release keywords for the --input flag in consume commands have been replaced with stable@latest and develop@latest respectively (#1044).

🛠️ Framework

  • ✨ Execute command added to run existing tests in live networks (#).
  • 🐞 Fixed consume hive commands from spawning different hive test suites during the same test execution when using xdist (#712).
  • ✨ consume hive command is now available to run all types of hive tests (#712).
  • ✨ Generated fixtures now contain the test index index.json by default (#716).
  • ✨ A metadata folder .meta/ now stores all fixture metadata files by default (#721).
  • 🐞 Fixed fill command index generation issue due to concurrency (#725).
  • ✨ Added with_all_evm_code_types, with_all_call_opcodes and with_all_create_opcodes markers, which allow automatic parametrization of tests to EOF (#610, #739).
  • ✨ Added with_all_system_contracts marker, which helps parametrize tests with all contracts that affect the chain on a system level (#739).
  • ✨ Code generators Conditional and Switch now support EOF by adding parameter evm_code_type (#610).
  • ✨ fill command now supports parameter --evm-code-type that can be (currently) set to legacy or eof_v1 to force all test smart contracts to deployed in normal or in EOF containers (#610).
  • 🐞 Fixed fixture index generation on EOF tests (#728).
  • 🐞 Fixes consume genesis mismatch exception for hive based simulators (#734).
  • ✨ Adds reproducible consume commands to hiveview (#717).
  • 💥 Added multiple exceptions to the EOF fixture format (#759).
  • ✨ Added optional parameter to all with_all_* markers to specify a lambda function that filters the parametrized values (#739).
  • ✨ Added extend_with_defaults utility function, which helps extend test case parameter sets with default values. @pytest.mark.parametrize (#739).
  • ✨ Added Container.Init to ethereum_test_types.EOF.V1 package, which allows generation of an EOF init container more easily (#739).
  • ✨ Introduce method valid_opcodes() to the fork class (#748).
  • 🐞 Fixed consume exit code return values, ensuring that pytest's return value is correctly propagated to the shell. This allows the shell to accurately reflect the test results (e.g., failures) based on the pytest exit code (#765).
  • ✨ Added a new flag --solc-version to the fill command, which allows the user to specify the version of the Solidity compiler to use when compiling Yul source code; this version will now be automatically downloaded by fill via solc-select (#772).
  • 🐞 Fix usage of multiple @pytest.mark.with_all* markers which shared parameters, such as with_all_call_opcodes and with_all_create_opcodes which shared evm_code_type, and now only parametrize compatible values (#762).
  • ✨ Added selector and marks fields to all @pytest.mark.with_all* markers, which allows passing lambda functions to select or mark specific parametrized values (see documentation for more information) (#762).
  • ✨ Improves consume input flags for develop and stable fixture releases, fixes --help flag for consume (#745).
  • 🔀 Return exit-code from consume commands (#766).
  • 🔀 Remove duplicate EOF container tests, automatically check for duplicates (#800).
  • 🔀 Fix DATALOAD pushed_stack_items calculation (#784).
  • ✨ Add evm_bytes rename and print asm (#844).
  • 🔀 Use the session_temp_folder introduced in #824 (#845).
  • 🐞 Don't treat eels resolutions as a fixture (#878).
  • ✨ Emphasize that no tests were executed during a fill pytest session (#887).
  • 🔀 Move generic code from TransitionTool to a new generic base class EthereumCLI (#894).
  • 🐞 Fix erroneous fork mes- 🔀 Fix max stack height calculation (#810).
  • ✨ Add storage key hint in Storage class (#917).
  • ✨ Add system to verify exception strings (#795).
  • 🔀 Fix FixedBytes assignment rules (#1010).
  • 🔀 Fix Address padding options (#1113).
  • 🔀 Add Container.expected_bytecode optional parameter (#737).
  • ✨ Add support for initcode_prefix on EOF Container.Init (#819).sage in pytest session header with development forks (#806).
  • 🐞 Fix Conditional code generator in EOF mode (#821).
  • Ensure that Block objects keep track of their fork, for example, when the block's rlp_modifier is not None (#854).
  • 🔀 ethereum_test_rpc library has been created with what was previously ethereum_test_tools.rpc (#822).
  • ✨ Add Wei type to ethereum_test_base_types which allows parsing wei amounts from strings like "1 ether", "1000 wei", "10**2 gwei", etc (#825).
  • ✨ Pin EELS versions in eels_resolutions.json and include this file in fixture releases (#872).
  • 🔀 Replace ethereum.base_types with ethereum-types (#850).
  • 💥 Rename the PragueEIP7692 fork to Osaka (#869).
  • ✨ Improve fill terminal output to emphasize that filling tests is not actually testing a client (#807).
  • ✨ Add the BlockchainTestEngine test spec type that only generates a fixture in the EngineFixture (blockchain_test_engine) format (#888).
  • 🔀 Move the evm_transition_tool package to ethereum_clis and derive the transition tool CL interfaces from a shared EthereumCLI class that can be reused for other sub-commands (#894).
  • ✨ Pass state_test property to T8N tools that support it (Only EELS at the time of merge) (#943).
  • ✨ Add the eofwrap cli used to wrap tests from ethereum/tests in an EOF container (#896).
  • 🔀 Improve gentest architecture with EthereumTestBaseModel and EthereumTestRootModel (#901).
  • 🔀 Migrate transaction test to state_test for gentest (#903).
  • 🔀 ethereum_test_forks forks now contain gas-calculating functions, which return the appropriate function to calculate the gas used by a transaction or memory function for the given fork (#779).
  • 🐞 Fix Bytecode class __eq__ method (#939).
  • 🔀 Update pydantic from 2.8.2 to 2.9.2 (#960).
  • ✨ Add the eest make test command, an interactive CLI that helps users create a new test module and function (#950).
  • ✨ Add the eest clean command that helps delete generated files and directories from the repository (#980).
  • ✨ Add framework changes for EIP-7742, required for Prague devnet-5 (#931).
  • ✨ Add the eest make env command that generates a default env file (env.yaml)(#996).
  • ✨ Generate Transaction Test type (#933).
  • ✨ Add a default location for evm logs (--evm-dump-dir) when filling tests (#999).
  • ✨ Slow tests now have greater timeout when making a request to the T8N server (#1037).
  • ✨ Introduce pytest.mark.parametrize_by_fork helper marker (#1019, #1057).
  • 🐞 fix(consume): allow absolute paths with --evm-bin (#1052).
  • ✨ Disable EIP-7742 framework changes for Prague (#1023).
  • ✨ Allow verification of the transaction receipt on executed test transactions (#1068).
  • ✨ Modify valid_at_transition_to marker to add keyword arguments subsequent_transitions and until to fill a test using multiple transition forks (#1081).
  • 🐞 fix(consume): use "HIVE_CHECK_LIVE_PORT" to signal hive to wait for port 8551 (Engine API port) instead of the 8545 port when running consume engine (#1095).
  • ✨ state_test, blockchain_test and blockchain_test_engine fixtures now contain the blobSchedule from EIP-7840, only for tests filled for Cancun and Prague forks (#1040).
  • 🔀 Change --dist flag to the default value, load, for better parallelism handling during test filling (#1118).
  • 🔀 Refactor framework code to use the ethereum-rlp package instead of ethereum.rlp, previously available in ethereum/execution-specs (#1180).
  • 🔀 Update EELS / execution-specs EEST dependency to 99238233 for EEST framework libraries and test case generation (#1181).
  • ✨ Add the consume cache command to cache fixtures before running consume commands (#1044).
  • ✨ The --input flag of the consume commands now supports parsing of tagged release names in the format <RELEASE_NAME>@<RELEASE_VERSION> (#1044).
  • 🐞 Fix stdout output when using the fill command (#1188).
  • ✨ Add tests for blockchain intermediate state verification (#1075).
  • ✨ Add Interactive CLI input functionality (#947).
  • 🔀 Rename EOFTest.data to EOFTest.container with rebase of EOFStateTest (#1145).
  • ✨ Turn on --traces for EELS + ethereum-specs-evm-resolver (#1174).

📋 Misc

  • ✨ Feature releases can now include multiple types of fixture tarball files from different releases that start with the same prefix (#736).
  • ✨ Releases for feature eip7692 now include both Cancun and Prague based tests in the same release, in files fixtures_eip7692.tar.gz and fixtures_eip7692-prague.tar.gz respectively (#743).
    ✨ Re-write the test case reference doc flow as a pytest plugin and add pages for test functions with a table providing an overview of their parametrized test cases (#801, #842).
  • 🔀 Simplify Python project configuration and consolidate it into pyproject.toml (#764).
  • ✨ Add dev docs to help using nectos/act (#776).
  • 🔀 Update uv.lock for updated solc deps (#782).
  • ✨ Enable coverage on any test change (#790).
  • 🔀 Created pytest_plugins.concurrency plugin to sync multiple xdist processes without using a command flag to specify the temporary working folder (#824)
  • 🔀 Move pytest plugin pytest_plugins.filler.solc to pytest_plugins.solc.solc (#823).
  • 🔀 Remove formats.py, embed properties as class vars (#826).
  • ✨ Add build-evm-base to docs deploy workflows (#829).
  • 🔀 Add links to the online test case docs in the EOF tracker (#838).
  • 🔀 Fix miscellaneous improvements to troubleshooting, navigation, styling (#840).
  • ✨ Include all parameters in test parameter datatables (#842).
  • ✨ Add info about ripemd160 & update running actions locally (#847).
  • ✨ Add SECURITY.md describing how to report vulnerabilities (#848).
  • 🔀 Change image from ubuntu-24.04 to ubuntu-latest in CI (#855).
  • 🐞 Asserts that the deploy docs tags workflow is only triggered for full releases (#857).
  • 🐞 Fix deploy docs tags workflow trigger (#858).
  • ✨ A new application-wide configuration manager provides access to environment and application configurations. (#892).
  • 🔀 Update the developer docs navigation (#898).
  • 🔀 Use jinja2 templating in gentest (#900).
  • ✨ Fix/add test github actions locally page (#909).
  • 🐞 Fix print fill output in coverage workflow on errors (#919).
  • 🐞 Use a local version of ethereum/execution-specs (EELS) when running the framework tests in CI (#997).
  • ✨ Use self-hosted runners for fixture building in CI (#1051).
  • ✨ Release tarballs now contain fixtures filled for all forks, not only the fork under active development and the fork currently deployed on mainnet (#1053).
  • ✨ StateTest fixture format now contains state field in each network post result, containing the decoded post allocation that results from the transaction execution (#1064).
  • ✨ Include EELS fork resolution information in filled json test fixtures (#1123).
  • 🔀 Updates ruff from version 0.8.2 to 0.9.4 (#1168).
  • 🔀 Update uv.lock to be compatible with uv>=0.5.22 (#1178).

🧪 Test Cases

  • ✨ Migrate validation tests EIP3540/validInvalidFiller.yml (#598).
  • ✨ EIP-4844 test tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py includes an EOF test case (#610).
  • ✨ Example test tests/frontier/opcodes/test_dup.py now includes EOF parametrization (#610).
  • ✨ Add EOFv1 function test - Call simple contract test (#695).
  • ✨ Add section size validation tests (#705).
  • ✨ Add deep and wide EOF subcontainers tests (#718).
  • ✨ Update EIP-7702 tests for Devnet-3 (#733)
  • ✨ Migrate "valid" EOFCREATE validation (#738).
  • ✨ Migrate tests for truncated sections (#740).
  • ✨ Add out of order container section test (#741).
  • ✨ Convert all opcodes validation test tests/frontier/opcodes/test_all_opcodes.py (#748).
  • 🔀 Add Test types with 128 inputs (#749).
  • 🔀 Use new marker to EOF-ize MCOPY test (#754).
  • ✨ Add EOF Tests from Fuzzing (#756).
  • ✨ Add embedded container tests (#763).
  • ✨ Validate EOF only opcodes are invalid in legacy (#768).
  • ✨ Update EOF tracker, add unimplemented tests (#773).
  • ✨ Add EIP-7620 EOFCREATE gas tests (#785).
  • ✨ Add more fuzzing discovered EOF tests (#789).
  • ✨ Add EOF tests for invalid non-returning sections (#794).
  • ✨ Test to ensure transient storage is cleared after transactions (#798).
  • ✨ Test that transient storage stays at correct address (#799).
  • ✨ Add EOFCREATE referencing the same subcontainer twice test (#809).
  • ✨ Add dangling data in subcontainer test (#812).
  • 🐞 Fix EIP-7702+EOF tests due to incorrect test expectations and faulty Conditional test generator in EOF mode (#821)
  • 🐞 Fix TSTORE EOF variant test (#831).
  • 🔀 Unify EOF return code constants (#834).
  • ✨ Add RJUMP* vs CALLF tests (#833).
  • ✨ Add tests to clarify "non-returning instruction" (#837).
  • ✨ Add double RJUMPI stack validation tests (#851).
  • ✨ Add unreachable code sections tests (#856).
  • 💥 PragueEIP7692 fork in tests has been updated to Osaka (#869)
  • ✨ Update EIP-6110, EIP-7002, EIP-7251, EIP-7685, and EIP-7702 tests for Devnet-4 (#832)
  • ✨ Add EIP-7069 and EIP-7620 failures and context vars tests (#836).
  • ✨ Add EOF EIP-4750 Stack validation in CALLF test (#889).
  • ✨ Add stack overflow by rule check to JUMPF tests (#902).
  • 🐞 Fix erroneous test withCALLF rule bug (#907).
  • ✨ Add test for EXTDELEGATECALL value cost (#911).
  • ✨ Add basic EOF execution tests (#912).
  • ✨ Add parametrized CALLF execution tests (#913).
  • ✨ Add CALLF execution tests (#914).
  • ✨ Add fibonacci and factorial CALLF tests (#915).
  • ✨ Add RJUMP* execution tests (#916).
  • ✨ EIP-7702 many delegations test (#923)
  • ✨ Add opcode validation tests (#932).
  • ✨ Add RJUMPI with JUMPF tests (#928).
  • ✨ EIP-7702 set code of non-empty-storage account test (#948)
  • ✨ Add PUSH* opcode tests (#975).
  • ✨ EIP-7702 implement 7702 test ideas (#981)
  • ✨ EIP-7702 Remove delegation behavior of EXTCODE* (#984)
  • ✨ Add EIP-7620 RETURNCONTRACT behavior verification test (#1109).
  • ✨ Add EIP-7069 p256verify EOF calls tests (#1021).
  • ✨ Add EIP-7480 DATACOPY edge cases tests (#1020).
  • ✨ Add EIP-7069 EXTCALL creation gas charge tests (#1025).
  • ✨ Add generic precompile-absence test (#1036)
  • ✨ Add test for EIP-2537 which uses the full discount table of G2 MSM (#1038)
  • ✨ EIP-7691 Blob throughput increase tests by parametrization of existing EIP-4844 tests (#1023, #1082)
  • ✨ Port calldatacopy test (#1056).
  • ✨ EIP-7623 Increase calldata cost (#1004, #1071)
  • ✨ Add CALLF invalid section index tests (#1111).
  • ✨ Add JUMPF invalid section index tests (#1112).
  • ✨ Add CALLF truncated immediate bytes tests (#1114).
  • ✨ EIP-152 Add tests for Blake2 compression function F precompile (#1067)
  • ✨ Add CALLF non-returning section tests (#1126).
  • ✨ Add DATALOADN truncated immediate bytes tests (#1127).
  • 🔀 Update EIP-7702 test expectations according to spec updates (#1129)
  • ✨ Add tests for CALLF and non-returning (#1140).
  • 🔀 Update EIP-7251 according to spec updates #9127, #9289 (#1024, #1155).
  • 🔀 Update EIP-7002 according to spec updates #9119, #9288 (#1024, #1155).
  • 🔀 Update EIP-2935 according to spec updates #9144, #9287 (#1046, #1155)
  • ✨ Add DATALOADN validation and execution tests (#1162).
  • ✨ Add EOF prefix tests (#1187).
  • ✨ Add tests for EOF code header missing (#1193).
  • ✨ Add tests for empty EOF type section (#1194).
  • ✨ Add tests for multiple EOF type sections (#1195).
  • ✨ Add EIP-7698 legacy EOF creation prevention tests (#1206).

New Contributors

Full Changelog: v3.0.0...v4.0.0

Contributors

reedsa, pk910, and 19 other contributors
Assets 5
Loading