|
| 1 | +# Writing Tests |
| 2 | + |
| 3 | +## Purpose of test specs in this repository |
| 4 | + |
| 5 | +The goal of the test specs included in this repository is to generate test vectors that can be consumed by any Execution client, and to verify that all of the clients agree on the same output after executing each test. |
| 6 | + |
| 7 | +Consensus is the most important aspect of any blockchain network, therefore, anything that modifies the state of the blockchain must be tested by at least one test in this repository. |
| 8 | + |
| 9 | +The tests focus on the EVM execution, therefore before being able to properly write a test, it is important to understand what the Ethereum Virtual Machine is and how it works. |
| 10 | + |
| 11 | + |
| 12 | +## Types of tests |
| 13 | + |
| 14 | +At the moment there are only two types of tests that can be produced by each test spec: |
| 15 | + |
| 16 | +- State Tests |
| 17 | +- Blockchain Tests |
| 18 | + |
| 19 | +The State tests span a single block and, ideally, a single transaction. |
| 20 | + |
| 21 | +Examples of State tests: |
| 22 | + |
| 23 | +- Test a single opcode behavior |
| 24 | +- Verify opcode gas costs |
| 25 | +- Test interactions between multiple smart contracts |
| 26 | +- Test creation of smart contracts |
| 27 | + |
| 28 | +The Blockchain tests span multiple blocks which may or may not contain transactions and mainly focus on the block to block effects to the Ethereum state. |
| 29 | + |
| 30 | +- Verify system-level operations such as coinbase balance updates or withdrawals |
| 31 | +- Verify fork transitions |
| 32 | +- Verify blocks with invalid transactions/properties are rejected |
| 33 | + |
| 34 | +## Adding a New Test |
| 35 | + |
| 36 | +All currently implemented tests can be found in the `fillers` |
| 37 | +directory, which is composed of many subdirectories, and each one represents a |
| 38 | +different test category. |
| 39 | + |
| 40 | +Source files included in each category contain one or multiple test specs |
| 41 | +represented as python functions, and each can in turn produce one or many test |
| 42 | +vectors. |
| 43 | + |
| 44 | +A new test can be added by either: |
| 45 | + |
| 46 | +- Adding a new `test_` python function to an existing file in any of the |
| 47 | + existing category subdirectories within `fillers`. |
| 48 | +- Creating a new source file in an existing category, and populating it with |
| 49 | + the new test function(s). |
| 50 | +- Creating an entirely new category by adding a subdirectory in |
| 51 | + `fillers` with the appropriate source files and test functions. |
| 52 | + |
| 53 | +## Test Spec Generator Functions |
| 54 | + |
| 55 | +Every test spec is a python generator function which can perform a single or |
| 56 | +multiple `yield` operations during its runtime to each time yield a single |
| 57 | +`StateTest`/`BlockchainTest` object. |
| 58 | + |
| 59 | +The test vector's generator function _must_ be decorated by only one of the |
| 60 | +following decorators: |
| 61 | +- `test_from` |
| 62 | +- `test_from_until` |
| 63 | +- `test_only` |
| 64 | + |
| 65 | +These decorators specify the forks on which the test vector is supposed to run. |
| 66 | + |
| 67 | +They also automatically append necessary information for the |
| 68 | +`ethereum_test_filling_tool` to process when the generator is being executed to |
| 69 | +fill the tests. |
| 70 | + |
| 71 | +The test vector function must take only one `str` parameter: the fork name. |
| 72 | + |
| 73 | +## `StateTest` Object |
| 74 | + |
| 75 | +The `StateTest` object represents a single test vector, and contains the |
| 76 | +following attributes: |
| 77 | + |
| 78 | +- env: Environment object which describes the global state of the blockchain |
| 79 | + before the test starts. |
| 80 | +- pre: Pre-State containing the information of all Ethereum accounts that exist |
| 81 | + before any transaction is executed. |
| 82 | +- post: Post-State containing the information of all Ethereum accounts that are |
| 83 | + created or modified after all transactions are executed. |
| 84 | +- txs: All transactions to be executed during the test vector runtime. |
| 85 | + |
| 86 | +## `BlockchainTest` Object |
| 87 | + |
| 88 | +The `BlockchainTest` object represents a single test vector that evaluates the |
| 89 | +Ethereum VM by attempting to append multiple blocks to the chain: |
| 90 | + |
| 91 | +- pre: Pre-State containing the information of all Ethereum accounts that exist |
| 92 | + before any block is executed. |
| 93 | +- post: Post-State containing the information of all Ethereum accounts that are |
| 94 | + created or modified after all blocks are executed. |
| 95 | +- blocks: All blocks to be appended to the blockchain during the test. |
| 96 | + |
| 97 | + |
| 98 | +## Pre/Post State of the Test |
| 99 | + |
| 100 | +The `pre` and `post` states are elemental to setup and then verify the outcome |
| 101 | +of the state test. |
| 102 | + |
| 103 | +Both `pre` and `post` are mappings of account addresses to `account` structures (see [more info](#the-account-object)). |
| 104 | + |
| 105 | + |
| 106 | +A single test vector can contain as many accounts in the `pre` and `post` states |
| 107 | +as required, and they can be also filled dynamically. |
| 108 | + |
| 109 | +`storage` of an account is a key/value dictionary, and its values are |
| 110 | +integers within range of `[0, 2**256 - 1]`. |
| 111 | + |
| 112 | +`txs` are the steps which transform the pre-state into the post-state and |
| 113 | +must perform specific actions within the accounts (smart contracts) that result |
| 114 | +in verifiable changes to the balance, nonce, and/or storage in each of them. |
| 115 | + |
| 116 | +`post` is compared against the outcome of the client after the execution |
| 117 | +of each transaction, and any differences are considered a failure |
| 118 | + |
| 119 | +When designing a test, all the changes must be ideally saved into the contract's |
| 120 | +storage to be able to verify them in the post-state. |
| 121 | + |
| 122 | +## Test Transactions |
| 123 | + |
| 124 | +Transactions can be crafted by sending them with specific `data` or to a |
| 125 | +specific account, which contains the code to be executed |
| 126 | + |
| 127 | +Transactions can also create more accounts, by setting the `to` field to an |
| 128 | +empty string. |
| 129 | + |
| 130 | +Transactions can be designed to fail, and a verification must be made that the |
| 131 | +transaction fails with the specific error that matches what is expected by the |
| 132 | +test. |
| 133 | + |
| 134 | +## Writing code for the accounts in the test |
| 135 | + |
| 136 | +Account bytecode can be embedded in the test accounts by adding it to the `code` |
| 137 | +field of the `account` object, or the `data` field of the `tx` object if the |
| 138 | +bytecode is meant to be treated as init code or call data. |
| 139 | + |
| 140 | +The code can be in either of the following formats: |
| 141 | +- `bytes` object, representing the raw opcodes in binary format |
| 142 | +- `str`, representing an hexadecimal format of the opcodes |
| 143 | +- `Code` compilable object |
| 144 | + |
| 145 | +Currently supported built-in compilable objects are: |
| 146 | + |
| 147 | +- `Yul` object containing [Yul source code][yul] |
| 148 | + |
| 149 | +`Code` objects can be concatenated together by using the `+` operator. |
| 150 | + |
| 151 | +## Verifying the Accounts' Post State |
| 152 | + |
| 153 | +The state of the accounts after all blocks/transactions have been executed is |
| 154 | +the way of verifying that the execution client actually behaves like the test |
| 155 | +expects. |
| 156 | + |
| 157 | +During their filling process, all tests automatically verify that the accounts |
| 158 | +specified in their `post` property actually match what was returned by the |
| 159 | +transition tool. |
| 160 | + |
| 161 | +Within the `post` dictionary object, an account address can be: |
| 162 | +- `None`: The account will not be checked for absence or existence in the |
| 163 | + result returned by the transition tool. |
| 164 | +- `Account` object: The test expects that this account exist and also has |
| 165 | + properties equal to the properties specified by the `Account` object. |
| 166 | +- `Account.NONEXISTENT`: The test expects that this account does not exist in |
| 167 | + the result returned by the transition tool, and if the account exists, |
| 168 | + it results in error. |
| 169 | + E.g. when the transaction creating a contract is expected to fail and the |
| 170 | + test wants to verify that the address where the contract was supposed to be |
| 171 | + created is indeed empty. |
| 172 | + |
| 173 | +## The `Account` object |
| 174 | + |
| 175 | +The `Account` object is used to specify the properties of an account to be |
| 176 | +verified in the post state. |
| 177 | + |
| 178 | +The python representation can be found in [src/ethereum_test_tools/common/types.py](src/ethereum_test_tools/common/types.py). |
| 179 | + |
| 180 | +It can verify the following properties of an account: |
| 181 | +- `nonce`: the scalar value equal to a) the number of transactions sent by |
| 182 | + an Externally Owned Account, b) the amount of contracts created by a contract. |
| 183 | + |
| 184 | +- `balance`: the amount of Wei (10<sup>-18</sup> Eth) the account has. |
| 185 | + |
| 186 | +- `code`: Bytecode contained by the account. To verify that an account contains |
| 187 | + no code, this property needs to be set to "0x" or "". |
| 188 | + |
| 189 | + It is not recommended to verify Yul compiled code in the output account, |
| 190 | + because the bytecode can change from version to version. |
| 191 | + |
| 192 | +- `storage`: Storage within the account represented as a `dict` object. |
| 193 | + All storage keys that are expected to be set must be specified, and if a |
| 194 | + key is skipped, it is implied that its expected value is zero. |
| 195 | + Setting this property to `{}` (empty `dict`), means that all the keys in the |
| 196 | + account must be unset (equal to zero). |
| 197 | + |
| 198 | +All account's properties are optional, and they can be skipped or set to `None`, |
| 199 | +which means that no check will be performed on that specific account property. |
| 200 | + |
| 201 | +## Verifying correctness of the new test |
| 202 | + |
| 203 | +A well written test performs a single verification output at a time. |
| 204 | + |
| 205 | +A verification output can be a single storage slot, the balance of an account, |
| 206 | +or a newly created contract. |
| 207 | + |
| 208 | +It is not recommended to use balance changes to verify test correctness, as it |
| 209 | +can be easily affected by gas cost changes in future EIPs. |
| 210 | + |
| 211 | +The best way to verify a transaction/block execution outcome is to check its |
| 212 | +storage. |
| 213 | + |
| 214 | +A test can be written as a negative verification. E.g. a contract is not |
| 215 | +created, or a transaction fails to execute or runs out of gas. |
| 216 | + |
| 217 | +These verifications must be carefully crafted because it is possible to end up |
| 218 | +having a false positive result, which means that the test passed but the |
| 219 | +intended verification was never made. |
| 220 | + |
| 221 | +To avoid these scenarios, it is important to have a separate verification to |
| 222 | +check that test is effective. E.g. when a transaction is supposed to fail, it |
| 223 | +is necessary to check that the failure error is actually the one expected by |
| 224 | +the test. |
| 225 | + |
| 226 | +## Failing or invalid transactions |
| 227 | + |
| 228 | +Transactions included in a StateTest are expected to be intrinsically valid, |
| 229 | +i.e. the account sending the transaction must have enough funds to cover the |
| 230 | +gas costs, the max fee of the transaction must be equal or higher than the |
| 231 | +base fee of the block, etc. |
| 232 | + |
| 233 | +An intrinsically valid transaction can still revert during its execution. |
| 234 | + |
| 235 | +Blocks in a BlockchainTest can contain intrinsically invalid transactions but |
| 236 | +in this case the block is expected to be completely rejected, along with all |
| 237 | +transactions in it, including other valid transactions. |
| 238 | + |
| 239 | +[t8n]: https://github.com/ethereum/go-ethereum/tree/master/cmd/evm |
| 240 | +[b11r]: https://github.com/ethereum/go-ethereum/pull/23843 |
| 241 | +[yul]: https://docs.soliditylang.org/en/latest/yul.html |
0 commit comments