You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
-1Lines changed: 0 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -109,7 +109,6 @@ cd execution-spec-tests
109
109
uv python install 3.11
110
110
uv python pin 3.11
111
111
uv sync --all-extras
112
-
uv run solc-select use 0.8.24 --always-install
113
112
```
114
113
115
114
See [Installation Troubleshooting](https://eest.ethereum.org/main/getting_started/installation_troubleshooting/) in the online docs if you encounter issues.
Copy file name to clipboardExpand all lines: docs/CHANGELOG.md
+11Lines changed: 11 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,6 +10,11 @@ Test fixtures for use by clients are available for each release on the [Github r
10
10
11
11
- Python 3.10 support was removed in this release ([#1808](https://github.com/ethereum/execution-spec-tests/pull/1808)).
12
12
13
+
#### 💥 Important Change for test contributors
14
+
15
+
- EEST no longer allows usage of Yul code in Python tests. From now on, please make use of our opcode wrapper. Yul code is now only allowed in the "static tests" located in `./tests/static/` (these are test cases defined by JSON and YAML files instead of Python test functions that were originally maintained in [ethereum/tests](https://github.com/ethereum/tests)).
16
+
- In order to fill the static tests (which is not the case by default), please ensure that `solc` is located in your `PATH`.
17
+
13
18
#### 💥 Important Change for `fill` Users
14
19
15
20
The output behavior of `fill` has changed ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)):
@@ -27,6 +32,12 @@ Users can select any of the artifacts depending on their testing needs for their
27
32
28
33
### 🛠️ Framework
29
34
35
+
#### 🔀 Refactoring
36
+
37
+
- 🔀 Move `TransactionType` enum from test file to proper module location in `ethereum_test_types.transaction_types` for better code organization and reusability.
38
+
- ✨ Opcode classes now validate keyword arguments and raise `ValueError` with clear error messages.
39
+
- 🔀 This PR removes the `solc` requirement to fill Python test cases. Regular test contributors no longer need to concern themselves with `solc` and, as such, the `solc-select` dependency has been removed. The remaining tests that used Yul have been ported to the EEST opcode wrapper mini-lang and the use of Yul in Python tests is no longer supported. Maintainers only: To fill the "static" JSON and YAML tests (`./tests/static/`) locally, `solc` (ideally v0.8.24) must be available in your PATH.
40
+
30
41
#### `fill`
31
42
32
43
- ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository ([#1590](https://github.com/ethereum/execution-spec-tests/pull/1590)).
Copy file name to clipboardExpand all lines: docs/dev/porting_legacy_tests.md
+2-4Lines changed: 2 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -85,16 +85,14 @@ Follow the hyperlinks for lost base coverage (`LBC`) to address coverage gaps. H
85
85
86
86
It's important to note that coverage helps identify missing test paths. If you believe the coverage loss is due to differences in "setup" code between frameworks and doesn't impact the feature you're testing, explain this in your PR. A team member can help with the review.
87
87
88
-
Also note that yul tests and possibly other tests used `CALLDATALOAD` that might no longer needed when designing a test with python. But we must always investigate if an opcode is not covered anymore to see if its okay.
89
-
90
-
For example, review the [discussion in this PR.](https://github.com/ethereum/execution-spec-tests/pull/975#issuecomment-2528792289)
88
+
For example, review the [discussion in this PR] to see an example of why and how coverage loss can occur.(https://github.com/ethereum/execution-spec-tests/pull/975#issuecomment-2528792289)
91
89
92
90
## Resolving Coverage Gaps from Yul Compilation
93
91
94
92
When porting tests from ethereum/tests, you may encounter coverage gaps that are false positives. This commonly occurs because:
95
93
96
94
-**Original tests** often used Yul to define smart contracts, and solc compilation introduces additional opcodes that aren't specifically under test
97
-
-**EEST ports**typically use the explicit EEST Opcode mini-language, which is more precise in opcode definition
95
+
-**EEST ports** use the explicit EEST Opcode mini-language, which is more precise in opcode definition
98
96
99
97
If coverage analysis shows missing opcodes that were only present due to Yul compilation artifacts (not the actual feature being tested), this can be resolved during PR review by adding the `coverage_missed_reason` parameter:
Copy file name to clipboardExpand all lines: docs/getting_started/installation.md
+2-12Lines changed: 2 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,27 +26,17 @@ If installed via `curl`, `uv` will download Python for your target platform if o
26
26
27
27
Clone [execution-spec-tests](https://github.com/ethereum/execution-spec-tests) and install its dependencies. We recommend using Python 3.12, the following uses `uv` to download and configures 3.12 to be the Python version used in execution-spec-tests:
Then follow [this guide](./installation_troubleshooting.md#problem-exception-failed-to-compile-yul-source) to build the `solc` binary from source and copy it to the expected location.
39
+
Static tests/maintainers only: To learn how to build the `solc` binary from source (optional) follow [this guide](./installation_troubleshooting.md#problem-exception-failed-to-compile-yul-source).
Copy file name to clipboardExpand all lines: docs/getting_started/installation_troubleshooting.md
+6-13Lines changed: 6 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,11 +38,11 @@ This page provides guidance on how to troubleshoot common issues that may arise
38
38
39
39
### Problem: `CERTIFICATE_VERIFY_FAILED`
40
40
41
-
!!! danger "Problem: `Failed to install solc ... CERTIFICATE_VERIFY_FAILED`"
42
-
When running either `uv run solc-select use 0.8.24 --always-install` or `fill` you encounter the following error:
41
+
!!! danger "Problem: `Failed to ... CERTIFICATE_VERIFY_FAILED`"
42
+
When running `fill` you might encounter the following error:
43
43
44
44
```bash
45
-
Exit: Failed to install solc version 0.8.24: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)>
45
+
Exit: Failed to ...: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)>
46
46
```
47
47
48
48
=== "Ubuntu"
@@ -66,15 +66,8 @@ This page provides guidance on how to troubleshoot common issues that may arise
66
66
67
67
### Problem: `Exception: failed to compile yul source`
68
68
69
-
!!! danger "Problem: `Running fill fails with tests that contain yul source code` on ARM platforms"
70
-
If you're using `x86_64`, try to manually run `solc-select` to install the required version of the `solc` binary:
71
-
72
-
```shell
73
-
uv run solc-select use 0.8.24 --always-install
74
-
```
75
-
76
-
Otherwise, this can happen when you're using an ARM64 OS but followed the x86-64 installation guide.
77
-
To resolve the issue you must build solidity from source (avoid 0.8.24 as it might require building z3 from source as well).
69
+
!!! danger "Problem: `Running fill on static_tests fails with tests that contain yul source code` on ARM platforms"
70
+
To resolve the issue you must acquire the `solc` binary e.g. by building solidity from source. The guide below installs v0.8.28 but you might prefer to choose a different version.
78
71
79
72
!!! success "Solution: Build solc from source"
80
73
The following steps have been tested on Ubuntu 24.04.2 LTS ARM64:
@@ -88,7 +81,7 @@ This page provides guidance on how to troubleshoot common issues that may arise
Running `uv run solc --version` should now return the expected version. Verify that `fill` works by running `uv run fill -m yul_test` in the execution-spec-tests root folder.
84
+
Running `uv run solc --version` should now return the expected version.
92
85
93
86
## Problem: `ValueError: unsupported hash type ripemd160`
Copy file name to clipboardExpand all lines: docs/writing_tests/tutorials/state_transition.md
+20-66Lines changed: 20 additions & 66 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,27 +9,12 @@ Before proceeding with this tutorial, it is assumed that you have prior knowledg
9
9
- Repository set-up, see [installation](../../getting_started/installation.md).and run an execution specification test as outlined in the .
10
10
- Able to run `fill`, see [Getting Started: Filling Tests](../../filling_tests/getting_started.md).
11
11
- Understand how to read a [static state transition test](https://ethereum-tests.readthedocs.io/en/latest/state-transition-tutorial.html#the-source-code).
12
-
- Know the basics of [Yul](https://docs.soliditylang.org/en/latest/yul.html), which is an EVM assembly language.
12
+
- Know the basics of the [EVM](https://www.evm.codes/).
13
13
- Familiarity with [Python](https://docs.python.org/3/tutorial/).
14
14
15
-
## Example Tests
15
+
## Example Test
16
16
17
-
The most effective method of learning how to write tests is to study a couple of straightforward examples. In this tutorial we will go over the [Yul](https://github.com/ethereum/execution-spec-tests/blob/main/tests/homestead/yul/test_yul_example.py#L19) state test.
18
-
19
-
### Yul Test
20
-
21
-
You can find the source code for the Yul test in [tests/homestead/yul/test_yul_example.py](../../tests/homestead/yul/test_yul_example/index.md).
22
-
It is the spec test equivalent of this [static test](https://github.com/ethereum/tests/blob/develop/src/GeneralStateTestsFiller/stExample/yulExampleFiller.yml).
23
-
24
-
Lets examine each section.
25
-
26
-
```python
27
-
"""
28
-
Test Yul Source Code Examples
29
-
"""
30
-
```
31
-
32
-
In Python, multi-line strings are denoted using `"""`. As a convention, a file's purpose is often described in the opening string of the file.
17
+
The most effective method of learning how to write tests is to study a straightforward example. In this tutorial we will go over a simple state test that adds two numbers and stores them in the storage.
33
18
34
19
```python
35
20
from ethereum_test_forks import Fork, Frontier, Homestead
@@ -39,7 +24,6 @@ from ethereum_test_tools import (
39
24
Environment,
40
25
StateTestFiller,
41
26
Transaction,
42
-
YulCompiler,
43
27
)
44
28
```
45
29
@@ -57,19 +41,19 @@ In this case, the decorator is a custom [pytest fixture](https://docs.pytest.org
57
41
To fill this test for all the specified forks, we can specify pytest's `-k` flag that [filters test cases by keyword expression](https://docs.pytest.org/en/latest/how-to/usage.html#specifying-tests-selecting-tests):
58
42
59
43
```console
60
-
fill -k test_yul
44
+
fill -k test_example
61
45
```
62
46
63
47
and to fill it for a specific fork range, we can provide the `--from` and `--until` command-line arguments:
Test that adds two numbers and stores the result in storage at key 0.
73
57
"""
74
58
```
75
59
@@ -93,7 +77,7 @@ In most tests the defaults are good enough.
93
77
94
78
For more information, [see the static test documentation](../../running_tests/test_formats/state_test.md).
95
79
96
-
####Pre State
80
+
### Pre State
97
81
98
82
For every test we need to define the pre-state requirements, so we are certain of what is on the "blockchain" before the transaction is executed.
99
83
It can be used as a [dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries), which is the Python term for an associative array, but the appropriate way to populate it is by using the methods `fund_eoa`, `deploy_contract` or `fund_address` from the `Alloc` object.
@@ -102,23 +86,18 @@ In this example we are using the `deploy_contract` method to deploy a contract t
102
86
103
87
```python
104
88
contract_address = pre.deploy_contract(
105
-
code=yul(
106
-
"""
107
-
{
108
-
function f(a, b) -> c {
109
-
c := add(a, b)
110
-
}
111
-
112
-
sstore(0, f(1, 2))
113
-
return(0, 32)
114
-
}
115
-
"""
116
-
),
89
+
code= Op.PUSH1(0)
90
+
+ Op.PUSH1(2)
91
+
+ Op.PUSH1(1)
92
+
+ Op.ADD
93
+
+ Op.SSTORE
94
+
+ Op.STOP
95
+
,
117
96
balance=0x0BA1A9CE0BA1A9CE,
118
97
)
119
98
```
120
99
121
-
Specifically we deploy a contract with yul code that adds two numbers and stores the result in storage.
100
+
Specifically we deploy a contract with EVM bytecode that adds two numbers and stores the result in storage.
122
101
123
102
```python
124
103
balance=0x0BA1A9CE0BA1A9CE,
@@ -134,38 +113,13 @@ As return value of the `deploy_contract` method we get the address where the con
134
113
135
114
```python
136
115
storage={
137
-
0x00: 0x00,
116
+
0x00: 0x03,
138
117
},
139
118
```
140
119
141
120
We could also specify a starting storage for the contract, which is done by adding a `storage` parameter to the `deploy_contract` method.
142
121
143
-
```python
144
-
code=yul(
145
-
```
146
-
147
-
Here we define the [Yul](https://docs.soliditylang.org/en/v0.8.17/yul.html) code for the contract. It is defined as a multi-line string and starts and ends with curly braces (`{ <yul> }`).
148
-
149
-
When running the test filler `fill`, the solidity compiler `solc` will automatically translate the Yul to EVM opcode at runtime.
150
-
151
-
!!! note
152
-
Currently Yul and direct EVM opcode are supported in execution spec tests.
153
-
154
-
```python
155
-
"""
156
-
{
157
-
function f(a, b) -> c {
158
-
c := add(a, b)
159
-
}
160
-
sstore(0, f(1, 2))
161
-
return(0, 32)
162
-
}
163
-
"""
164
-
```
165
-
166
-
Within this example test Yul code we have a function definition, and inside it we are using the Yul `add` instruction. When compiled with`solc` it translates the instruction directly to the `ADD` opcode. For further Yul instructions [see here](https://docs.soliditylang.org/en/latest/yul.html#evm-dialect). Notice that function is utilized with the Yul `sstore` instruction, which stores the result of `add(1, 2)` to the storage address `0x00`.
167
-
168
-
Generally for execution spec tests the `sstore` instruction acts as a high-level assertion method to check pre to post-state changes. The test filler achieves this by verifying that the correct value is held within post-state storage, hence we can validate that the Yul code has run successfully.
122
+
Generally for execution spec tests the `sstore` instruction acts as a high-level assertion method to check pre to post-state changes. The test filler achieves this by verifying that the correct value is held within post-state storage, hence we can validate that the code has run successfully.
169
123
170
124
```python
171
125
sender= pre.fund_eoa(amount=0x0BA1A9CE0BA1A9CE)
@@ -223,15 +177,15 @@ For more information, [see the static test documentation](../../running_tests/te
223
177
224
178
This is the post-state which is equivalent to [`expect`](https://ethereum-tests.readthedocs.io/en/latest/test_filler/state_filler.html#expect) in static tests, but without the indexes. It is similar to the pre-state, except that we do not need to specify everything, only those accounts and fields we wish to test.
225
179
226
-
In this case, we look at the storage of the contract we called and add to it what we expect to see. In this example storage cell `0x00` should be `0x03`asin the pre-state we essentially stored the result of the Yul instruction `add(1, 2)`.
180
+
In this case, we look at the storage of the contract we called and add to it what we expect to see. In this example storage cell `0x00` should be `0x03`asin the pre-state we essentially stored the result of the instruction `add(1, 2)`.
227
181
228
182
#### State Test
229
183
230
184
```python
231
185
state_test(env=env, pre=pre, post=post, tx=tx)
232
186
```
233
187
234
-
This line calls the wrapper to the `StateTest`object that provides all the objects required (for example, the fork parameter) in order to fill the test, generate the test fixtures and write them to file (by default, `./fixtures/<blockchain,state>_tests/example/yul_example/test_yul.json`).
188
+
This line calls the wrapper to the `StateTest`object that provides all the objects required (for example, the fork parameter) in order to fill the test, generate the test fixtures and write them to file (by default, `./fixtures/<blockchain,state>_tests/example/test_example.json`).
0 commit comments