|
| 1 | +# Using Test Fixtures Generated by execution-spec-tests |
| 2 | + |
| 3 | +@ethereum/execution-spec-tests generates JSON test fixtures that can be consumed by execution clients directly or via the [Hive `pyspec` simulator](https://github.com/ethereum/hive/tree/master/simulators/ethereum/pyspec#readme): |
| 4 | + |
| 5 | +1. **Clients** can execute the tests directly against their EVM by implementing a command analogous to the go-ethereum's `evm blocktest` command, see [cmd/evm/blockrunner.go](https://github.com/ethereum/go-ethereum/blob/509a64ffb9405942396276ae111d06f9bded9221/cmd/evm/blockrunner.go#L39). |
| 6 | +2. **Hive** executes the tests against a fully instantiated client instance using the Engine API. |
| 7 | + |
| 8 | +Here's a top-level comparison of these two approaches: |
| 9 | + |
| 10 | +| Consumed via | Scope | Pros | Cons | |
| 11 | +| --- | --- | --- | --- | |
| 12 | +| `blocktest`-like command | Module test | - Fast feedback loop<br/>- Less complex | - Smaller coverage scope | |
| 13 | +| `hive --sim ethereum/pyspec` | System test / Integration test | - Wider Coverage Scope<br/>- Tests more of the client stack | - Slower feedback loop<br/>- Harder to debug | |
| 14 | + |
| 15 | +!!! note "Executing a `t8n` tool via `fill` is not considered to be the actual test" |
| 16 | + |
| 17 | + The `fill` command uses `t8n` tools to generate fixtures. Whilst this will provide basic sanity checking of EVM behavior and a sub-set of post conditions are typically checked within test cases, it is not considered the actual test. The actual test is the execution of the fixture against the EVM which will check the entire post allocation and typically use different code paths than `t8n` commands. |
| 18 | + |
| 19 | +!!! note "Running `blocktest` directly within the execution-spec-tests framework" |
| 20 | + |
| 21 | + It's possible to execute `evm blocktest` directly within the execution-spec-tests framework. This is intended to verify fixture generation, see [Debugging `t8n` Tools](./debugging_t8n_tools.md). |
| 22 | + |
| 23 | +## Release Formats |
| 24 | + |
| 25 | +The @ethereum/execution-spec-tests repository provides [releases](https://github.com/ethereum/execution-spec-tests/releases) of fixtures in various formats (as of 2023-10-16): |
| 26 | + |
| 27 | +| Release Artifact | Consumer | Fork/feature scope | |
| 28 | +| ------------------------------ | -------- | ------------------ | |
| 29 | +| `fixtures.tar.gz` | Clients | All tests until the last stable fork | "Must pass" | |
| 30 | +| `fixtures_develop.tar.gz` | Clients | All tests until the last development fork | |
| 31 | +| `fixtures_hive.tar.gaz` | Hive | All tests until the last stable fork in hive format | |
| 32 | +| `fixtures_develop_hive.tar.gz` | Hive | All tests until the last development fork in hive format | |
| 33 | + |
| 34 | +The Hive format uses Engine API directives instead of the usual BlockchainTest format. |
| 35 | + |
| 36 | +## Obtaining the Most Recent Release Artifacts |
| 37 | + |
| 38 | +Artifacts can be downloaded directly from [the release page](https://github.com/ethereum/execution-spec-tests/releases). The following script demonstrates how the most recent release version of a specific artifact can be downloaded using the Github API: |
| 39 | + |
| 40 | +```bash |
| 41 | +#!/bin/bash |
| 42 | + |
| 43 | +# requires jq |
| 44 | +# sudo apt install jq |
| 45 | + |
| 46 | +# The following two artifacts are intended for consumption by clients: |
| 47 | +# - fixtures.tar.gz: Generated up to the last deployed fork. |
| 48 | +# - fixtures_develop.tar.gz: Generated up to and including the latest dev fork. |
| 49 | +# As of Oct 2023, dev is Cancun, deployed is Shanghai. |
| 50 | + |
| 51 | +ARTIFACT="fixtures_develop.tar.gz" |
| 52 | + |
| 53 | +OWNER="ethereum" |
| 54 | +REPO="execution-spec-tests" |
| 55 | + |
| 56 | +DOWNLOAD_URL=$(curl -s https://api.github.com/repos/$OWNER/$REPO/releases/latest \ |
| 57 | + | jq -r '.assets[] | select(.name=="'$ARTIFACT'").browser_download_url') |
| 58 | + |
| 59 | +# Sanity check for the download URL: contains a version tag prefixed with "v" |
| 60 | +if [[ "$DOWNLOAD_URL" =~ v[0-9]+\.[0-9]+\.[0-9]+ ]]; then |
| 61 | + curl -LO $DOWNLOAD_URL |
| 62 | +else |
| 63 | + echo "Error: URL does not contain a valid version tag (URL: ${DOWNLOAD_URL})." |
| 64 | + exit 1 |
| 65 | +fi |
| 66 | +``` |
0 commit comments