Skip to content

Commit d7c51a1

Browse files
refactor(consume): avoid github api calls for direct url fixture inputs (ethereum#1788)
* feat(consume): add `--no-api-calls` flag for --input=<github-release-url> * docs(consume): update consume cache docs for the `--no-api-calls` flag * docs(hive): align sub-section title style * docs: update changelog * Apply suggestions from code review Co-authored-by: spencer <[email protected]> * refactor(consume): simplify API behavior for direct URLs Remove the --no-api-calls flag and instead make the behavior automatic: - Direct GitHub release URLs: No API calls, no release page output - Release specifiers (stable@latest): API calls required, includes release page This provides better CI/UX without requiring additional flags. * docs(changelog): update consume API behavior entry Describe the automatic API call avoidance for direct URLs without mentioning any flags since we implemented automatic behavior. * Apply suggestions from code review Co-authored-by: spencer <[email protected]> --------- Co-authored-by: spencer <[email protected]>
1 parent 8b5fedc commit d7c51a1

File tree

5 files changed

+207
-9
lines changed

5 files changed

+207
-9
lines changed

docs/CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ Users can select any of the artifacts depending on their testing needs for their
4141

4242
#### `consume`
4343

44+
- 🔀 `consume` now automatically avoids GitHub API calls when using direct release URLs (better for CI environments), while release specifiers like `stable@latest` continue to use the API for version resolution ([#1788](https://github.com/ethereum/execution-spec-tests/pull/1788)).
45+
4446
#### `execute`
4547

4648
- ✨ Add `blob_transaction_test` execute test spec, which allows tests that send blob transactions to a running client and verifying its `engine_getBlobsVX` endpoint behavior ([#1644](https://github.com/ethereum/execution-spec-tests/pull/1644)).

docs/running_tests/consume/cache.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ Releases can be downloaded using EEST tooling without (manually) cloning and ins
3939
Release page: https://github.com/ethereum/execution-spec-tests/releases/tag/v4.5.0
4040
```
4141

42+
**Note:** Use direct URLs to avoid GitHub API calls (better for CI environments). Version specifiers like `stable@latest` will always use the GitHub API to resolve versions. More details on the arguments to `--input` are provided below.
43+
4244
**Explanation:** `uv` creates a local Python virtual environment in `~/.cache/uv/`, installs EEST and executes the `consume cache` command to resolve and download the release, which gets cached at `~/.cache/ethereum-execution-spec-tests`. Subsequent commands will use the cached version of the fixtures.
4345

4446
## The `--input` Flag to Specify Fixtures
@@ -106,6 +108,12 @@ All remote fixture sources are automatically cached to avoid repeated downloads:
106108
~/.cache/ethereum-execution-spec-tests/cached_downloads/
107109
```
108110

111+
You can override this location with the `--cache-folder` flag:
112+
113+
```bash
114+
uv run consume cache --input stable@latest --cache-folder /path/to/custom/cache
115+
```
116+
109117
**Cache structure:**
110118

111119
```text
@@ -137,3 +145,26 @@ All remote fixture sources are automatically cached to avoid repeated downloads:
137145
## The Fixture Index File
138146

139147
The [`fill` command](../../filling_tests/index.md) generates a JSON file `<fixture_path>/.meta/index.json` that indexes the fixtures its generated. This index file is used by `consume` commands to allow fast collection of test subsets specified on the command-line, for example, via the `--sim.limit` flag. For help with `--sim.limit` when running `./hive`, see [Hive Common Options](../hive/common_options.md), for an overview of other available test selection flags when running `consume` directly, see [Useful Pytest Options](../useful_pytest_options.md).
148+
149+
## CI-Friendly Behavior for Direct URLs
150+
151+
When using direct GitHub release URLs (instead of version specifiers), the consume command automatically avoids unnecessary GitHub API calls to prevent rate limiting in CI environments:
152+
153+
```console
154+
consume cache --input=https://github.com/ethereum/execution-spec-tests/releases/download/v4.5.0/fixtures_stable.tar.gz
155+
```
156+
157+
**API Call Behavior:**
158+
159+
-**Direct URLs**: No API calls made, cleaner output (no "Release page:" line).
160+
- ℹ️ **Version specifiers**: API calls required to resolve versions, includes release page info.
161+
162+
Examples:
163+
164+
```console
165+
# No API calls - direct download
166+
consume cache --input=https://github.com/ethereum/execution-spec-tests/releases/download/v4.5.0/fixtures_stable.tar.gz
167+
168+
# API calls required - version resolution
169+
consume cache --input=stable@latest
170+
```

docs/running_tests/hive/dev_mode.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ This avoids running the simulator in a dockerized environment and has several ad
2121
- EEST is installed, see [Installation](../../getting_started/installation.md)
2222
- Hive is built, see [Hive](../hive/index.md#quick-start).
2323

24-
## Hive Dev Setup (Linux)
24+
## Hive Dev Setup on Linux
2525

2626
1. Start Hive in development mode, e.g.:
2727

@@ -50,7 +50,7 @@ This avoids running the simulator in a dockerized environment and has several ad
5050
uv run consume rlp --input stable@latest
5151
```
5252

53-
## macOS Setup
53+
## Hive Dev Setup on macOS
5454

5555
Due to Docker running within a VM on macOS, the host machine and Docker containers don't share the same network namespace, preventing direct communication with Hive's development server. To run development mode on macOS, you have the following options:
5656

src/pytest_plugins/consume/consume.py

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -129,14 +129,14 @@ def from_input(cls, input_source: str) -> "FixturesSource":
129129
@classmethod
130130
def from_release_url(cls, url: str) -> "FixturesSource":
131131
"""Create a fixture source from a supported github repo release URL."""
132-
release_page = get_release_page_url(url)
133132
downloader = FixtureDownloader(url, CACHED_DOWNLOADS_DIRECTORY)
134133
was_cached, path = downloader.download_and_extract()
134+
135135
return cls(
136136
input_option=url,
137137
path=path,
138138
url=url,
139-
release_page=release_page,
139+
release_page="",
140140
is_local=False,
141141
was_cached=was_cached,
142142
)
@@ -319,11 +319,10 @@ def pytest_configure(config): # noqa: D103
319319
reason += "Fixtures already cached."
320320
elif not config.fixtures_source.is_local:
321321
reason += "Fixtures downloaded and cached."
322-
reason += (
323-
f"\nPath: {config.fixtures_source.path}\n"
324-
f"Input: {config.fixtures_source.url or config.fixtures_source.path}\n"
325-
f"Release page: {config.fixtures_source.release_page or 'None'}"
326-
)
322+
reason += f"\nPath: {config.fixtures_source.path}"
323+
reason += f"\nInput: {config.fixtures_source.url or config.fixtures_source.path}"
324+
if config.fixtures_source.release_page:
325+
reason += f"\nRelease page: {config.fixtures_source.release_page}"
327326
pytest.exit(
328327
returncode=0,
329328
reason=reason,
Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
"""Test the simplified consume behavior for different input types."""
2+
3+
from pathlib import Path
4+
from unittest.mock import MagicMock, patch
5+
6+
from pytest_plugins.consume.consume import FixturesSource
7+
8+
9+
class TestSimplifiedConsumeBehavior:
10+
"""Test suite for the simplified consume behavior."""
11+
12+
def test_fixtures_source_from_release_url_no_api_calls(self):
13+
"""Test that direct release URLs do not make API calls for release page."""
14+
test_url = "https://github.com/ethereum/execution-spec-tests/releases/download/v3.0.0/fixtures_develop.tar.gz"
15+
16+
with patch("pytest_plugins.consume.consume.FixtureDownloader") as mock_downloader:
17+
mock_instance = MagicMock()
18+
mock_instance.download_and_extract.return_value = (False, Path("/tmp/test"))
19+
mock_downloader.return_value = mock_instance
20+
21+
source = FixturesSource.from_release_url(test_url)
22+
23+
# Verify no release page is set for direct URLs
24+
assert source.release_page == ""
25+
assert source.url == test_url
26+
assert source.input_option == test_url
27+
28+
def test_fixtures_source_from_release_spec_makes_api_calls(self):
29+
"""Test that release specs still make API calls and get release page."""
30+
test_spec = "stable@latest"
31+
32+
with patch("pytest_plugins.consume.consume.get_release_url") as mock_get_url:
33+
mock_get_url.return_value = "https://github.com/ethereum/execution-spec-tests/releases/download/v3.0.0/fixtures_stable.tar.gz"
34+
with patch("pytest_plugins.consume.consume.get_release_page_url") as mock_get_page:
35+
mock_get_page.return_value = (
36+
"https://github.com/ethereum/execution-spec-tests/releases/tag/v3.0.0"
37+
)
38+
with patch("pytest_plugins.consume.consume.FixtureDownloader") as mock_downloader:
39+
mock_instance = MagicMock()
40+
mock_instance.download_and_extract.return_value = (False, Path("/tmp/test"))
41+
mock_downloader.return_value = mock_instance
42+
43+
source = FixturesSource.from_release_spec(test_spec)
44+
45+
# Verify API calls were made and release page is set
46+
mock_get_url.assert_called_once_with(test_spec)
47+
mock_get_page.assert_called_once_with(
48+
"https://github.com/ethereum/execution-spec-tests/releases/download/v3.0.0/fixtures_stable.tar.gz"
49+
)
50+
assert (
51+
source.release_page
52+
== "https://github.com/ethereum/execution-spec-tests/releases/tag/v3.0.0"
53+
)
54+
55+
def test_fixtures_source_from_regular_url_no_release_page(self):
56+
"""Test that regular URLs (non-GitHub) don't have release page."""
57+
test_url = "http://example.com/fixtures.tar.gz"
58+
59+
with patch("pytest_plugins.consume.consume.FixtureDownloader") as mock_downloader:
60+
mock_instance = MagicMock()
61+
mock_instance.download_and_extract.return_value = (False, Path("/tmp/test"))
62+
mock_downloader.return_value = mock_instance
63+
64+
source = FixturesSource.from_url(test_url)
65+
66+
# Verify no release page for regular URLs
67+
assert source.release_page == ""
68+
assert source.url == test_url
69+
70+
def test_output_formatting_without_release_page_for_direct_urls(self):
71+
"""Test output formatting when release page is empty for direct URLs."""
72+
from unittest.mock import MagicMock
73+
74+
from pytest import Config
75+
76+
config = MagicMock(spec=Config)
77+
config.fixtures_source = MagicMock()
78+
config.fixtures_source.was_cached = False
79+
config.fixtures_source.is_local = False
80+
config.fixtures_source.path = Path("/tmp/test")
81+
config.fixtures_source.url = "https://github.com/ethereum/execution-spec-tests/releases/download/v3.0.0/fixtures_develop.tar.gz"
82+
config.fixtures_source.release_page = "" # Empty for direct URLs
83+
84+
# Simulate the output generation logic from pytest_configure
85+
reason = ""
86+
if config.fixtures_source.was_cached:
87+
reason += "Fixtures already cached."
88+
elif not config.fixtures_source.is_local:
89+
reason += "Fixtures downloaded and cached."
90+
reason += f"\nPath: {config.fixtures_source.path}"
91+
reason += f"\nInput: {config.fixtures_source.url or config.fixtures_source.path}"
92+
if config.fixtures_source.release_page:
93+
reason += f"\nRelease page: {config.fixtures_source.release_page}"
94+
95+
assert "Release page:" not in reason
96+
assert "Path:" in reason
97+
assert "Input:" in reason
98+
99+
def test_output_formatting_with_release_page_for_specs(self):
100+
"""Test output formatting when release page is present for release specs."""
101+
from unittest.mock import MagicMock
102+
103+
from pytest import Config
104+
105+
config = MagicMock(spec=Config)
106+
config.fixtures_source = MagicMock()
107+
config.fixtures_source.was_cached = False
108+
config.fixtures_source.is_local = False
109+
config.fixtures_source.path = Path("/tmp/test")
110+
config.fixtures_source.url = "https://github.com/ethereum/execution-spec-tests/releases/download/v3.0.0/fixtures_stable.tar.gz"
111+
config.fixtures_source.release_page = (
112+
"https://github.com/ethereum/execution-spec-tests/releases/tag/v3.0.0"
113+
)
114+
115+
# Simulate the output generation logic from pytest_configure
116+
reason = ""
117+
if config.fixtures_source.was_cached:
118+
reason += "Fixtures already cached."
119+
elif not config.fixtures_source.is_local:
120+
reason += "Fixtures downloaded and cached."
121+
reason += f"\nPath: {config.fixtures_source.path}"
122+
reason += f"\nInput: {config.fixtures_source.url or config.fixtures_source.path}"
123+
if config.fixtures_source.release_page:
124+
reason += f"\nRelease page: {config.fixtures_source.release_page}"
125+
126+
assert (
127+
"Release page: https://github.com/ethereum/execution-spec-tests/releases/tag/v3.0.0"
128+
in reason
129+
)
130+
131+
132+
class TestFixturesSourceFromInput:
133+
"""Test the from_input method without no_api_calls parameter."""
134+
135+
def test_from_input_handles_release_url(self):
136+
"""Test that from_input properly handles release URLs."""
137+
test_url = "https://github.com/ethereum/execution-spec-tests/releases/download/v3.0.0/fixtures_develop.tar.gz"
138+
139+
with patch.object(FixturesSource, "from_release_url") as mock_from_release_url:
140+
mock_from_release_url.return_value = MagicMock()
141+
142+
FixturesSource.from_input(test_url)
143+
144+
mock_from_release_url.assert_called_once_with(test_url)
145+
146+
def test_from_input_handles_release_spec(self):
147+
"""Test that from_input properly handles release specs."""
148+
test_spec = "stable@latest"
149+
150+
with patch.object(FixturesSource, "from_release_spec") as mock_from_release_spec:
151+
mock_from_release_spec.return_value = MagicMock()
152+
153+
FixturesSource.from_input(test_spec)
154+
155+
mock_from_release_spec.assert_called_once_with(test_spec)
156+
157+
def test_from_input_handles_regular_url(self):
158+
"""Test that from_input properly handles regular URLs."""
159+
test_url = "http://example.com/fixtures.tar.gz"
160+
161+
with patch.object(FixturesSource, "from_url") as mock_from_url:
162+
mock_from_url.return_value = MagicMock()
163+
164+
FixturesSource.from_input(test_url)
165+
166+
mock_from_url.assert_called_once_with(test_url)

0 commit comments

Comments
 (0)