Skip to content

feat(cli,fill): make the eip version checker a separate pytest cli tool#1537

Merged
danceratopz merged 25 commits intomainfrom
move-eip-version-check-to-a-separate-tool
May 6, 2025
Merged

feat(cli,fill): make the eip version checker a separate pytest cli tool#1537
danceratopz merged 25 commits intomainfrom
move-eip-version-check-to-a-separate-tool

Conversation

@danceratopz
Copy link
Member

@danceratopz danceratopz commented May 5, 2025

🗒️ Description

  • Don't load the pytest_plugins.spec_version_checker.spec_version_checker plugin as part of fill or execute.
  • Allow specification of a Github PAT to avoid Github API rate limiting.
  • Create a new separate pytest-based CLI check_eip_versions that only performs EIP spec version checking in a pytest session.
  • Add a new Github Action to run check_eip_versions daily and create an issue upon failure (example issues chore(tests): eip spec references outdated #1553 (prague mismatches only) and chore(tests): eip spec references outdated #1552 (all forks).

Note

This PR unfortunately adds a new pytest ini pytest-check-eip-versions.ini which is essentially a duplicate of fill's pytest.ini. This is an ugly code duplication, that will likely cause check_eip_versions to break if fill's options get changed w/o changing the check_eip_versions pytest ini. The issue is that the pytest_plugins.spec_version_checker.spec_version_checker needs to be loaded first, which doesn't seem to be possible if used in conjunction with -c pytest.ini.

🔗 Related Issues

Fixes #533, #1552, #1553

Todo before merge

  • Update docs (adding a PAT, etc).
  • Update references.
  • Remove "on pull request" / "on push" build triggers in action (testing only).

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.

@danceratopz danceratopz added scope:pytest Scope: Changes EEST's pytest plugins type:feat type: Feature scope:fill Scope: fill command scope:cli Scope: Changes to `ethereum_clis` labels May 5, 2025
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left just one comment regarding the frequency of the runs for the new CI workflow.

@spencer-tb
Copy link
Collaborator

Tagging this comment, as another alternative to prevent rate limiting! I really think we should use it. Essentially a single endpoint call that gets all the EIP sha's at once! Check it out :)

https://api.github.com/repos/ethereum/EIPs/git/trees/master?recursive=1

#1357 (review)

@danceratopz
Copy link
Member Author

danceratopz commented May 6, 2025

Tagging this comment, as another alternative to prevent rate limiting! I really think we should use it. Essentially a single endpoint call that gets all the EIP sha's at once! Check it out :)

https://api.github.com/repos/ethereum/EIPs/git/trees/master?recursive=1

#1357 (review)

That's a nice solution, but I'm out of steam for this topic/PR. For now this (or using a env var) is working nicely:

uv run check_eip_versions --github-token=(gh auth token)

Tbh, the plugin should be re-written to not collect the evm python test cases at all (it currently collects ~50,000 test cases). I think it should be possible to simply discover test modules (containing "eip") and then create the eip version check test.

Feel free to PR your suggestion, if you fancy. Otherwise, we collect these ideas into a follow-up issue.

@spencer-tb
Copy link
Collaborator

Created an issue :)

Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@danceratopz danceratopz merged commit 846e462 into main May 6, 2025
22 checks passed
@danceratopz danceratopz deleted the move-eip-version-check-to-a-separate-tool branch May 6, 2025 15:21
felix314159 pushed a commit to felix314159/execution-spec-tests that referenced this pull request May 16, 2025
kclowes pushed a commit to kclowes/execution-spec-tests that referenced this pull request Oct 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope:cli Scope: Changes to `ethereum_clis` scope:fill Scope: fill command scope:pytest Scope: Changes EEST's pytest plugins type:feat type: Feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(fill): fix misleading test collection reporting in terminal output

3 participants