Skip to content

Conversation

spencer-tb
Copy link
Contributor

@spencer-tb spencer-tb commented Aug 21, 2025

🗒️ Description

Problem

The coverage workflow's fill_prepatched_tests.sh script fails when tests cannot run due to fork compatibility issues. This occurs when:

  • Tests require future forks (Prague/Osaka) but the script is limited to --until=Cancun
  • All collected tests get filtered out at execution time, resulting in "no tests ran"
  • Script continues execution despite no fixtures being generated

This occurs in #1929 due to the small change in tests/prague/eip7702_set_code_tx/test_set_code_txs.py and via the removal of "!tests/prague/**" in coverage.yaml from #1970.

🔗 Related Issues or PRs

#1929

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

@spencer-tb spencer-tb added scope:ci Scope: Continuous Integration type:chore Type: Chore labels Aug 21, 2025
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, there's actually a specific return code for pytest when there were no tests collected: https://docs.pytest.org/en/stable/reference/exit-codes.html
But I'm not sure if we return this code from the fill command so this should be ok.

@marioevz
Copy link
Member

LGTM, there's actually a specific return code for pytest when there were no tests collected: https://docs.pytest.org/en/stable/reference/exit-codes.html But I'm not sure if we return this code from the fill command so this should be ok.

I just checked and we do actually return the same return code as pytest, then we could do:

uv run fill $MODIFIED_DELETED_FILES --clean --until=$FILL_UNTIL --evm-bin evmone-t8n --block-gas-limit $BLOCK_GAS_LIMIT -m "state_test or blockchain_test" --output $BASE_TEST_PATH > >(tee -a filloutput.log) 2> >(tee -a filloutput.log >&2)
FILL_RETURN_CODE=$?
if [ $FILL_RETURN_CODE -eq 5 ]; then
    echo "any_modified_fixtures=false" >> "$GITHUB_OUTPUT"
    exit 0
elif [ $FILL_RETURN_CODE -ne 0 ]; then
    echo "Error: failed to generate .py tests from before the PR."
    exit $FILL_RETURN_CODE
fi

@spencer-tb spencer-tb force-pushed the coverage-ignore-case branch from 0362b78 to 8a9ed05 Compare August 22, 2025 14:25
@spencer-tb
Copy link
Contributor Author

LGTM, there's actually a specific return code for pytest when there were no tests collected: https://docs.pytest.org/en/stable/reference/exit-codes.html But I'm not sure if we return this code from the fill command so this should be ok.

I just checked and we do actually return the same return code as pytest, then we could do:

uv run fill $MODIFIED_DELETED_FILES --clean --until=$FILL_UNTIL --evm-bin evmone-t8n --block-gas-limit $BLOCK_GAS_LIMIT -m "state_test or blockchain_test" --output $BASE_TEST_PATH > >(tee -a filloutput.log) 2> >(tee -a filloutput.log >&2)
FILL_RETURN_CODE=$?
if [ $FILL_RETURN_CODE -eq 5 ]; then
    echo "any_modified_fixtures=false" >> "$GITHUB_OUTPUT"
    exit 0
elif [ $FILL_RETURN_CODE -ne 0 ]; then
    echo "Error: failed to generate .py tests from before the PR."
    exit $FILL_RETURN_CODE
fi

Thanks! Updated with this. Removed the Prague bump as it requires us to update the evmone docker image.

@spencer-tb spencer-tb force-pushed the coverage-ignore-case branch from 8a9ed05 to cb2295c Compare August 22, 2025 14:27
@spencer-tb spencer-tb merged commit f9bfe4d into main Aug 22, 2025
1 check passed
@spencer-tb spencer-tb deleted the coverage-ignore-case branch August 22, 2025 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:ci Scope: Continuous Integration type:chore Type: Chore
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants