Skip to content

Run mutation tests in CI for changed files #1364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: forks/osaka
Choose a base branch
from
Open

Run mutation tests in CI for changed files #1364

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
349e909
run mutmut on changed files
souradeep-das Aug 8, 2025
5942527
style fix
souradeep-das Aug 8, 2025
6cdcdaa
fix review comments
souradeep-das Aug 13, 2025
3ff814d
update test run
souradeep-das Aug 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions .github/workflows/test.yaml
View file
Open in desktop
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mutation tests might take a while to run - so I'm wondering how we should run them - per PR or commit on main

If several files have changes the time to run on CI also might be unrealistic. In which case I was thinking if we should add more limitations on when to run these tests (or patch mutmut to be able to run mutation tests for specific functions (as opposed to files).
Attached a test run on the main comment ^

Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,105 @@ jobs:
- uses: ./.github/actions/setup-env
- name: Run optimized tests
run: tox -e optimized

changed-forks:
runs-on: ubuntu-latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
runs-on: ubuntu-latest
runs-on: [self-hosted-ghr, size-xl-x64]

This is the magic sauce that runs the job on a beefier machine.

needs: static
outputs:
changed-forks: ${{ steps.filter-tests.outputs.forks }}
fork-mapping: ${{ steps.filter-tests.outputs.fork_mapping }}
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install project for fork detection
run: pip install .
- name: Find changed valid forks
id: set-forks
run: |
git fetch origin ${{ github.event.pull_request.base.ref }}
VALID_FORKS=$(python -c "import json; from ethereum_spec_tools.forks import Hardfork; print(json.dumps([f.short_name for f in Hardfork.discover()]))")
CHANGED_FORKS=$(git diff --name-only origin/${{ github.event.pull_request.base.ref }} ${{ github.sha }} -- src/ethereum/ | awk -F'/' '{print $3}' | sort -u)
readarray -t VALID_FORKS_ARR < <(echo "$VALID_FORKS" | jq -r '.[]')
FILTERED=$(for f in $CHANGED_FORKS; do for v in "${VALID_FORKS_ARR[@]}"; do [ "$f" = "$v" ] && echo $f; done; done | jq -R . | jq -s -c .)
echo "FILTERED (final forks for matrix): $FILTERED"
echo "forks=$FILTERED" >> $GITHUB_OUTPUT
- name: Filter out forks without tests
id: filter-tests
run: |
FORK_DATA=$(python -c "
import sys
sys.path.insert(0, 'tests')
from json_infra import FORKS
import json
eels_to_json = {}
for json_fork, config in FORKS.items():
eels_to_json[config['eels_fork']] = json_fork
print(json.dumps(eels_to_json))
")

AVAILABLE_EELS_FORKS=$(echo "$FORK_DATA" | jq -r 'keys[]')
FILTERED_WITH_TESTS=$(for f in $(echo '${{ steps.set-forks.outputs.forks }}' | jq -r '.[]'); do
if echo "$AVAILABLE_EELS_FORKS" | grep -q "^$f$"; then
echo $f
fi
done | jq -R . | jq -s -c .)
echo "FILTERED_WITH_TESTS: $FILTERED_WITH_TESTS"
echo "forks=$FILTERED_WITH_TESTS" >> $GITHUB_OUTPUT
echo "fork_mapping=$FORK_DATA" >> $GITHUB_OUTPUT

mutation-test:
runs-on: [self-hosted-ghr, size-xl-x64]
needs: [static, changed-forks]
if: github.event_name == 'pull_request'
strategy:
matrix:
fork: ${{ fromJson(needs.changed-forks.outputs.changed-forks) }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install project with test dependencies
run: pip install '.[test]'
- name: Install mutmut
run: pip install git+https://github.com/souradeep-das/mutmut.git@cc99e509092ed232600712d9d77fb38be31f74ec
- name: Find changed files for fork
id: changed
run: |
git fetch origin ${{ github.event.pull_request.base.ref }}
echo "changed_files<<EOF" >> $GITHUB_OUTPUT
(git diff --name-only origin/${{ github.event.pull_request.base.ref }} ${{ github.sha }} -- src/ethereum/${{ matrix.fork }}/ || true) >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
- name: Display changed files
run: |
echo "Changed files:"
echo "${{ steps.changed.outputs.changed_files }}"
- name: Skip if no changes for this fork
if: steps.changed.outputs.changed_files == ''
run: echo "No changes for ${{ matrix.fork }}, skipping mutmut."
- name: Copy dependencies for mutation testing
if: steps.changed.outputs.changed_files != ''
run: |
mkdir -p mutants/src
cp -r src/* mutants/src/
- name: Determine JSON fork name
if: steps.changed.outputs.changed_files != ''
id: json-fork
run: |
JSON_FORK=$(echo '${{ needs.changed-forks.outputs.fork-mapping }}' | jq -r '."${{ matrix.fork }}"')
echo "json_fork=$JSON_FORK" >> $GITHUB_OUTPUT
echo "Found JSON fork name: $JSON_FORK for eels fork: ${{ matrix.fork }}"
- name: Run mutmut
if: steps.changed.outputs.changed_files != ''
run: |
FILES=$(echo "${{ steps.changed.outputs.changed_files }}" | paste -sd, -)
mutmut --paths-to-mutate $FILES --tests-dir tests/json_infra/ --pytest-extra-args '-m "not slow and not angry_mutant" --ignore-glob=tests/json_infra/fixtures/* --fork ${{ steps.json-fork.outputs.json_fork }}' --debug run
- name: Show mutmut results
if: steps.changed.outputs.changed_files != ''
run: mutmut results