Setup mutation testing #686

staabm · 2025-10-04T11:46:08Z

requires

staabm · 2025-10-04T12:33:45Z

tests/Infection/TrinaryLogicMutator.php

+		return 'TrinaryLogicMutator';
+	}
+
+	public function canMutate(Node $node): bool


atm this class replaces any call to a method named yes, or no.
for now it is not scoped to the TrinaryLogic class.

atm this mutator works for these classes in the phpstan-src codebase - which I think is not a bad thing

staabm · 2025-10-04T12:34:21Z

current challenge: figure out a way to kill mutants across github action jobs - I will think about that

phpstan.neon

.github/workflows/platform-test.yml

ondrejmirtes

The test is failing sporadically after introducing the random order. It's probably because of:

phpstan-doctrine/tests/Rules/Doctrine/ORM/entity-manager.php

Lines 34 to 37 in 5eaf37b

    
           Type::overrideType( 
        
           	'date', 
        
           	DateTimeImmutableType::class, 
        
           );

(which sometimes gets executed before this test and sometimes not).

I guess the test that executes this should clean up after itself.

.github/workflows/build.yml

ondrejmirtes

Also I don't see PHPStan configured as mutant killer here.

ondrejmirtes · 2025-10-06T11:56:24Z

Makefile

+.PHONY: infection
+infection:
+	composer require --dev infection/infection -W --ignore-platform-req=ext-mongodb
+	vendor/bin/infection --logger-github=false


Why turning off the logger-github?

Also personally I wouldn't add this to Makefile. We'll need a more complex logic in the GHA job.

added it to makefile to easy running infection locally (to reproduce CI problems)

Why turning off the logger-github?

running infection in 3 php versions likely spams the 'files changed tab' with annotations.
I think we need a separate job which accumulates the separate job results and merges them togehter so we get a single final result with less false positives (mutation job on php 8.3 might error about a mutation which can only be killed in the php 8.1 test-suite)

mutation job on php 8.3 might error about a mutation which can only be killed in the php 8.1 test-suite

I had this concern as well but I talked to Maks about this and given that Infection would only mutate covered lines then this shouldn't happen. If a test ran on PHP 8.3 covers a line and we mutate that line, it should fail the test on 8.3.

it should fail the test on 8.3.

but the very same mutation will happen on 8.1/8.2 and no test will kill the mutation (in case it is a PHP 8.3 only test)

If this was really true Infection would be unusable for us. We'd have to be able to invoke each phase separately and modify the results before feeding them to the next phase. (Run tests on each PHP version, feed the data into Infection, let it mutate the sources, run tests on each PHP version again ourselves and so on.)

I think we could run infection on multiple php versions in parallel.. collect the mutations afterwards in a new followup github action and report only those as a problem, which were not killed in any of the previous jobs.
(as long as we do not do source transformations, the mutations should be the same on all 3 jobs)

alternatively we could run infection on a single php version only

ondrejmirtes · 2025-10-06T12:00:27Z

Some comments posted by me disappeared. I linked the --git-diff-filter / --git-diff-base / --git-diff-lines options

ondrejmirtes

The PHAR distribution is the recommended one for installing Infection. I think we could just add infection to tools in setup-php action.

staabm · 2025-10-06T14:26:04Z

Some comments posted by me disappeared. I linked the --git-diff-filter / --git-diff-base / --git-diff-lines options

yeah, I did not yet activate these options, because then there would be no results reported in this PR.

staabm · 2025-10-06T14:28:36Z

The PHAR distribution is the recommended one for installing Infection. I think we could just add infection to tools in setup-php action.

I think this only make sense in case we don't want run infection locally. the more we use GitHub Actions only stuff, the harder it is to run the very same setup on a local computer.

staabm · 2025-10-06T14:32:12Z

The test is failing sporadically after introducing the random order. It's probably because of:

I did not yet see a test, which started failling more often since random order was enabled.
which particular test do you mean?

the build on the master branch is broken long before random order with the same errors

ondrejmirtes · 2025-10-06T15:21:51Z

I think this only make sense in case we don't want run infection locally.

Yeah, I think we don't. Given the support for many PHP versions (and that we only typically have a single PHP version running locally), I think it's unlikely we'd run Infection locally.

Also I'm thinking we'll be passing GitHub Actions-specific env variables to the CLI options anyway.

ondrejmirtes · 2025-10-06T15:25:26Z

Why is the build green even when there are escaped mutants?

staabm · 2025-10-06T15:27:38Z

Why is the build green even when there are escaped mutants?

because we did not yet define a "min MSI", meaning there is no minimum threshold yet.
if we want the test to fail when at least a single escaped mutant occurs, we need min-msi=100

ondrejmirtes · 2025-10-06T15:28:23Z

Yeah, I want that, alongside the Git-filtering options.

We can demonstrate the failing by adding some new change here in this PR temporarily.

staabm · 2025-10-06T16:15:52Z

Also I don't see PHPStan configured as mutant killer here.

I think, as long as we only mutate with TrinaryLogicMutator its pretty unlikely that the PHPStan killer will help us.
in addition the process is already slow and adding the PHPStan killer to the mix will make it a even slower

ondrejmirtes · 2025-10-06T16:17:36Z

I really want it though. I don't want to write tests for things that would error in PHPStan. Also we'd have more incentives to make it faster 😊

staabm · 2025-10-06T20:11:03Z

.github/workflows/build.yml

+  mutation-testing:
+    name: "Mutation Testing"
+    runs-on: "ubuntu-latest"
+    needs: ["tests", "static-analysis"]


Phpunit killer requires a green test run
Phpstan killer requires a green phpstan run

ondrejmirtes · 2025-10-06T20:37:34Z

Make sure to carry over PHPStan's result cache, probably with upload and download-artifact actions. It should speed up the build significantly.

.github/workflows/build.yml

staabm · 2025-10-07T10:01:03Z

@maks-rafalko I wonder why infection treats this PHPStan result as a killed mutant?
it reads like PHPStan did not report an error at all, so I would expect a escaped mutant.

infection.log:

Escaped mutants:
================

Timed Out mutants:
==================

Skipped mutants:
================

Killed by Test Framework mutants:
=================================

Killed by Static Analysis mutants:
==================================

1) /home/runner/work/phpstan-doctrine/phpstan-doctrine/src/Rules/Doctrine/ORM/QueryBuilderDqlRule.php:74    [M] TrinaryLogicMutator [ID] e7038292ed4647a2b1851489639d8e24

@@ @@
         }
         // testing stuff
         $obj = (new ObjectType('Doctrine\ORM\QueryBuilder'))->isSuperTypeOf($calledOnType);
-        if ($obj->yes()) {
+        if (!$obj->no()) {
             $x = 1;
         } else {
             $x = 2;

$ '/home/runner/work/phpstan-doctrine/phpstan-doctrine/vendor/bin/phpstan' '--tmp-file=/tmp/infection/mutant.e7038292ed4647a2b1851489639d8e24.infection.php' '--instead-of=/home/runner/work/phpstan-doctrine/phpstan-doctrine/src/Rules/Doctrine/ORM/QueryBuilderDqlRule.php' '--configuration=/tmp/infection/phpstan.e7038292ed4647a2b1851489639d8e24.infection.neon' '--error-format=json' '--no-progress' '-vv' '--fail-without-result-cache'
  {"totals":{"errors":0,"file_errors":0},"files":{},"errors":[]}
  
  Result cache not used because extension file /home/runner/work/phpstan-doctrine/phpstan-doctrine/src/Rules/Doctrine/ORM/QueryBuilderDqlRule.php hash does not match.
  Result cache was not saved because of --tmp-file and --instead-of CLI options passed (editor mode).
  Elapsed time: 29 seconds
  Used memory: 234.5 MB


Errors mutants:
===============

Syntax Errors mutants:
======================

maks-rafalko · 2025-10-07T10:06:16Z

@maks-rafalko I wonder why infection treats this PHPStan result as a killed mutant?

I would debug the exit code. Here is the logic how Infection determines if it's killed or escaped for PHPStan process output.

https://github.com/infection/infection/blob/466255c58cdc6dbe307012cc077b9088ff3cfaa5/src/StaticAnalysis/PHPStan/Mutant/PHPStanMutantExecutionResultFactory.php#L94

We discussed with @ondrejmirtes that checking non-zero exit code is not reliable, but AFAIK nothing has been implemented on PHPStan side to improve it, so we still check (non-)zero exit code.

staabm · 2025-10-07T10:21:24Z

local debugging reveals: we get a exit code of 2, because infection internally passes --fail-without-result-cache and the result cache is not used as the PHPStan output describes:

Result cache not used because extension file /Users/m.staab/dvl/phpstan-doctrine/src/Rules/Doctrine/ORM/QueryBuilderDqlRule.php hash does not match.
Result cache was not saved because of --tmp-file and --instead-of CLI options passed (editor mode).

ondrejmirtes · 2025-10-07T10:23:58Z

Yeah, that's tricky. I suppose we should have a special exit code for when the result is green but result cache was not used (so that Infection does not use it to kill mutants).

Or maybe Infection can stop passing --fail-without-result-cache altogether.

staabm · 2025-10-07T10:26:51Z

Or maybe Infection can stop passing --fail-without-result-cache altogether.

I think thats the way to go. not everyone has the tools to use a result cache in CI.

maks-rafalko · 2025-10-07T10:28:10Z

Or maybe Infection can stop passing --fail-without-result-cache altogether.

I remember I've added it by @staabm's suggestion to avoid the cases when for some (unexpected) reason we miss the cache, so for me this option did make sense so far.

Can I ask you why do we have here

Result cache not used because extension file /src/Rules/Doctrine/ORM/QueryBuilderDqlRule.php hash does not match.

Is it the issue that can be fixed here? Or is it expected?

staabm · 2025-10-07T10:33:24Z

Can I ask you why do we have here

Result cache not used because extension file /src/Rules/Doctrine/ORM/QueryBuilderDqlRule.php hash does not match.

Is it the issue that can be fixed here? Or is it expected?

my guess is that this problem occurs because phpstan-doctrine is a phpstan extension repository (regular non phpstan extension projects would not have this case if I got it right)

staabm · 2025-10-07T10:35:22Z

I remember I've added it by @staabm's suggestion to avoid the cases when for some (unexpected) reason we miss the cache, so for me this option did make sense so far.

ohh I remember that.. while running infection, we actually have a primed cache (caused by the initial test-run). so we don't need CI tooling for the cache to work.

ondrejmirtes · 2025-10-07T10:37:43Z

This can happen in any project with custom rules or other extensions. When Infection actually decides to mutate an extension file, it's inevitable the result cache won't be used.

maks-rafalko · 2025-10-07T11:26:43Z

This can happen in any project with custom rules or other extensions. When Infection actually decides to mutate an extension file, it's inevitable the result cache won't be used.

so do you say here we have a cache miss issue because Infection mutates the extension's file?
does it mean we need to remove passing --fail-without-result-cache by Infection?

I'm not sure I understand all the cases to make a decision.

ondrejmirtes · 2025-10-07T11:33:57Z

I think what we're facing here is that we want to make sure the user primed the PHPStan cache before running Infection. But it can still be invalidated in these cases.

--fail-without-result-cache can sometimes fill this purpose but it's not perfect.

staabm commented Oct 4, 2025

View reviewed changes

staabm mentioned this pull request Oct 5, 2025

failling CI build #688

Open

ondrejmirtes requested changes Oct 6, 2025

View reviewed changes

phpstan.neon Show resolved Hide resolved

.github/workflows/platform-test.yml Outdated Show resolved Hide resolved

ondrejmirtes requested changes Oct 6, 2025

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

ondrejmirtes requested changes Oct 6, 2025

View reviewed changes

staabm mentioned this pull request Oct 6, 2025

Run tests using global type overrides in isolation #692

Merged

staabm force-pushed the infect branch from 15986a1 to 08d77e1 Compare October 6, 2025 15:52

staabm commented Oct 6, 2025

View reviewed changes

staabm force-pushed the infect branch from 97dbece to 49b7131 Compare October 7, 2025 06:23

staabm mentioned this pull request Oct 7, 2025

Move phpstan analyse paths into phpstan.neon #694

Merged

staabm added 7 commits October 7, 2025 11:29

Setup mutation testing

8a4e7cd

Update platform-test.yml

65db2c8

fix built

d3a6e60

fix build

69b69ca

run infection in all test jobs

baab8a3

Update platform-test.yml

b44bb9c

Update platform-test.yml

d223186

staabm added 10 commits October 7, 2025 11:29

Update build.yml

e509b9a

use local temp

2eecbb9

unique artifact name

704bce2

Update build.yml

19706cb

Update build.yml

2caea8d

fix

9f9de8f

Update infection.json5

c4d823e

Update infection.json5

93d3d97

Update infection.json5

01a9197

simplify

58e765a

clxmstaab force-pushed the infect branch from d8ea57f to 58e765a Compare October 7, 2025 09:29

staabm added 4 commits October 7, 2025 11:36

increase timeout

378e518

Update infection.json5

b3b8b99

upload infection log as artifact to ease debugging

67cf382

Update build.yml

c0e5252

maks-rafalko reviewed Oct 7, 2025

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

enable debugging

d954a45

pin infection version

13ad641

update to infection:0.31.3

3bddff7

Setup mutation testing #686

Are you sure you want to change the base?

Setup mutation testing #686

Uh oh!

Conversation

staabm commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

staabm commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ondrejmirtes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ondrejmirtes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

staabm Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ondrejmirtes commented Oct 6, 2025

Uh oh!

ondrejmirtes left a comment

Choose a reason for hiding this comment

Uh oh!

staabm commented Oct 6, 2025

Uh oh!

staabm commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

staabm commented Oct 6, 2025

Uh oh!

ondrejmirtes commented Oct 6, 2025

Uh oh!

ondrejmirtes commented Oct 6, 2025

Uh oh!

staabm commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ondrejmirtes commented Oct 6, 2025

Uh oh!

staabm commented Oct 6, 2025

Uh oh!

ondrejmirtes commented Oct 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ondrejmirtes commented Oct 6, 2025

Uh oh!

Uh oh!

staabm commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maks-rafalko commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

staabm commented Oct 7, 2025

Uh oh!

staabm commented Oct 4, 2025 •

edited

Loading

staabm commented Oct 4, 2025 •

edited

Loading

staabm Oct 6, 2025 •

edited

Loading

staabm commented Oct 6, 2025 •

edited

Loading

staabm commented Oct 6, 2025 •

edited

Loading

staabm commented Oct 7, 2025 •

edited

Loading

maks-rafalko commented Oct 7, 2025 •

edited

Loading

maks-rafalko commented Oct 7, 2025 •

edited

Loading