Skip to content

Commit c16d87c

Browse files
committed
feat: update mutants doc format & some extra context
1 parent 0875d10 commit c16d87c

File tree

2 files changed

+26
-9
lines changed

2 files changed

+26
-9
lines changed

.github/workflows/pr-differences-mutants.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,6 @@ jobs:
120120

121121
runs-on: ubuntu-latest
122122

123-
if: always()
124123
needs:
125124
[
126125
check-big-packages-and-shards,

docs/ci-release.md

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -230,11 +230,23 @@ ex: Branch is named `develop` and the PR is numbered `113`
230230

231231
## Mutation Testing
232232

233-
When a new Pull Request (PR) is submitted, this feature evaluates the quality of the tests added or modified in the PR. It checks the new and altered functions through mutation testing. Mutation testing involves making small changes (mutations) to the code to check if the tests can detect these changes. The mutations are run with or without a [Github Actions matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs). The matrix is used when there is a large number of mutations to run.
233+
When a new Pull Request (PR) is submitted, this feature evaluates the quality of the tests added or modified in the PR.
234+
It checks the new and altered functions through mutation testing.
235+
Mutation testing involves making small changes (mutations) to the code to check if the tests can detect these changes.
234236

235-
Since mutation testing is directly correlated to the written tests, there are slower packages (due to the quantity or time it takes to run the tests) like `stackslib` or `stacks-node`. These mutations are run separately from the others, with one or more parallel jobs, depending on the amount of mutations found.
237+
The mutations are run with or without a [Github Actions matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs).
238+
The matrix is used when there is a large number of mutations to run ([check doc specific cases](https://github.com/stacks-network/actions/blob/main/stacks-core/mutation-testing/check-packages-and-shards/README.md#outputs)).
239+
We utilize a matrix strategy with shards to enable parallel execution in GitHub Actions.
240+
This approach allows for the concurrent execution of multiple jobs across various runners.
241+
The total workload is divided across all shards, effectively reducing the overall duration of a workflow because the time taken is approximately the total time divided by the number of shards (+ initial build & test time).
242+
This is particularly advantageous for large packages that have significant build and test times, as it enhances efficiency and speeds up the process.
236243

237-
Once all the jobs have finished testing mutants, the last job collects all the tested mutations from the previous jobs, combines them and outputs them to the `Summary` section of the workflow, at the bottom of the page. There, you can find all mutants on categories, with links to the function they tested, and a short description on how to fix the issue. The PR should only be approved/merged after all the mutants tested are in the `Caught` category.
244+
Since mutation testing is directly correlated to the written tests, there are slower packages (due to the quantity or time it takes to run the tests) like `stackslib` or `stacks-node`.
245+
These mutations are run separately from the others, with one or more parallel jobs, depending on the amount of mutations found.
246+
247+
Once all the jobs have finished testing mutants, the last job collects all the tested mutations from the previous jobs, combines them and outputs them to the `Summary` section of the workflow, at the bottom of the page.
248+
There, you can find all mutants on categories, with links to the function they tested, and a short description on how to fix the issue.
249+
The PR should only be approved/merged after all the mutants tested are in the `Caught` category.
238250

239251
### Time required to run the workflow based on mutants outcome and packages' size
240252

@@ -250,13 +262,19 @@ File:
250262

251263
### Mutant Outcomes
252264

253-
- caught — A test failed with this mutant applied. This is a good sign about test coverage.
265+
- caught — A test failed with this mutant applied.
266+
This is a good sign about test coverage.
254267

255-
- missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. In any case, you may wish to add a better test.
268+
- missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage.
269+
Or, it may be that the mutant is undistinguishable from the correct code.
270+
In any case, you may wish to add a better test.
256271

257-
- unviable — The attempted mutation doesn't compile. This is inconclusive about test coverage, since the function's return structure may not implement `Default::default()` (one of the mutations applied), hence causing the compile to fail. It is recommended to add `Default` implementation for the return structures of these functions, only mark that the function should be skipped as a last resort.
272+
- unviable — The attempted mutation doesn't compile.
273+
This is inconclusive about test coverage, since the function's return structure may not implement `Default::default()` (one of the mutations applied), hence causing the compile to fail.
274+
It is recommended to add `Default` implementation for the return structures of these functions, only mark that the function should be skipped as a last resort.
258275

259-
- timeout — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and only mark the function to be skipped if necessary.
276+
- timeout — The mutation caused the test suite to run for a long time, until it was eventually killed.
277+
You might want to investigate the cause and only mark the function to be skipped if necessary.
260278

261279
### Skipping Mutations
262280

@@ -273,7 +291,7 @@ To mark functions as skipped, so they are not mutated:
273291

274292
- You can avoid adding the dependency by using the slightly longer `#[cfg_attr(test, mutants::skip)]`.
275293

276-
**Example:**
294+
### Example
277295

278296
```rust
279297
use std::time::{Duration, Instant};

0 commit comments

Comments
 (0)