Skip to content

Commit 12388b7

Browse files
committed
feat: add documentation for mutation testing
1 parent 298e8e0 commit 12388b7

File tree

1 file changed

+69
-0
lines changed

1 file changed

+69
-0
lines changed

docs/ci-release.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,4 +228,73 @@ ex: Branch is named `develop` and the PR is numbered `113`
228228
- `stacks-core:2.1.0.0.0`
229229
- `stacks-core:latest`
230230

231+
## Mutation Testing
232+
233+
When a new Pull Request (PR) is submitted, this feature evaluates the quality of the tests added or modified in the PR. It checks the new and altered functions through mutation testing. Mutation testing involves making small changes (mutations) to the code to check if the tests can detect these changes. The mutations are run with or without a [Github Actions matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs). The matrix is used when there is a large number of mutations to run.
234+
235+
Since mutation testing is directly correlated to the written tests, there are slower packages (due to the quantity or time it takes to run the tests) like `stackslib` or `stacks-node`. These mutations are run separately from the others, with one or more parallel jobs, depending on the amount of mutations found.
236+
237+
Once all the jobs have finished testing mutants, the last job collects all the tested mutations from the previous jobs, combines them and outputs them to the `Summary` section of the workflow, at the bottom of the page. There, you can find all mutants on categories, with links to the function they tested, and a short description on how to fix the issue. The PR should only be approved/merged after all the mutants tested are in the `Caught` category.
238+
239+
File:
240+
241+
- [PR Differences Mutants](../.github/workflows/pr-differences-mutants.yml)
242+
243+
### Mutant Outcomes
244+
245+
- caught — A test failed with this mutant applied. This is a good sign about test coverage.
246+
247+
- missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. In any case, you may wish to add a better test.
248+
249+
- unviable — The attempted mutation doesn't compile. This is inconclusive about test coverage, since the function's return structure may not implement `Default::default()` (one of the mutations applied), hence causing the compile to fail. It is recommended to add `Default` implementation for the return structures of these functions, only mark that the function should be skipped as a last resort.
250+
251+
- timeout — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and only mark the function to be skipped if necessary.
252+
253+
### Skipping Mutations
254+
255+
Some functions may be inherently hard to cover with tests, for example if:
256+
257+
- Generated mutants cause tests to hang.
258+
- You've chosen to test the functionality by human inspection or some higher-level integration tests.
259+
- The function has side effects or performance characteristics that are hard to test.
260+
- You've decided that the function is not important to test.
261+
262+
To mark functions as skipped, so they are not mutated:
263+
264+
- Add a Cargo dependency of the [mutants](https://crates.io/crates/mutants) crate, version `0.0.3` or later (this must be a regular `dependency`, not a `dev-dependency`, because the annotation will be on non-test code) and mark functions with `#[mutants::skip]`, or
265+
266+
- You can avoid adding the dependency by using the slightly longer `#[cfg_attr(test, mutants::skip)]`.
267+
268+
**Example:**
269+
270+
```rust
271+
use std::time::{Duration, Instant};
272+
273+
/// Returns true if the program should stop
274+
#[cfg_attr(test, mutants::skip)] // Returning false would cause a hang
275+
fn should_stop() -> bool {
276+
true
277+
}
278+
279+
pub fn controlled_loop() {
280+
let start = Instant::now();
281+
for i in 0.. {
282+
println!("{}", i);
283+
if should_stop() {
284+
break;
285+
}
286+
if start.elapsed() > Duration::from_secs(60 * 5) {
287+
panic!("timed out");
288+
}
289+
}
290+
}
291+
292+
mod test {
293+
#[test]
294+
fn controlled_loop_terminates() {
295+
super::controlled_loop()
296+
}
297+
}
298+
```
299+
231300
---

0 commit comments

Comments
 (0)