You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/ci-release.md
+69Lines changed: 69 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -228,4 +228,73 @@ ex: Branch is named `develop` and the PR is numbered `113`
228
228
- `stacks-core:2.1.0.0.0`
229
229
- `stacks-core:latest`
230
230
231
+
## Mutation Testing
232
+
233
+
When a new Pull Request (PR) is submitted, this feature evaluates the quality of the tests added or modified in the PR. It checks the new and altered functions through mutation testing. Mutation testing involves making small changes (mutations) to the code to check if the tests can detect these changes. The mutations are run with or without a [Github Actions matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs). The matrix is used when there is a large number of mutations to run.
234
+
235
+
Since mutation testing is directly correlated to the written tests, there are slower packages (due to the quantity or time it takes to run the tests) like `stackslib` or `stacks-node`. These mutations are run separately from the others, with one or more parallel jobs, depending on the amount of mutations found.
236
+
237
+
Once all the jobs have finished testing mutants, the last job collects all the tested mutations from the previous jobs, combines them and outputs them to the `Summary` section of the workflow, at the bottom of the page. There, you can find all mutants on categories, with links to the function they tested, and a short description on how to fix the issue. The PR should only be approved/merged after all the mutants tested are in the `Caught` category.
- caught — A test failed with this mutant applied. This is a good sign about test coverage.
246
+
247
+
- missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. In any case, you may wish to add a better test.
248
+
249
+
- unviable — The attempted mutation doesn't compile. This is inconclusive about test coverage, since the function's return structure may not implement `Default::default()` (one of the mutations applied), hence causing the compile to fail. It is recommended to add `Default` implementation for the return structures of these functions, only mark that the function should be skipped as a last resort.
250
+
251
+
- timeout — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and only mark the function to be skipped if necessary.
252
+
253
+
### Skipping Mutations
254
+
255
+
Some functions may be inherently hard to cover with tests, for example if:
256
+
257
+
- Generated mutants cause tests to hang.
258
+
- You've chosen to test the functionality by human inspection or some higher-level integration tests.
259
+
- The function has side effects or performance characteristics that are hard to test.
260
+
- You've decided that the function is not important to test.
261
+
262
+
To mark functions as skipped, so they are not mutated:
263
+
264
+
- Add a Cargo dependency of the [mutants](https://crates.io/crates/mutants) crate, version `0.0.3` or later (this must be a regular `dependency`, not a `dev-dependency`, because the annotation will be on non-test code) and mark functions with `#[mutants::skip]`, or
265
+
266
+
- You can avoid adding the dependency by using the slightly longer `#[cfg_attr(test, mutants::skip)]`.
267
+
268
+
**Example:**
269
+
270
+
```rust
271
+
use std::time::{Duration, Instant};
272
+
273
+
/// Returns true if the program should stop
274
+
#[cfg_attr(test, mutants::skip)] // Returning false would cause a hang
275
+
fn should_stop() -> bool {
276
+
true
277
+
}
278
+
279
+
pub fn controlled_loop() {
280
+
let start = Instant::now();
281
+
for i in 0.. {
282
+
println!("{}", i);
283
+
if should_stop() {
284
+
break;
285
+
}
286
+
if start.elapsed() > Duration::from_secs(60 * 5) {
0 commit comments