You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/ci-release.md
+26-8Lines changed: 26 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -230,11 +230,23 @@ ex: Branch is named `develop` and the PR is numbered `113`
230
230
231
231
## Mutation Testing
232
232
233
-
When a new Pull Request (PR) is submitted, this feature evaluates the quality of the tests added or modified in the PR. It checks the new and altered functions through mutation testing. Mutation testing involves making small changes (mutations) to the code to check if the tests can detect these changes. The mutations are run with or without a [Github Actions matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs). The matrix is used when there is a large number of mutations to run.
233
+
When a new Pull Request (PR) is submitted, this feature evaluates the quality of the tests added or modified in the PR.
234
+
It checks the new and altered functions through mutation testing.
235
+
Mutation testing involves making small changes (mutations) to the code to check if the tests can detect these changes.
234
236
235
-
Since mutation testing is directly correlated to the written tests, there are slower packages (due to the quantity or time it takes to run the tests) like `stackslib` or `stacks-node`. These mutations are run separately from the others, with one or more parallel jobs, depending on the amount of mutations found.
237
+
The mutations are run with or without a [Github Actions matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs).
238
+
The matrix is used when there is a large number of mutations to run ([check doc specific cases](https://github.com/stacks-network/actions/blob/main/stacks-core/mutation-testing/check-packages-and-shards/README.md#outputs)).
239
+
We utilize a matrix strategy with shards to enable parallel execution in GitHub Actions.
240
+
This approach allows for the concurrent execution of multiple jobs across various runners.
241
+
The total workload is divided across all shards, effectively reducing the overall duration of a workflow because the time taken is approximately the total time divided by the number of shards (+ initial build & test time).
242
+
This is particularly advantageous for large packages that have significant build and test times, as it enhances efficiency and speeds up the process.
236
243
237
-
Once all the jobs have finished testing mutants, the last job collects all the tested mutations from the previous jobs, combines them and outputs them to the `Summary` section of the workflow, at the bottom of the page. There, you can find all mutants on categories, with links to the function they tested, and a short description on how to fix the issue. The PR should only be approved/merged after all the mutants tested are in the `Caught` category.
244
+
Since mutation testing is directly correlated to the written tests, there are slower packages (due to the quantity or time it takes to run the tests) like `stackslib` or `stacks-node`.
245
+
These mutations are run separately from the others, with one or more parallel jobs, depending on the amount of mutations found.
246
+
247
+
Once all the jobs have finished testing mutants, the last job collects all the tested mutations from the previous jobs, combines them and outputs them to the `Summary` section of the workflow, at the bottom of the page.
248
+
There, you can find all mutants on categories, with links to the function they tested, and a short description on how to fix the issue.
249
+
The PR should only be approved/merged after all the mutants tested are in the `Caught` category.
238
250
239
251
### Time required to run the workflow based on mutants outcome and packages' size
240
252
@@ -250,13 +262,19 @@ File:
250
262
251
263
### Mutant Outcomes
252
264
253
-
- caught — A test failed with this mutant applied. This is a good sign about test coverage.
265
+
- caught — A test failed with this mutant applied.
266
+
This is a good sign about test coverage.
254
267
255
-
- missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. In any case, you may wish to add a better test.
268
+
- missed — No test failed with this mutation applied, which seems to indicate a gap in test coverage.
269
+
Or, it may be that the mutant is undistinguishable from the correct code.
270
+
In any case, you may wish to add a better test.
256
271
257
-
- unviable — The attempted mutation doesn't compile. This is inconclusive about test coverage, since the function's return structure may not implement `Default::default()` (one of the mutations applied), hence causing the compile to fail. It is recommended to add `Default` implementation for the return structures of these functions, only mark that the function should be skipped as a last resort.
272
+
- unviable — The attempted mutation doesn't compile.
273
+
This is inconclusive about test coverage, since the function's return structure may not implement `Default::default()` (one of the mutations applied), hence causing the compile to fail.
274
+
It is recommended to add `Default` implementation for the return structures of these functions, only mark that the function should be skipped as a last resort.
258
275
259
-
- timeout — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and only mark the function to be skipped if necessary.
276
+
- timeout — The mutation caused the test suite to run for a long time, until it was eventually killed.
277
+
You might want to investigate the cause and only mark the function to be skipped if necessary.
260
278
261
279
### Skipping Mutations
262
280
@@ -273,7 +291,7 @@ To mark functions as skipped, so they are not mutated:
273
291
274
292
- You can avoid adding the dependency by using the slightly longer `#[cfg_attr(test, mutants::skip)]`.
0 commit comments