Skip to content

Commit 530b689

Browse files
ASuciuXASuciuX
committed
docs & script for running cargo mutants locally on CI limitation
1 parent 05280cd commit 530b689

File tree

2 files changed

+193
-0
lines changed

2 files changed

+193
-0
lines changed

‎contrib/tools/local-mutation-testing.sh

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
# Install cargo-mutants
6+
cargo install --version 24.7.1 cargo-mutants --locked
7+
8+
# Create diff file between current branch and develop branch
9+
git diff origin/develop...HEAD > git.diff
10+
11+
# Remove git diff files about removed/renamed files
12+
awk '
13+
/^diff --git/ {
14+
diff_line = $0
15+
getline
16+
if ($0 !~ /^(deleted file mode|similarity index)/) {
17+
print diff_line
18+
print
19+
}
20+
}
21+
!/^(diff --git|deleted file mode|similarity index|rename from|rename to)/ {print}
22+
' git.diff > processed.diff
23+
24+
# Extract mutants based on the processed diff
25+
cargo mutants --in-diff processed.diff --list > all_mutants.txt
26+
27+
# Create a directory for organizing mutants
28+
mkdir -p mutants_by_package
29+
30+
# Organize mutants into files based on their main folder
31+
while IFS= read -r line; do
32+
package=$(echo "$line" | cut -d'/' -f1)
33+
34+
case $package in
35+
"stackslib")
36+
echo "$line" >> "mutants_by_package/stackslib.txt"
37+
;;
38+
"testnet")
39+
echo "$line" >> "mutants_by_package/stacks-node.txt"
40+
;;
41+
"stacks-signer")
42+
echo "$line" >> "mutants_by_package/stacks-signer.txt"
43+
;;
44+
*)
45+
echo "$line" >> "mutants_by_package/small-packages.txt"
46+
;;
47+
esac
48+
done < all_mutants.txt
49+
50+
# Function to run mutants for a package
51+
run_mutants() {
52+
local package=$1
53+
local threshold=$2
54+
local output_dir=$3
55+
local mutant_file="mutants_by_package/${package}.txt"
56+
57+
if [ ! -f "$mutant_file" ]; then
58+
echo "No mutants found for $package"
59+
return
60+
fi
61+
62+
local regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "$mutant_file" | paste -sd'|' -)
63+
local mutant_count=$(cargo mutants -F "$regex_pattern" -E ": replace .{1,2} with .{1,2} in " --list | wc -l)
64+
65+
if [ "$mutant_count" -gt "$threshold" ]; then
66+
echo "Running mutants for $package ($mutant_count mutants)"
67+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
68+
-F "$regex_pattern" \
69+
-E ": replace .{1,2} with .{1,2} in " \
70+
--output "$output_dir" \
71+
--test-tool=nextest \
72+
--package "$package" \
73+
-- --all-targets --test-threads 1
74+
75+
echo $? > "${output_dir}/exit_code.txt"
76+
else
77+
echo "Skipping $package, only $mutant_count mutants (threshold: $threshold)"
78+
fi
79+
}
80+
81+
# Run mutants for each wanted package
82+
run_mutants "stacks-signer" 500 "./stacks-signer_mutants"
83+
run_mutants "stacks-node" 540 "./stacks-node_mutants"
84+
run_mutants "stackslib" 72 "./stackslib_mutants"

‎docs/mutation-testing.md

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Mutation Testing
2+
3+
This document describes how to run mutation testing locally to mimic the outcome of a PR, without the CI limitation it provides by timing out after 6 hours.
4+
[Here is the script](../contrib/tools/local-mutation-testing.sh) to run mutation locally running the mutants created by the changes between the current branch and develop. It does automatically all the steps explained below.
5+
6+
From the root level of the stacks-core repository run
7+
```sh
8+
./contrib/tools/local-mutation-testing.sh
9+
```
10+
11+
## Prerequirements
12+
13+
Install the cargo mutants library
14+
```sh
15+
cargo install --version 24.7.1 cargo-mutants --locked
16+
```
17+
18+
19+
## Steps
20+
1. be on source branch you would use for the PR.
21+
2. create diff file comparing this branch with the `develop` branch
22+
```sh
23+
git diff origin/develop..HEAD > git.diff
24+
```
25+
3. clean up the diff file and create auxiliary files
26+
```sh
27+
awk '
28+
/^diff --git/ {
29+
diff_line = $0
30+
getline
31+
if ($0 !~ /^(deleted file mode|similarity index)/) {
32+
print diff_line
33+
print
34+
}
35+
}
36+
!/^(diff --git|deleted file mode|similarity index|rename from|rename to)/ {print}
37+
' git.diff > processed.diff
38+
39+
# Extract mutants based on the processed diff
40+
cargo mutants --in-diff processed.diff --list > all_mutants.txt
41+
42+
# Create a directory for organizing mutants
43+
mkdir -p mutants_by_package
44+
45+
# Organize mutants into files based on their main folder
46+
while IFS= read -r line; do
47+
package=$(echo "$line" | cut -d'/' -f1)
48+
49+
case $package in
50+
"stackslib")
51+
echo "$line" >> "mutants_by_package/stackslib.txt"
52+
;;
53+
"testnet")
54+
echo "$line" >> "mutants_by_package/stacks-node.txt"
55+
;;
56+
"stacks-signer")
57+
echo "$line" >> "mutants_by_package/stacks-signer.txt"
58+
;;
59+
*)
60+
echo "$line" >> "mutants_by_package/small-packages.txt"
61+
;;
62+
esac
63+
done < all_mutants.txt
64+
```
65+
4. based on the package required to run the mutants for
66+
a. stackslib package
67+
```sh
68+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/stackslib.txt" | paste -sd'|' -)
69+
70+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
71+
-F "$regex_pattern" \
72+
-E ": replace .{1,2} with .{1,2} in " \
73+
--output "./stackslib_mutants" \
74+
--test-tool=nextest \
75+
-- --all-targets --test-threads 1
76+
```
77+
b. stacks-node (testnet) package
78+
```sh
79+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/testnet.txt" | paste -sd'|' -)
80+
81+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
82+
-F "$regex_pattern" \
83+
-E ": replace .{1,2} with .{1,2} in " \
84+
--output "./testnet_mutants" \
85+
--test-tool=nextest \
86+
-- --all-targets --test-threads 1
87+
```
88+
c. stacks-signer
89+
```sh
90+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/stacks-signer.txt" | paste -sd'|' -)
91+
92+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
93+
-F "$regex_pattern" \
94+
-E ": replace .{1,2} with .{1,2} in " \
95+
--output "./stacks-signer_mutants" \
96+
--test-tool=nextest \
97+
-- --all-targets --test-threads 1
98+
```
99+
d. all other packages combined
100+
```sh
101+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/small-packages.txt" | paste -sd'|' -)
102+
103+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
104+
-F "$regex_pattern" \
105+
-E ": replace .{1,2} with .{1,2} in " \
106+
--output "./small-packages_mutants" \
107+
--test-tool=nextest \
108+
-- --all-targets --test-threads 1
109+
```

0 commit comments

Comments
 (0)