Skip to content

Commit 4867505

Browse files
kantaikantai
authored
Merge pull request #5057 from stacks-network/docs/local-mutantion-testing
docs & script for running cargo mutants locally on CI limitation
2 parents beeebe6 + 5cc974e commit 4867505

File tree

2 files changed

+233
-0
lines changed

2 files changed

+233
-0
lines changed

‎contrib/tools/local-mutation-testing.sh

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
# Install cargo-mutants
6+
cargo install --version 24.7.1 cargo-mutants --locked
7+
8+
# Create diff file between current branch and develop branch
9+
git diff origin/develop...HEAD > git.diff
10+
11+
# Remove git diff files about removed/renamed files
12+
awk '
13+
/^diff --git/ {
14+
diff_line = $0
15+
getline
16+
if ($0 !~ /^(deleted file mode|similarity index)/) {
17+
print diff_line
18+
print
19+
}
20+
}
21+
!/^(diff --git|deleted file mode|similarity index|rename from|rename to)/ {print}
22+
' git.diff > processed.diff
23+
24+
# Extract mutants based on the processed diff
25+
cargo mutants --in-diff processed.diff --list > all_mutants.txt
26+
27+
# Create a directory for organizing mutants
28+
mkdir -p mutants_by_package
29+
30+
# Organize mutants into files based on their main folder
31+
while IFS= read -r line; do
32+
package=$(echo "$line" | cut -d'/' -f1)
33+
34+
case $package in
35+
"stackslib")
36+
echo "$line" >> "mutants_by_package/stackslib.txt"
37+
;;
38+
"testnet")
39+
echo "$line" >> "mutants_by_package/stacks-node.txt"
40+
;;
41+
"stacks-signer")
42+
echo "$line" >> "mutants_by_package/stacks-signer.txt"
43+
;;
44+
*)
45+
echo "$line" >> "mutants_by_package/small-packages.txt"
46+
;;
47+
esac
48+
done < all_mutants.txt
49+
50+
# Function to run mutants for a package
51+
run_mutants() {
52+
local package=$1
53+
local threshold=$2
54+
local output_dir=$3
55+
local mutant_file="mutants_by_package/${package}.txt"
56+
57+
if [ ! -f "$mutant_file" ]; then
58+
echo "No mutants found for $package"
59+
return 0
60+
fi
61+
62+
local regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "$mutant_file" | paste -sd'|' -)
63+
local mutant_count=$(cargo mutants -F "$regex_pattern" -E ": replace .{1,2} with .{1,2} in " --list | wc -l)
64+
65+
if [ "$mutant_count" -gt "$threshold" ]; then
66+
echo "Running mutants for $package ($mutant_count mutants)"
67+
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
68+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
69+
-F "$regex_pattern" \
70+
-E ": replace .{1,2} with .{1,2} in " \
71+
--output "$output_dir" \
72+
--test-tool=nextest \
73+
--package "$package" \
74+
-- --all-targets --test-threads 1 || true
75+
76+
echo $? > "${output_dir}/exit_code.txt"
77+
else
78+
echo "Skipping $package, only $mutant_count mutants (threshold: $threshold)"
79+
fi
80+
81+
return 0
82+
}
83+
84+
# Run mutants for each wanted package
85+
run_mutants "stacks-signer" 500 "./stacks-signer_mutants" || true
86+
run_mutants "stacks-node" 540 "./stacks-node_mutants" || true
87+
run_mutants "stackslib" 72 "./stackslib_mutants" || true

‎docs/mutation-testing.md

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# Mutation Testing
2+
3+
This document describes how to run mutation testing locally to mimic the outcome of a PR, without the CI limitation it provides by timing out after 6 hours.
4+
[Here is the script](../contrib/tools/local-mutation-testing.sh) to run the tests locally by running the mutants created by the changes between `HEAD` and develop.
5+
It does automatically all the steps explained below.
6+
7+
From the root level of the stacks-core repository run
8+
```sh
9+
./contrib/tools/local-mutation-testing.sh
10+
```
11+
12+
## Prerequirements
13+
14+
Install the cargo mutants library
15+
```sh
16+
cargo install --version 24.7.1 cargo-mutants --locked
17+
```
18+
19+
20+
## Steps
21+
1. Be on source branch you would use for the PR.
22+
2. Create diff file comparing this branch with the `develop` branch
23+
```sh
24+
git diff origin/develop..HEAD > git.diff
25+
```
26+
3. Clean up the diff file and create auxiliary files
27+
```sh
28+
awk '
29+
/^diff --git/ {
30+
diff_line = $0
31+
getline
32+
if ($0 !~ /^(deleted file mode|similarity index)/) {
33+
print diff_line
34+
print
35+
}
36+
}
37+
!/^(diff --git|deleted file mode|similarity index|rename from|rename to)/ {print}
38+
' git.diff > processed.diff
39+
40+
# Extract mutants based on the processed diff
41+
cargo mutants --in-diff processed.diff --list > all_mutants.txt
42+
43+
# Create a directory for organizing mutants
44+
mkdir -p mutants_by_package
45+
46+
# Organize mutants into files based on their main folder
47+
while IFS= read -r line; do
48+
package=$(echo "$line" | cut -d'/' -f1)
49+
50+
case $package in
51+
"stackslib")
52+
echo "$line" >> "mutants_by_package/stackslib.txt"
53+
;;
54+
"testnet")
55+
echo "$line" >> "mutants_by_package/stacks-node.txt"
56+
;;
57+
"stacks-signer")
58+
echo "$line" >> "mutants_by_package/stacks-signer.txt"
59+
;;
60+
*)
61+
echo "$line" >> "mutants_by_package/small-packages.txt"
62+
;;
63+
esac
64+
done < all_mutants.txt
65+
```
66+
4. Based on the package required to run the mutants for
67+
a. Stackslib package
68+
```sh
69+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/stackslib.txt" | paste -sd'|' -)
70+
71+
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
72+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
73+
-F "$regex_pattern" \
74+
-E ": replace .{1,2} with .{1,2} in " \
75+
--output "./stackslib_mutants" \
76+
--test-tool=nextest \
77+
-- --all-targets --test-threads 1
78+
```
79+
b. Stacks-node (testnet) package
80+
```sh
81+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/testnet.txt" | paste -sd'|' -)
82+
83+
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
84+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
85+
-F "$regex_pattern" \
86+
-E ": replace .{1,2} with .{1,2} in " \
87+
--output "./testnet_mutants" \
88+
--test-tool=nextest \
89+
-- --all-targets --test-threads 1
90+
```
91+
c. Stacks-signer
92+
```sh
93+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/stacks-signer.txt" | paste -sd'|' -)
94+
95+
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
96+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
97+
-F "$regex_pattern" \
98+
-E ": replace .{1,2} with .{1,2} in " \
99+
--output "./stacks-signer_mutants" \
100+
--test-tool=nextest \
101+
-- --all-targets --test-threads 1
102+
```
103+
d. All other packages combined
104+
```sh
105+
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/small-packages.txt" | paste -sd'|' -)
106+
107+
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
108+
-F "$regex_pattern" \
109+
-E ": replace .{1,2} with .{1,2} in " \
110+
--output "./small-packages_mutants" \
111+
--test-tool=nextest \
112+
-- --all-targets --test-threads 1
113+
```
114+
115+
## How to run one specific mutant to test it
116+
117+
Example of output which had a missing mutant
118+
```sh
119+
MISSED stacks-signer/src/runloop.rs:424:9: replace <impl SignerRunLoop for RunLoop<Signer, T>>::run_one_pass -> Option<Vec<SignerResult>> with None in 3.0s build + 9.3s test
120+
```
121+
122+
Example of fix for it
123+
```sh
124+
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
125+
cargo mutants -vV \
126+
-F "replace process_stackerdb_event" \
127+
-E ": replace <impl SignerRunLoop for RunLoop<Signer, T>>::run_one_pass -> Option<Vec<SignerResult>> with None in " \
128+
--test-tool=nextest \
129+
-- \
130+
--run-ignored all \
131+
--fail-fast \
132+
--test-threads 1
133+
```
134+
135+
General command to run
136+
```sh
137+
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
138+
cargo mutants -vV \
139+
-F "replace process_stackerdb_event" \
140+
-E ": replace [modify this] with [modify this] in " \
141+
--test-tool=nextest \
142+
-- \
143+
--run-ignored all \
144+
--fail-fast \
145+
--test-threads 1
146+
```

0 commit comments

Comments
 (0)