-
Notifications
You must be signed in to change notification settings - Fork 375
init mutation-testing skill #140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
13fc84b
init mutation-testing skill
bohendo cce9c5d
Merge branch 'main' into init-mutation-testing
bohendo 30fdde3
fix metadata
bohendo 92f1583
init codex symlink
bohendo d098977
polish according to review feedback
bohendo a0ea77e
Merge branch 'main' into init-mutation-testing
dguido File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| { | ||
| "name": "mutation-testing", | ||
| "version": "1.0.0", | ||
| "description": "Configures mewt or muton mutation testing campaigns — scopes targets, tunes timeouts, and optimizes long-running runs. Use when the user mentions mewt, muton, mutation testing, or wants to configure or optimize a mutation testing campaign.", | ||
| "author": { | ||
| "name": "Trail of Bits", | ||
| "email": "opensource@trailofbits.com", | ||
| "url": "https://github.com/trailofbits" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # Mutation Testing — Campaign Configuration (mewt/muton) | ||
|
|
||
| Helps configure [mewt](https://github.com/trailofbits/mewt) or [muton](https://github.com/trailofbits/muton) mutation testing campaigns — scoping targets, tuning timeouts, and optimizing long-running runs so you can execute `mewt run` or `muton run` with confidence. | ||
|
|
||
| > **Note**: muton and mewt share identical interfaces. This plugin uses `mewt` in examples, but all commands work the same with `muton`. File names change accordingly: `mewt.toml` → `muton.toml`, `mewt.sqlite` → `muton.sqlite`. | ||
|
|
||
| ## What It Does | ||
|
|
||
| Walks through a 5-phase configuration workflow: | ||
|
|
||
| 1. **Initialize and validate targets** — run `mewt init`, review auto-generated config, fix include/ignore patterns | ||
| 2. **Generate mutants and assess scope** — count mutants, time the test command, estimate campaign duration | ||
| 3. **Decide on optimization** — choose between full run, targeted components, high/medium severity only, or two-phase campaign | ||
| 4. **Validate test command and timeout** — verify the command works; set manual timeout for recompilation-heavy languages (Solidity/Foundry, heavy C++) | ||
| 5. **Final validation checklist** — confirm config, mutant count, target selection, and timeout before running | ||
|
|
||
| ## When to Use | ||
|
|
||
| - Setting up a new mutation testing campaign | ||
| - Optimizing a campaign that would take too long to run | ||
| - Diagnosing why no mutants are generated or why the test command fails | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - [mewt](https://github.com/trailofbits/mewt) v3.0.0+ or [muton](https://github.com/trailofbits/muton) v3.0.0+ installed | ||
| - A test suite runnable from the command line | ||
| - Source code in a supported language (Rust, Solidity, Go, Python, JavaScript, C/C++) | ||
|
|
||
| ## Example Usage | ||
|
|
||
| ``` | ||
| User: "Help me set up mewt for this Solidity project" | ||
| User: "Configure muton for this FunC codebase" | ||
| User: "My mewt campaign would take 30 hours — how do I optimize it?" | ||
| → Guides through configuration, scope assessment, and optimization | ||
| ``` | ||
|
|
||
| ## References | ||
|
|
||
| - [mewt GitHub Repository](https://github.com/trailofbits/mewt) | ||
| - [Use mutation testing to find the bugs your tests don't catch](https://blog.trailofbits.com/2025/09/18/use-mutation-testing-to-find-the-bugs-your-tests-dont-catch/) — Trail of Bits blog post | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| --- | ||
| name: mutation-testing | ||
| description: "Configures mewt or muton mutation testing campaigns — scopes targets, tunes timeouts, and optimizes long-running runs. Use when the user mentions mewt, muton, mutation testing, or wants to configure or optimize a mutation testing campaign." | ||
| allowed-tools: | ||
| - Read | ||
| - Write | ||
| - Bash | ||
| - Grep | ||
| --- | ||
|
|
||
| # Mutation Testing — Campaign Configuration (mewt/muton) | ||
|
|
||
| > **Note**: muton and mewt share identical interfaces. All examples use `mewt` commands, but they work exactly the same with `muton`. File names change accordingly: `mewt.toml` → `muton.toml`, `mewt.sqlite` → `muton.sqlite`. | ||
|
|
||
| ## When to Use | ||
|
|
||
| Use this skill when the user: | ||
| - Mentions "mewt", "muton", or "mutation testing" | ||
| - Needs to configure or optimize a mutation testing campaign | ||
| - Wants to run `mewt run` and needs help getting set up first | ||
|
|
||
| ## When NOT to Use | ||
|
|
||
| Do not use this skill when the user: | ||
| - Wants to analyze or report on completed campaign results | ||
| - Asks about tests or coverage without mentioning mutation testing | ||
|
|
||
| --- | ||
|
|
||
| ## Quick Start | ||
|
|
||
| Load [workflows/configuration.md](workflows/configuration.md) — a 5-phase guide from `mewt init` to a validated, ready-to-run campaign. | ||
|
|
||
| **General question or unfamiliar command?** | ||
| Run `mewt --help` or `mewt <subcommand> --help`, then assist. | ||
|
|
||
| --- | ||
|
|
||
| ## Reference Index | ||
|
|
||
| | File | Content | | ||
| |------|---------| | ||
| | [workflows/configuration.md](workflows/configuration.md) | 5-phase guide: init, scope, optimize, validate, run | | ||
| | [references/optimization-strategies.md](references/optimization-strategies.md) | Per-file targeting, two-phase campaigns, mutation type filtering | | ||
|
|
||
| --- | ||
|
|
||
| ## Essential Commands | ||
|
|
||
| ```bash | ||
| # Initialize and mutate | ||
| mewt init # Create mewt.toml and mewt.sqlite | ||
| mewt mutate [paths] # Generate mutants without running tests | ||
| mewt run [paths] # Run the full campaign | ||
|
|
||
| # Inspect configuration and scope | ||
| mewt print config # View effective configuration | ||
| mewt print targets # Table of all targeted files | ||
| mewt print mutations --language [lang] # Available mutation types | ||
| mewt status # Mutant count and per-file breakdown | ||
|
|
||
| # Investigate specific mutants | ||
| mewt print mutants --target [path] # All mutants for a file | ||
| mewt print mutants --severity high # Filter by severity | ||
| mewt print mutant --id [id] # View mutated code diff | ||
| mewt test --ids [ids] # Re-test specific mutants | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## What Results Mean | ||
|
|
||
| - **Caught/TestFail**: Tests detected the mutation (good) | ||
| - **Uncaught**: Mutation survived — indicates untested logic | ||
| - **Timeout**: Tests took too long, inconclusive | ||
| - **Skipped**: A more severe mutant already failed on the same line |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.