Merge pull request #119 from sourcefrog/return-error

Optionally return errors from functions returning Result

Author: sourcefrog

.cargo/mutants.toml

@@ -1,3 +1,4 @@
 # cargo-mutants configuration
 
-exclude_globs = [ "src/console.rs" ]
+error_values = ["::anyhow::anyhow!(\"mutated\")"]
+exclude_globs = ["src/console.rs"]

.vscode/settings.json

@@ -0,0 +1 @@
+{}

Cargo.toml

@@ -89,6 +89,7 @@ exclude = [
     "testdata/tree/cfg_attr_mutants_skip",
     "testdata/tree/cfg_attr_test_skip",
     "testdata/tree/dependency",
+    "testdata/tree/error_value",
     "testdata/tree/everything_skipped",
     "testdata/tree/factorial",
     "testdata/tree/fails_without_feature",

NEWS.md

@@ -11,6 +11,11 @@
 
 - Minimum supported Rust version increased to 1.65 due to changes in dependencies.
 
+- New `--error` option, to cause functions returning `Result` to be mutated to return the
+  specified error.
+
+- New `--no-config` option, to disable reading `.cargo/mutants.toml`.
+
 ## 1.2.2
 
 Released 2023-04-01

book/src/SUMMARY.md

@@ -8,13 +8,15 @@
   - [Exit codes](exit-codes.md)
   - [The `mutants.out` directory](mutants-out.md)
 - [Skipping untestable code](skip.md)
+  - [Skipping functions with an attribute](attrs.md)
+  - [Filtering files](skip_files.md)
+  - [Filtering functions and mutants](filter_mutants.md)
 - [Controlling cargo-mutants](controlling.md)
   - [Listing and previewing mutations](list.md)
-  - [Filtering files](skip_files.md)
-  - [Filtering mutants](filter_mutants.md)
   - [Workspaces and packages](workspaces.md)
   - [Passing options to Cargo](cargo-args.md)
-  - [The `mutants.toml` config file](config.md)
+- [Generating mutants](mutants.md)
+  - [Error values](error-values.md)
 - [Improving performance](performance.md)
   - [Parallelism](parallelism.md)
 - [Integrations](integrations.md)

book/src/attrs.md

@@ -0,0 +1,51 @@
+# Skipping functions with an attribute
+
+To mark functions as skipped, so they are not mutated:
+
+1. Add a Cargo dependency on the [mutants](https://crates.io/crates/mutants)
+   crate, version "0.0.3" or later. (This must be a regular `dependency` not a
+   `dev-dependency`, because the annotation will be on non-test code.)
+
+2. Mark functions with `#[mutants::skip]` or other attributes containing
+   `mutants::skip` (e.g. `#[cfg_attr(test, mutants::skip)]`).
+
+The `mutants` create is tiny and the attribute has no effect on the compiled
+code. It only flags the function for cargo-mutants. However, you can avoid the
+dependency by using the slightly longer `#[cfg_attr(test, mutants::skip)]` form.
+
+**Note:** Currently, `cargo-mutants` does not (yet) evaluate attributes like
+`cfg_attr`, it only looks for the sequence `mutants::skip` in the attribute.
+
+You may want to also add a comment explaining why the function is skipped.
+
+For example:
+
+```rust
+use std::time::{Duration, Instant};
+
+/// Returns true if the program should stop
+#[cfg_attr(test, mutants::skip)] // Returning false would cause a hang
+fn should_stop() -> bool {
+    true
+}
+
+pub fn controlled_loop() {
+    let start = Instant::now();
+    for i in 0.. {
+        println!("{}", i);
+        if should_stop() {
+            break;
+        }
+        if start.elapsed() > Duration::from_secs(60 * 5) {
+            panic!("timed out");
+        }
+    }
+}
+
+mod test {
+    #[test]
+    fn controlled_loop_terminates() {
+        super::controlled_loop()
+    }
+}
+```

book/src/cargo-args.md

@@ -18,18 +18,28 @@ For example
 cargo mutants -- --cargo-arg=--release
 ```
 
+or in `.cargo/mutants.toml`:
+
+```toml
+additional_cargo_args = ["--all-features"]
+```
+
 ## Arguments to `cargo test`
 
 Command-line options following a `--` delimiter are passed through to
 `cargo test`. For example, this can be used to pass `--all-targets` which (unobviously)
 excludes doctests. (If the doctests are numerous and slow, and not relied upon to catch bugs, this can improve performance.)
 
-These options can also be configured statically with the `additional_cargo_test_args` key in `.cargo/mutants.toml`.
-
 ```shell
 cargo mutants -- --all-targets
 ```
 
+These options can also be configured statically with the `additional_cargo_test_args` key in `.cargo/mutants.toml`:
+
+```toml
+additional_cargo_test_args = ["--jobs=1"]
+```
+
 ## Arguments to test binaries
 
 You can use a second double-dash to pass options through to the test targets:

book/src/config.md

@@ -1,16 +1,30 @@
 # Controlling cargo-mutants
 
-`cargo mutants` takes various options to control how it runs. These options are shown in `cargo mutants --help` and are described in detail in this section.
+`cargo mutants` takes various options to control how it runs.
+
+These options, can, in general, be passed on the command line, set in a `.cargo/mutants.toml`
+file in the source tree, or passed in `CARGO_MUTANTS_` environment variables. Not every
+method of setting an option is available for every option, however, as some would not
+make sense or be useful.
+
+For options that take a list of values, values from the configuration file are appended
+to values from the command line.
+
+For options that take a single value, the value from the command line takes precedence.
+
+`--no-config` can be used to disable reading the configuration file.
+
+## Execution order
 
 By default, mutants are run in a randomized order, so as to surface results from
 different parts of the codebase earlier. This can be disabled with
-`--no-shuffle`, in which case mutants will run in the same order shown by
-`--list`: in order by file name and within each file in the order they appear in
+`--no-shuffle`, in which case mutants will run in order by file name and within each file in the order they appear in
 the source.
 
 ## Source directory location
 
-`-d`, `--dir`: Test the Rust tree in the given directory, rather than the default directory.
+`-d`, `--dir`: Test the Rust tree in the given directory, rather than the source tree
+enclosing the working directory where cargo-mutants is launched.
 
 ## Console output
 
@@ -20,10 +34,6 @@ the source.
 
 `--no-times`: Don't print elapsed times.
 
-## Environment variables
-
-A few options that may be useful to set globally can be configured through environment 
-variables:
-
-* `CARGO_MUTANTS_JOBS`
-* `CARGO_MUTANTS_TRACE_LEVEL`
+`-L`, `--level`, and `$CARGO_MUTANTS_TRACE_LEVEL`: set the verbosity of trace
+output to stdout. The default is `info`, and it can be increased to `debug` or
+`trace`.

book/src/controlling.md

@@ -0,0 +1,32 @@
+# Generating error values
+
+cargo-mutants can be configured to generate mutants that return an error value from functions that return a Result.
+
+This will flag cases where no test fails if the function returns an error: that might happen if there are _only_ tests for the error cases and not for the Ok case.
+
+Since crates can choose to use any type for their error values,
+cargo-mutants must be told how to construct an appropriate error.
+
+The `--error` command line option and the `error_value` configuration option specify an error value to use.
+
+These options can be repeated or combined, which might be useful
+if there are multiple error types in the crate. On any one mutation site, probably only one of the error values will be viable, and cargo-mutants will discover that and use it.
+
+The error value can be any Rust expression that evaluates to a value of the error type. It should not include the `Err` wrapper, because cargo-mutants will add that.
+
+For example, if your crate uses `anyhow::Error` as its error type, you might use `--error '::anyhow::anyhow!("error")'`.
+
+If you have your own error type, you might use `--error 'crate::MyError::Generic'`.
+
+Since the correct error type is a property of the source tree, the configuration should typically go into `.cargo/mutants.toml` rather than being specified on the command line:
+
+```toml
+error_values = ["::anyhow::anyhow!(\"mutated\")"]
+```
+
+To see only the mutants generated by this configuration, you
+can use a command like this:
+
+```sh
+cargo r mutants -F anyhow -vV -j4
+```