Feature request: with --fail-fast, kill other currently running tests

[sourcefrog]

Motivation

I want nextest to be faster than it is today. In particular, its subprocess execution model is faster than cargo test in some situations, but can be slower when some tests fail.

In particular I'm looking at this because cargo-mutants runs many iterations of different test suites, most of which are expected to fail, and I notice that on at least some trees it's significantly slower than libtest.

Today, when some tests are failing:

• cargo test and libtest continue to run all the tests in that target, and then it exits.
• nextest allows any other tests currently running on other threads to finish, and then it exits.

As a simplified example, suppose we have these tests:

// tests/fast.rs
#[test]
fn fast_passes() {
    assert!(true);
}

#[test]
fn fast_fails() {
    assert!(false, "This test should fail");
}

// tests/slow.rs
use std::{thread::sleep, time::Duration};

#[test]
fn slow_passes() {
    sleep(Duration::from_secs(10));
    assert!(true);
}

(This is also in https://github.com/sourcefrog/nextest-stragglers)

In this example, cargo test finishes on my laptop in 0.063s, and cargo nextest run takes 10.060s, because it waits for the slow_passes test to finish. (The difference could be even more extreme if there are integration tests that take more than 10s, which I think is not uncommon or unreasonable.)

; cargo nextest run
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.00s
────────────
 Nextest run ID 4feac082-9c79-4ecb-b041-03d919034e8c with nextest profile: default
    Starting 3 tests across 3 binaries
        PASS [   0.004s] nextest-stragglers::fast fast_passes
        FAIL [   0.004s] nextest-stragglers::fast fast_fails
  stdout ───

    running 1 test
    test fast_fails ... FAILED

    failures:

    failures:
        fast_fails

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

  stderr ───

    thread 'fast_fails' panicked at tests/fast.rs:8:5:
    This test should fail
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

  Cancelling due to test failure: 1 test still running
        PASS [  10.005s] nextest-stragglers::slow slow_passes
────────────
     Summary [  10.006s] 3 tests run: 2 passed, 1 failed, 0 skipped
        FAIL [   0.004s] nextest-stragglers::fast fast_fails
error: test run failed

If this minimization seems unrealistic, imagine a tree which has many sub-100ms unit tests and some >10s integration tests: there's a good chance nextest will wait for at least one integration test to complete, even if a unit tests caught the bug. I think many Rust trees have something like this structure.

Proposal

I'd like an additional option to "kill stragglers" once a test fails. Nextest already seems to have some logic to kill tests that exceed a timeout, so I'd propose to just hook up that code to fire immediately, or after a short timeout, once it decides the suite has failed.

I suspect this mirrors what many humans would do by hitting ctrl-c once nextest says it's cancelling, but it'll make it happen automatically under cargo-mutants, in CI, in background windows, etc.

I realize there might be some cases where test suites leak resources when killed, and so users might not want this, but I think in general sending SIGINT should be reasonable, and it can just be an option.

As a variation we could give stragglers a fraction of a second, or a configurable interval, to finish naturally before trying to kill them.

I think a good precedent for this is the way that Github Actions and other CI systems will terminate other parallel jobs when fail-fast is configured. I think Bazel will also kill jobs with --test-keep-going=false.

I'd suggest possibly this should be on by default with --ff to give the strongest fail-fast behavior.

Alternatives

No response

Additional context

No response

[sunshowers]

Hey! Yeah I was thinking about this a few weeks ago too. I think this can definitely be an optional mode (as you mentioned, all the implementation logic exists and it just needs to be wired up), though I'm not sure it should be the default -- because it's nice for other tests not to fail if the first one does. In interactive use, yeah, the idea is that if you do want to exit immediately you'll press Ctrl-C.