Switch to nextest for testing (#753)

* Tweak comment

* Add nextest to CI

* Add nextest as default test task

* Add a `justfile` for easy test running

* Add nextest and just info to `BUILDING.md`

Author: DavisVaughan

.config/nextest.toml

@@ -0,0 +1,11 @@
+[profile.ci]
+# Print out output for failing tests as soon as they fail, and also at the end
+# of the run (for easy scrollability).
+failure-output = "immediate-final"
+
+# Do not cancel the test run on the first failure.
+fail-fast = false
+
+# Mark tests that take longer than 60s as slow.
+# Terminate after 60s as a stop-gap measure to terminate on deadlock.
+slow-timeout = { period = "60s", terminate-after = 1 }

.github/workflows/test-linux.yml

@@ -5,7 +5,7 @@ on:
   workflow_dispatch:
     inputs:
       ssh:
-        description: 'Set up an SSH session before running `cargo test`?'
+        description: 'Set up an SSH session before running `cargo nextest run`?'
         type: boolean
         required: true
         default: false
@@ -45,6 +45,11 @@ jobs:
         # Cache isn't useful on nightly, it would be thrown away every day
         if: matrix.config.rust != 'nightly'
 
+      - name: Install nextest
+        uses: taiki-e/install-action@v2
+        with:
+          tool: cargo-nextest
+
       - name: Install R
         uses: r-lib/actions/setup-r@v2
         with:
@@ -72,8 +77,10 @@ jobs:
 
       - name: Build
         run: |
-            cargo build
+          cargo build
 
-      - name: Run Unit Tests
+      - name: Run Tests
+        env:
+          NEXTEST_PROFILE: "ci"
         run: |
-            cargo test --verbose -- --nocapture
+          cargo nextest run

.github/workflows/test-macos.yml

@@ -25,6 +25,11 @@ jobs:
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
 
+      - name: Install nextest
+        uses: taiki-e/install-action@v2
+        with:
+          tool: cargo-nextest
+
       - name: Install R
         uses: r-lib/actions/setup-r@v2
         with:
@@ -43,8 +48,10 @@ jobs:
 
       - name: Build
         run: |
-            cargo build
+          cargo build
 
-      - name: Run Unit Tests
+      - name: Run Tests
+        env:
+          NEXTEST_PROFILE: "ci"
         run: |
-            cargo test --verbose -- --nocapture
+          cargo nextest run

.github/workflows/test-windows.yml

@@ -25,6 +25,11 @@ jobs:
       - name: Rust Cache
         uses: Swatinem/rust-cache@v2
 
+      - name: Install nextest
+        uses: taiki-e/install-action@v2
+        with:
+          tool: cargo-nextest
+
       - name: Install R
         uses: r-lib/actions/setup-r@v2
         with:
@@ -43,9 +48,10 @@ jobs:
 
       - name: Build
         run: |
-            cargo build
+          cargo build
 
-      - name: Run Unit Tests
-        # Very loud on windows CI, which has been a source of instability
+      - name: Run Tests
+        env:
+          NEXTEST_PROFILE: "ci"
         run: |
-            cargo test -vv -- --nocapture
+          cargo nextest run

.vscode/tasks.json

@@ -15,7 +15,7 @@
         },
         {
             "type": "cargo",
-            "command": "test",
+            "command": "nextest",
             "problemMatcher": [
                 "$rustc",
                 "$rust-panic"
@@ -24,7 +24,10 @@
                 "kind": "test",
                 "isDefault": true
             },
-            "label": "rust: cargo test"
+            "args": [
+                "run"
+            ],
+            "label": "rust: cargo nextest run"
         }
     ]
 }

BUILDING.md

@@ -35,9 +35,9 @@ You will usually want to tweak the **ark** environment for development; add this
 
 This enables backtrace capturing in [anyhow](https://docs.rs/anyhow) errors and sets internal crates to log at TRACE level and external dependencies to log at WARN. Setting the latter to more verbose levels can dramatically decrease performance. See the documentation in the [tracing_subscriber](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html) crate for more fine-grained tuning of the `RUST_LOG` environment variable.
 
-## Test with Positron
+## Test with Release Positron
 
-To test the dev build of ARK on Positron, you can open Positron's user settings
+To test the dev build of ARK on Release Positron, you can open Positron's user settings
 and change option `Positron > R > Kernel: Path` (ID: `positron.r.kernel.path`)
 to the location of the binary.
 
@@ -46,3 +46,31 @@ to the location of the binary.
     "positron.r.kernel.path": "/path/to/ark/target/debug/ark",
 }
 ```
+
+With a development version of Positron, a development version of Ark is automatically detected as long as the `positron/` and `ark/` folders are at the same depth, i.e.:
+
+```
+| files/
+| |- ark/
+| |- positron/
+```
+
+## Testing
+
+We use [nextest](https://nexte.st/) for testing rather than a standard `cargo test`, primarily because nextest runs each test in its own process rather than in its own thread.
+This is critical for us, as Ark has global objects that can only be set up once per process (such as setup around the R process itself).
+Additionally, using one process per test means that it is impossible for one test to interfere with another (so you don't have to worry about test cleanup, particularly if you add objects to R's global environment).
+Tests are still run in parallel, using multiple processes, and this ends up being quite fast and reliable.
+
+Install the nextest cli tool using a [prebuilt binary](https://nexte.st/docs/installation/pre-built-binaries/).
+
+Run tests locally with `just test` (which runs `cargo nextest run`) or `Tasks: Run Test Task` in VS Code (which you can bind to a keyboard shortcut).
+Run insta snapshot tests in "update" mode with `just test-insta` (which runs `cargo insta test --test-runner nextest`).
+
+On CI we use the nextest profile found in `.config/nextest.toml`.
+
+## Just
+
+We use [just](https://github.com/casey/just) as a simple command runner, and the shortcuts live in `justfile`.
+On macOS, install just with `brew install just`.
+For other platforms, see the just README.

crates/harp/build.rs

@@ -17,7 +17,7 @@ fn main() {
     //
     // We also do this for ark.
     //
-    // We don't generate a main `harp.exe` binary, but `cargo test` does generate a `harp-*.exe`
+    // We don't generate a main `harp.exe` binary, but `cargo nextest run` does generate a `harp-*.exe`
     // binary for unit testing, and those unit tests also start R and test UTF-8 related capabilities!
     // So we need that test executable to include a manifest file too.
     let resource = Path::new("resources")

justfile

@@ -0,0 +1,14 @@
+
+# Run the tests
+test:
+  cargo nextest run
+
+# Run the tests in verbose mode
+# `--no-capture` forces stdout/stderr to be shown for all tests, not just failing ones,
+# and also forces them to be run sequentially so you don't see interleaved live output
+test-verbose:
+  cargo nextest run --no-capture
+
+# Run the insta tests in update mode
+test-insta:
+  cargo insta test --test-runner nextest