test_runner: support running tests in process

This commit introduces a new --experimental-test-isolation flag
that, when set to 'none', causes the test runner to execute all
tests in the same process. By default, this is the main test
runner process, but if watch mode is enabled, it spawns a separate
process that runs all of the tests.

The default value of the new flag is 'process', which uses the
existing behavior of running each test file in its own child
process.

It is worth noting that when the isolation mode is 'none', globals
and all other top level logic (such as top level before() and after()
hooks) is shared among all files.

Co-authored-by: Moshe Atlow <[email protected]>
PR-URL: #53927
Fixes: #51548
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>

Author: 2 people

doc/api/cli.md

@@ -1091,6 +1091,20 @@ generated as part of the test runner output. If no tests are run, a coverage
 report is not generated. See the documentation on
 [collecting code coverage from tests][] for more details.
 
+### `--experimental-test-isolation=mode`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+> Stability: 1.0 - Early development
+
+Configures the type of test isolation used in the test runner. When `mode` is
+`'process'`, each test file is run in a separate child process. When `mode` is
+`'none'`, all test files run in the same process as the test runner. The default
+isolation mode is `'process'`. This flag is ignored if the `--test` flag is not
+present. See the [test runner execution model][] section for more information.
+
 ### `--experimental-test-module-mocks`
 
 <!-- YAML
@@ -2196,7 +2210,9 @@ added:
 -->
 
 The maximum number of test files that the test runner CLI will execute
-concurrently. The default value is `os.availableParallelism() - 1`.
+concurrently. If `--experimental-test-isolation` is set to `'none'`, this flag
+is ignored and concurrency is one. Otherwise, concurrency defaults to
+`os.availableParallelism() - 1`.
 
 ### `--test-coverage-exclude`
 
@@ -2361,7 +2377,7 @@ added: v22.3.0
 
 > Stability: 1.0 - Early development
 
-Regenerates the snapshot file used by the test runner for [snapshot testing][].
+Regenerates the snapshot files used by the test runner for [snapshot testing][].
 Node.js must be started with the `--experimental-test-snapshots` flag in order
 to use this functionality.
 
@@ -3534,6 +3550,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
 [snapshot testing]: test.md#snapshot-testing
 [syntax detection]: packages.md#syntax-detection
 [test reporters]: test.md#test-reporters
+[test runner execution model]: test.md#test-runner-execution-model
 [timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
 [tracking issue for user-land snapshots]: https://github.com/nodejs/node/issues/44014
 [ways that `TZ` is handled in other environments]: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html

doc/api/test.md

@@ -445,18 +445,26 @@ in the [test runner execution model][] section.
 
 ### Test runner execution model
 
-Each matching test file is executed in a separate child process. The maximum
-number of child processes running at any time is controlled by the
-[`--test-concurrency`][] flag. If the child process finishes with an exit code
-of 0, the test is considered passing. Otherwise, the test is considered to be a
-failure. Test files must be executable by Node.js, but are not required to use
-the `node:test` module internally.
+When process-level test isolation is enabled, each matching test file is
+executed in a separate child process. The maximum number of child processes
+running at any time is controlled by the [`--test-concurrency`][] flag. If the
+child process finishes with an exit code of 0, the test is considered passing.
+Otherwise, the test is considered to be a failure. Test files must be executable
+by Node.js, but are not required to use the `node:test` module internally.
 
 Each test file is executed as if it was a regular script. That is, if the test
 file itself uses `node:test` to define tests, all of those tests will be
 executed within a single application thread, regardless of the value of the
 `concurrency` option of [`test()`][].
 
+When process-level test isolation is disabled, each matching test file is
+imported into the test runner process. Once all test files have been loaded, the
+top level tests are executed with a concurrency of one. Because the test files
+are all run within the same context, it is possible for tests to interact with
+each other in ways that are not possible when isolation is enabled. For example,
+if a test relies on global state, it is possible for that state to be modified
+by a test originating from another file.
+
 ## Collecting code coverage
 
 > Stability: 1 - Experimental
@@ -933,7 +941,7 @@ the [`--experimental-test-snapshots`][] command-line flag.
 Snapshot files are generated by starting Node.js with the
 [`--test-update-snapshots`][] command-line flag. A separate snapshot file is
 generated for each test file. By default, the snapshot file has the same name
-as `process.argv[1]` with a `.snapshot` file extension. This behavior can be
+as the test file with a `.snapshot` file extension. This behavior can be
 configured using the `snapshot.setResolveSnapshotPath()` function. Each
 snapshot assertion corresponds to an export in the snapshot file.
 
@@ -1239,6 +1247,9 @@ added:
   - v18.9.0
   - v16.19.0
 changes:
+  - version: REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/53927
+    description: Added the `isolation` option.
   - version: v22.6.0
     pr-url: https://github.com/nodejs/node/pull/53866
     description: Added the `globPatterns` option.
@@ -1274,8 +1285,13 @@ changes:
   * `inspectPort` {number|Function} Sets inspector port of test child process.
     This can be a number, or a function that takes no arguments and returns a
     number. If a nullish value is provided, each process gets its own port,
-    incremented from the primary's `process.debugPort`.
-    **Default:** `undefined`.
+    incremented from the primary's `process.debugPort`. This option is ignored
+    if the `isolation` option is set to `'none'` as no child processes are
+    spawned. **Default:** `undefined`.
+  * `isolation` {string} Configures the type of test isolation. If set to
+    `'process'`, each test file is run in a separate child process. If set to
+    `'none'`, all test files run in the current process. **Default:**
+    `'process'`.
   * `only`: {boolean} If truthy, the test context will only run tests that
     have the `only` option set
   * `setup` {Function} A function that accepts the `TestsStream` instance
@@ -1727,9 +1743,9 @@ added: v22.3.0
 
 * `fn` {Function} A function used to compute the location of the snapshot file.
   The function receives the path of the test file as its only argument. If the
-  `process.argv[1]` is not associated with a file (for example in the REPL),
-  the input is undefined. `fn()` must return a string specifying the location of
-  the snapshot file.
+  test is not associated with a file (for example in the REPL), the input is
+  undefined. `fn()` must return a string specifying the location of the snapshot
+  snapshot file.
 
 This function is used to customize the location of the snapshot file used for
 snapshot testing. By default, the snapshot filename is the same as the entry

doc/node.1

@@ -185,6 +185,9 @@ Enable the experimental node:sqlite module.
 .It Fl -experimental-test-coverage
 Enable code coverage in the test runner.
 .
+.It Fl -experimental-test-isolation Ns = Ns Ar mode
+Configures the type of test isolation used in the test runner.
+.
 .It Fl -experimental-test-module-mocks
 Enable module mocking in the test runner.
 .

lib/internal/main/test_runner.js

@@ -21,7 +21,7 @@ markBootstrapComplete();
 
 const options = parseCommandLine();
 
-if (isUsingInspector()) {
+if (isUsingInspector() && options.isolation === 'process') {
   process.emitWarning('Using the inspector with --test forces running at a concurrency of 1. ' +
   'Use the inspectPort option to run with concurrency');
   options.concurrency = 1;

lib/internal/test_runner/harness.js

@@ -334,4 +334,5 @@ module.exports = {
   after: hook('after'),
   beforeEach: hook('beforeEach'),
   afterEach: hook('afterEach'),
+  startSubtestAfterBootstrap,
 };