Use of debugger incompatible with console testing framework

[cefn]

Launching the tests within a VSCode Javascript Debug Terminal or probably any debug process fails all CLI tests. The node process uncontrollably writes the following lines for every attached VM, breaking the strategy of console snapshot assertions...

Debugger attached.␊
Waiting for the debugger to disconnect...␊

See upstream nodejs/node#34799

A strategy that allows narrower test assertions (rather than the whole snapshot of the whole CLI output) would enable people to interactively debug tests at the same time as observing actual test failures. However, I don't know if that is compatible with the testing framework used here.

A strategy that worked for us for CLI testing in jest/vitest was not to rely on a console testing framework, but explicitly capture stderr and stdout and use e.g. expect.stringMatching() or other more 'forgiving' tests of console output, that allows there to be variable output which doesn't break the test assertion. This also makes tests more maintainable generally since unrelated changes (e.g. changing some greeting text) won't break every test - only some actual test of the greeting text.

Interception example

// using test utility
    test("Exits with error when invoking unknown command", async () => {
      const { stdout, stderr } = process;
      using callsByStream = interceptStreams({ stdout, stderr });

      // mock the CLI command
      using processMock = mockCommandLineProcess(
        `npx @roku-web-starter/cli this-is-not-a-command`,
      );

      // run the CLI tool
      await executeCommand();

      expect(
        [...callsByStream.stdout, ...callsByStream.stderr].some((message) =>
          message.includes(`Unknown command`),
        ),
      ).toBe(true);

      const [firstExitArgs] = processMock.exit.mock.calls;
      expect(firstExitArgs).toMatchObject([1]);
    });

// test utility
type WriteMethod = NodeJS.WriteStream["write"];

export function interceptStreams<StreamName extends string>(
  streams: Record<StreamName, NodeJS.WriteStream>,
  passthru = false,
) {
  const streamEntries = Object.entries(streams) as [
    StreamName,
    NodeJS.WriteStream,
  ][];

  const writeByStream = {} as Record<StreamName, WriteMethod>;
  const callsByStream = Object.fromEntries(
    Object.keys(streams).map((name) => [name, []]),
  ) as unknown as Record<StreamName, string[]>;

  // substitute stream write method
  for (const [name, stream] of streamEntries) {
    // prepare interception - deliberately stores unbound method
    // eslint-disable-next-line @typescript-eslint/unbound-method
    const { write } = stream;
    const calls: string[] = [];

    // keep per-stream records
    writeByStream[name] = write;
    callsByStream[name] = calls;

    // intercept
    const interceptor = ((...args: Parameters<typeof write>) => {
      const [bufferOrString] = args;
      calls.push(bufferOrString.toString());
      if (passthru) {
        return write.call(stream, ...args);
      }
      return true;
    }) as WriteMethod;
    stream.write = interceptor;
  }

  const stopInterception = () => {
    // restore stream write methods
    for (const [name, stream] of streamEntries) {
      stream.write = writeByStream[name];
    }
  };

  return {
    ...callsByStream,
    [Symbol.dispose]: stopInterception,
  };
}

[keithamus]

This doesn't look to be filed in the right repo.

Closing this.

[cefn]

Confused, sorry @keithamus

This is related to running npm run test from the sort-package-json repo, while trying to debug observed test failures. In particular the tests under the cli suite which asserts on console output.

The tests fail whenever run in a debugger because node debugger output breaks the test console output assertions from ava. I don't know enough about the console integration to know if there's a different way to make assertions than 'all console output should be exactly this string' which is bound to break for the debugger case, hence sharing a pattern we use (which turns console outputs into an array that we can assert on more selectively).

The implicit assertions that break are like...

sort-package-json/tests/snapshots/cli.js.md

Lines 19 to 30 in e3f2370

	stdout: `Usage: sort-package-json [options] [file/glob ...]␊
	␊
	Sort package.json files.␊
	If file/glob is omitted, './package.json' file will be processed.␊
	␊
	-c, --check Check if files are sorted␊
	-q, --quiet Don't output success messages␊
	-h, --help Display this help␊
	-i, --ignore An array of glob patterns to ignore␊
	-v, --version Display the package version␊
	--stdin Read package.json from stdin␊
	␊

...because they end up with the extra console lines added by the node debugger.

Issue report and suggestion may be unwelcome or stupid I accept, but it's definitely against your repo.

Is debugging tests in node a reasonable target?

[cefn]

For future reference, this can be fixed temporarily in your local by editing at

sort-package-json/tests/_helpers.js

Line 222 in e3f2370

(error, stdout, stderr) => {

to strip out the extra stderr lines. Please leave closed.

          stderr: stderr
            .replace(/Debugger attached.\s/, '')
            .replace(/Waiting for the debugger to disconnect...\s/, ''),

[keithamus]

Apologies. It wasn't evident reading the OP how this related to the project. I see now, with the extra context.

[keithamus]

@cefn let's leave this open and perhaps look at a way to resolve it that doesn't require users of vsdcode to comment out parts of the test harness.