🛠 Repo: CI run failures (test failures)

[mark-wiemer]

Tooling Suggestion Checklist

• I have tried restarting my IDE and the issue persists.
• I have pulled the latest main branch of the repository.
• I have read and agree to Mocha's Code of Conduct and Contributing Guidelines
• I have searched for related issues and issues with the faq label, but none matched my issue.
• I want to provide a PR to resolve this

Overview

2026-02-16: Codecov has been pretty stable recently, and #5409 fixes some reliability issues with --watch tests (see also #5714). 2 out of 20 runs (4875 to 4894) failed, with 3 jobs failing and 3 total tests failing:

• 4880 - reporters:
  • XUnit reporter when 'reporterOptions.output' is provided when fileStream cannot be created when given an invalid pathname
    • "before each" hook for "should throw system error": Timeout of 1000ms exceeded
      • new failure, likely a rare timeout issue, we can double timeout here and monitor
      • possibly new because Mocha previously didn't report hook failures? 🐛 Bug: Hook failures change the shape of the test suite #1955
    • "after each" hook for "should throw system error": TypeError: cleanup is not a function
      • new failure, likely connected to the before each hook failing
• 4880 - Codecov: no test failures
• 4894 - watch: "--watch when enabled reruns test when file matching --watch-files is removed"
  • Should be fixed by fix: begin running tests immediately instead of waiting for watcher, reload required files in watch mode #5409 or its followup work
  • This failure is common as can be seen by the many links to this issue from affected PRs. We are prioritizing reviewing that PR to resolve this and other issues promptly :)

Older

2025-11-21: Yes, some checks are now failing due to Codecov. I'll reach out to Codecov support when I have some time.

2025-10-02:

4 of 20 runs (IDs 4283-4302) failed due to test issues, 1 failed due to Coveralls (ref #3943):

Run(s)	Config	Test name	Err msg	Comments
4283	20.19.4 Windows	Mocha "before each" hook for "should delegate Mocha.unloadFile() for each item in its list of files"	Timeout of 1000ms exceeded	new failure
4286	24.6.0 Windows	--parallel pool shutdown during normal ...	expected [ false, true, true ] to have items satisfying false	new failure
4288, 4290	20.19.4 Windows	--watch when enabled reruns test when file under --watch-files changes	expected 1 to be 2	was rare failure before as well

2025-08-17

• Run #4174: --exit tests occasionally fail (~2% chance), re-opened this one at medium priority
• Run #4201: --parallel test failing, increase timeout (2025-08-22), second instance total across thousands of runs of that test

Original overview:

The GitHub action to run all tests on various versions of Node and various operating systems has started returning intermittent test failures again.

• Passing run
• Failing run

The only difference (netlify.toml changed)

[mark-wiemer]

Will investigate this tomorrow if time/energy allows. Triggered a run on main branch hoping for a clean repro. Most recent runs on main do seem to have at least one failure :(

[jdmarshall]

Looks like a timing bug in the watch tests.

I wonder if this relates to #5372

If you want this all to go faster it probably needs changes so that mocha uses 'process.send' for lifecycle events like the watch operation being ready to run. That would likely shave a lot of time off of your tests as well, instead of waiting 4 seconds for all of them when it looks like even that is not quite enough.

[mark-wiemer]

Thanks, the goal for now is getting the tests to just pass consistently, then we can look at performance. I'm focusing on another issue right now but hope to come back to this one soon (TM)

[jdmarshall]

I was poking at it, but out of the box I’m getting consistent failures of the integration tests, not intermittent.

[mark-wiemer]

Rerunning a batch of 10 test runs and will check tomorrow, but the May 11th batch was intermittent. There have been some good bugfixes since then but nothing specifically addressing these. I was on a break but I'm back now and will actually really truly work on this tomorrow :)

[mark-wiemer]

@jdmarshall if you're still interested I'm curious if the consistent failures you're getting are for the same tests that are intermittently failing in GitHub Actions? Any failures are a problem, consistent or intermittent! Feel free to open a separate bug if you think they're unrelated

[jdmarshall]

It appears to be the same two every time.

For sure it has happened at least twice in my scroll buffer for that window.

M2 Mac, so who knows what cpu specific timing differences there might be.

  340 passing (2m)
  1 pending
  2 failing

  1) root hooks
       when mocha run in serial mode
         support ESM via .js extension w/o type=module
           should fail due to ambiguous file type
             with --no-experimental-detect-module:
     
expected Promise when fulfilled to contain output /SyntaxError: Unexpected token/
  expected
  {
    output: '\n\n  0 passing (1ms)\n\n',
    code: 0,
    args: [
      '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js',
      '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
      '--no-experimental-detect-module',
      '--no-color',
      '--no-bail',
      '--no-parallel'
    ],
    command: '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js --require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js --no-experimental-detect-module --no-color --no-bail --no-parallel'
  }
  to contain output /SyntaxError: Unexpected token/

  
  expected Promise when fulfilled to contain output /SyntaxError: Unexpected token/
    expected
    {
      output: '\n\n  0 passing (1ms)\n\n',
      code: 0,
      args: [
        '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
        '--no-experimental-detect-module',
        '--no-color',
        '--no-bail',
        '--no-parallel'
      ],
    }
    to contain output /SyntaxError: Unexpected token/
      at Array.forEach (<anonymous>)
  From previous event:
      at Context.<anonymous> (test/integration/plugins/root-hooks.spec.js:146:18)
      at callFn (lib/runnable.js:112:584)
      at Runnable.run (lib/runnable.js:112:57)
      at Runner.runTest (lib/runner.js:220:1054)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:248:654
      at next (lib/runner.js:197:422)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:197:809
      at next (lib/runner.js:185:579)
      at Immediate.<anonymous> (lib/runner.js:189:1253)
      at process.processImmediate (node:internal/timers:485:21)
      set UNEXPECTED_FULL_TRACE=true to see the full stack trace

  2) root hooks
       when mocha run in serial mode
         support ESM via .js extension w/o type=module
           should fail due to ambiguous file type
             with --experimental-detect-module:
     
expected Promise when fulfilled to contain output /Cannot require\(\) ES Module/
  expected
  {
    output: '\n\n  0 passing (1ms)\n\n',
    code: 0,
    args: [
      '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js',
      '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
      '--experimental-detect-module',
      '--no-color',
      '--no-bail',
      '--no-parallel'
    ],
    command: '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js --require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js --experimental-detect-module --no-color --no-bail --no-parallel'
  }
  to contain output /Cannot require\(\) ES Module/

  
  expected Promise when fulfilled to contain output /Cannot require\(\) ES Module/
    expected
    {
      output: '\n\n  0 passing (1ms)\n\n',
      code: 0,
      args: [
        '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
        '--experimental-detect-module',
        '--no-color',
        '--no-bail',
        '--no-parallel'
      ],
    }
    to contain output /Cannot require\(\) ES Module/
      at Array.forEach (<anonymous>)
  From previous event:
      at Context.<anonymous> (test/integration/plugins/root-hooks.spec.js:166:18)
      at callFn (lib/runnable.js:112:584)
      at Runnable.run (lib/runnable.js:112:57)
      at Runner.runTest (lib/runner.js:220:1054)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:248:654
      at next (lib/runner.js:197:422)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:197:809
      at next (lib/runner.js:185:579)
      at Immediate.<anonymous> (lib/runner.js:189:1253)
      at process.processImmediate (node:internal/timers:485:21)
      set UNEXPECTED_FULL_TRACE=true to see the full stack trace



ERROR: "test-node:integration" exited with 2.
ERROR: "test-node" exited with 1.

[jdmarshall]

I wonder if I should try plopping it into a docker image.