Skip to content
Test Shared libraries

test_runner: support expecting a test-case to fail #1486

Sign in to view logs

test_runner: support expecting a test-case to fail

test_runner: support expecting a test-case to fail #1486

Triggered via pull request December 8, 2025 20:00
@JakobJingleheimerJakobJingleheimer
synchronize #60669
JakobJingleheimer:test_runner/feat/should-fail
Status Failure
Total duration 41m 50s
Artifacts 1

test-shared.yml

on: pull_request
Build slim tarball
1m 55s
Build slim tarball
Matrix: build
Fit to window
Zoom out
Zoom in

Annotations

8 errors
aarch64-darwin: with shared libraries
Process completed with exit code 2.
aarch64-darwin: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync throw fail'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: mixing des
aarch64-linux: with shared libraries
Process completed with exit code 2.
aarch64-linux: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync throw fail'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: mixing des
x86_64-linux: with shared libraries
Process completed with exit code 2.
x86_64-linux: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync throw fail'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: mixing des
x86_64-darwin: with shared libraries
Process completed with exit code 2.
x86_64-darwin: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'should not count as a failure'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync throw fail'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: mixing des

Artifacts

Produced during runtime
Name Size Digest
tarballs
47.8 MB
sha256:64642f7438f034111da2bd5152547b13a0718230d78f469add8d391dea69961c