Skip to content
Test Shared libraries

test_runner: support expecting a test-case to fail #1461

Sign in to view logs

test_runner: support expecting a test-case to fail

test_runner: support expecting a test-case to fail #1461

Triggered via pull request December 7, 2025 19:26
@JakobJingleheimerJakobJingleheimer
ready_for_review #60669
JakobJingleheimer:test_runner/feat/should-fail
Status Failure
Total duration 3h 26m 32s
Artifacts 1

test-shared.yml

on: pull_request
Build slim tarball
1m 29s
Build slim tarball
Matrix: build
Fit to window
Zoom out
Zoom in

Annotations

8 errors
aarch64-darwin: with shared libraries
Process completed with exit code 2.
aarch64-darwin: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected ... Skipped lines 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + - 'ok 1 - sync pass todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + - 'ok 2 - sync pass todo with message # TODO this is a passing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + - 'not ok 3 - sync todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + " failureType: 'testCodeFailure'\n" + ... ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + - 'not ok 4 - sync todo with message # TODO this is a failing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + - 'ok 5 - sync skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + - 'ok 6 - sync skip pass with message # SKIP this is skipped\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + - 'ok 7 - sync pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + - 'not ok 8 - sync throw fail\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + - 'ok 9 - async skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + - 'ok 10 - async pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: mixing describe/it and test should work\n' + + 'ok 17 - mixing describe/it and test should work\n' + - 'ok 11 - mixing describe/it and test should work\n' + ' ---\n' + ' duration_ms: *\n' + "
aarch64-linux: with shared libraries
Process completed with exit code 2.
aarch64-linux: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected ... Skipped lines 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + - 'ok 1 - sync pass todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + - 'ok 2 - sync pass todo with message # TODO this is a passing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + - 'not ok 3 - sync todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + " failureType: 'testCodeFailure'\n" + ... ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + - 'not ok 4 - sync todo with message # TODO this is a failing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + - 'ok 5 - sync skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + - 'ok 6 - sync skip pass with message # SKIP this is skipped\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + - 'ok 7 - sync pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + - 'not ok 8 - sync throw fail\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + - 'ok 9 - async skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + - 'ok 10 - async pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: mixing describe/it and test should work\n' + + 'ok 17 - mixing describe/it and test should work\n' + - 'ok 11 - mixing describe/it and test should work\n' + ' ---\n' + ' duration_ms: *\n' + "
x86_64-linux: with shared libraries
Process completed with exit code 2.
x86_64-linux: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected ... Skipped lines 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + - 'ok 1 - sync pass todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + - 'ok 2 - sync pass todo with message # TODO this is a passing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + - 'not ok 3 - sync todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + " failureType: 'testCodeFailure'\n" + ... ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + - 'not ok 4 - sync todo with message # TODO this is a failing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + - 'ok 5 - sync skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + - 'ok 6 - sync skip pass with message # SKIP this is skipped\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + - 'ok 7 - sync pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + - 'not ok 8 - sync throw fail\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + - 'ok 9 - async skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + - 'ok 10 - async pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: mixing describe/it and test should work\n' + + 'ok 17 - mixing describe/it and test should work\n' + - 'ok 11 - mixing describe/it and test should work\n' + ' ---\n' + ' duration_ms: *\n' + "
x86_64-darwin: with shared libraries
Process completed with exit code 2.
x86_64-darwin: with shared libraries: test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected ... Skipped lines 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + - 'ok 1 - sync pass todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + - 'ok 2 - sync pass todo with message # TODO this is a passing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + - 'not ok 3 - sync todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + " failureType: 'testCodeFailure'\n" + ... ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + - 'not ok 4 - sync todo with message # TODO this is a failing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + - 'ok 5 - sync skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + - 'ok 6 - sync skip pass with message # SKIP this is skipped\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + - 'ok 7 - sync pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + - 'not ok 8 - sync throw fail\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + - 'ok 9 - async skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + - 'ok 10 - async pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: mixing describe/it and test should work\n' + + 'ok 17 - mixing describe/it and test should work\n' + - 'ok 11 - mixing describe/it and test should work\n' + ' ---\n' + ' duration_ms: *\n' + "

Artifacts

Produced during runtime
Name Size Digest
tarballs
47.8 MB
sha256:02c143ad1b9308b49d540c4d0de4d44da2b317a81f7b67688a45edce1ab84624