Fix incorrect PASS status when all contexts are skipped #368

Copilot · 2025-11-04T05:09:29Z

Tests were incorrectly marked as PASS when all their contexts were skipped due to platform/browser incompatibility, creating false positives in test reports.

Root Cause

Context results used inconsistent types:

Skipped contexts (platform/browser mismatch): { status: "SKIPPED" } (object)
Completed contexts: "PASS", "FAIL", "SKIPPED" (string)

Test result aggregation logic used string comparison (context.result === "SKIPPED"), which failed for object results, causing tests to default to PASS.

Changes

src/tests.js: Normalized skipped context results to use string "SKIPPED" instead of object { status: "SKIPPED" }
src/tests.js: Error messages moved to resultDescription field to maintain consistency
test/core.test.js: Added test case verifying proper SKIPPED propagation through context → test → spec hierarchy
.gitignore: Excluded test/artifacts/output/ directory

Example

Before:

// Context skipped due to platform mismatch
contextReport = { result: { status: "SKIPPED" }, ...contextReport };

// Later comparison fails
testReport.contexts.filter((c) => c.result === "SKIPPED")  // Returns []

After:

// Context skipped due to platform mismatch  
contextReport = { result: "SKIPPED", ...contextReport };

// Comparison succeeds
testReport.contexts.filter((c) => c.result === "SKIPPED")  // Returns skipped contexts

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

accounts.google.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=3651 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.yaWICA --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,2913513491401440591,2411595973271585913,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --allow-pre-commit-input --auto-select-desktop-capture-source=RECORD_ME --disable-REDACTED-networking --disable-client-side-phishing-detection --disable-default-apps --disable-features=IgnoreDuplicateNavs,Prewarm --disable-gpu --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-chrome-browser-cloud-management --enable-logging --headless --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=0 --test-type=webdriver --use-mock-keychain --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.yaWICA --noerrdialogs --ozone-platform=headless --ozone-override-screen-size=800,600 --use-angle=swiftshader-webgl data:, (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=9274 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.oyHA3L --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,13501584547293152914,8236781060677532180,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
ads.mozilla.org
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile4wGNIR (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileT5ngTn (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileRdxEoK (dns block)
app.posthog.com
- Triggering command: node /home/REDACTED/work/core/core/node_modules/.bin/mocha test/core.test.js (dns block)
- Triggering command: node /tmp/reproduce-issue.js (dns block)
clients2.google.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=3651 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.yaWICA --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,2913513491401440591,2411595973271585913,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --allow-pre-commit-input --auto-select-desktop-capture-source=RECORD_ME --disable-REDACTED-networking --disable-client-side-phishing-detection --disable-default-apps --disable-features=IgnoreDuplicateNavs,Prewarm --disable-gpu --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-chrome-browser-cloud-management --enable-logging --headless --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=0 --test-type=webdriver --use-mock-keychain --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.yaWICA --noerrdialogs --ozone-platform=headless --ozone-override-screen-size=800,600 --use-angle=swiftshader-webgl data:, (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=9274 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.oyHA3L --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,13501584547293152914,8236781060677532180,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
detectportal.firefox.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile4wGNIR (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile6yuiTo (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileT5ngTn (dns block)
doc-detective.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileXiGEv0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofilerZ3yiP (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileqWd4Y5 (dns block)
duckduckgo.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileKi8O2t (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile9xkCDp (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile9A2GfI (dns block)
fonts.googleapis.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileT5ngTn (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileRdxEoK (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileEkkaYw (dns block)
google.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=3651 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.yaWICA --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,2913513491401440591,2411595973271585913,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=9274 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.oyHA3L --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,13501584547293152914,8236781060677532180,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=14590 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.viAiYl --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,17551932443807437963,8083423237786817116,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
incoming.telemetry.mozilla.org
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile4wGNIR (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileT5ngTn (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofilebDg0Au (dns block)
kennethreitz.org
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileT5ngTn (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileRdxEoK (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileEkkaYw (dns block)
prod.ohttp-gateway.prod.webservices.mozgcp.net
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile4wGNIR (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile6yuiTo (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileT5ngTn (dns block)
spocs.getpocket.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofilebDg0Au (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileRdxEoK (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileXiGEv0 (dns block)
support.mozilla.org
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofile4wGNIR (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileXiGEv0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileaL4Sk6 (dns block)
www.doc-detective.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileXiGEv0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofilerZ3yiP (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileqWd4Y5 (dns block)
www.duckduckgo.com
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileaL4Sk6 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/firefox/linux-nightly_146.0a1/firefox/firefox --marionette --headless -no-remote -profile /tmp/rust_mozprofileKi8O2t (dns block)
- Triggering command: node /home/REDACTED/work/core/core/node_modules/.bin/mocha test/core.test.js (dns block)
www.google.com
- Triggering command: node /home/REDACTED/work/core/core/node_modules/.bin/mocha test/core.test.js (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=network --no-sandbox --use-angle=swiftshader-webgl --crashpad-handler-pid=3651 --enable-crash-reporter=, --noerrdialogs --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.yaWICA --change-stack-guard-on-fork=enable --shared-files=network_parent_dirs_pipe:100,v8_context_snapshot_data:101 --field-trial-handle=3,i,2913513491401440591,2411595973271585913,262144 --disable-features=IgnoreDuplicateNavs,PaintHolding,Prewarm --variations-seed-version --trace-process-track-uuid=3190708989122997041 --enable-logging --log-level=0 (dns block)
- Triggering command: /home/REDACTED/work/core/core/browser-snapshots/chrome/linux-142.0.7444.59/chrome-linux64/chrome --allow-pre-commit-input --auto-select-desktop-capture-source=RECORD_ME --disable-REDACTED-networking --disable-client-side-phishing-detection --disable-default-apps --disable-features=IgnoreDuplicateNavs,Prewarm --disable-gpu --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-chrome-browser-cloud-management --enable-logging --headless --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=0 --test-type=webdriver --use-mock-keychain --user-data-dir=/tmp/.org.chromium.Chromium.scoped_dir.yaWICA --noerrdialogs --ozone-platform=headless --ozone-override-screen-size=800,600 --use-angle=swiftshader-webgl data:, (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

This section details on the original issue you should resolve

<issue_title>Inaccurate skipped reporting </issue_title>
<issue_description>When building the test results, if all contexts are skipped (meaning 0 passed and 0 failed), the containing test should be marked skipped. Today the test is marked as passed, which is a false positive.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Inaccurate skipped reporting #367

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

coderabbitai · 2025-11-04T05:09:37Z

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Co-authored-by: hawkeyexl <[email protected]>

promptless · 2025-11-04T19:19:32Z

📝 Documentation updates detected!

New suggestion: Document SKIPPED test result status and context matching behavior

Initial plan

e1afc01

Copilot AI assigned Copilot and hawkeyexl Nov 4, 2025

Copilot started work on behalf of hawkeyexl November 4, 2025 05:09 View session

Copilot AI and others added 4 commits November 4, 2025 05:18

Initial exploration - identified the root cause

765e9ec

Co-authored-by: hawkeyexl <[email protected]>

Fix inaccurate skipped reporting and add test

8d41d01

Co-authored-by: hawkeyexl <[email protected]>

Improve test comment clarity

b9a9f03

Co-authored-by: hawkeyexl <[email protected]>

Remove test artifact and update gitignore

e8a45f4

Co-authored-by: hawkeyexl <[email protected]>

Copilot AI changed the title ~~[WIP] Fix inaccurate reporting of skipped tests~~ Fix incorrect PASS status when all contexts are skipped Nov 4, 2025

Copilot AI requested a review from hawkeyexl November 4, 2025 05:43

Copilot finished work on behalf of hawkeyexl November 4, 2025 05:43

Dep updates

f0eef77

hawkeyexl marked this pull request as ready for review November 4, 2025 19:07

hawkeyexl approved these changes Nov 4, 2025

View reviewed changes

hawkeyexl merged commit 6a61bf6 into main Nov 4, 2025
19 checks passed

hawkeyexl deleted the copilot/fix-inaccurate-skipped-reporting branch November 4, 2025 19:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix incorrect PASS status when all contexts are skipped #368

Fix incorrect PASS status when all contexts are skipped #368

Uh oh!

Copilot AI commented Nov 4, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Nov 4, 2025

Review skipped

Uh oh!

promptless bot commented Nov 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Fix incorrect PASS status when all contexts are skipped #368

Fix incorrect PASS status when all contexts are skipped #368

Uh oh!

Conversation

Copilot AI commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root Cause

Changes

Example

I tried to connect to the following addresses, but was blocked by firewall rules:

Comments on the Issue (you are @copilot in this section)

Uh oh!

coderabbitai bot commented Nov 4, 2025

Review skipped

Uh oh!

promptless bot commented Nov 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Nov 4, 2025 •

edited

Loading