Skip to content

docs: Document SKIPPED test result status and context matching behavior #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: Document SKIPPED test result status and context matching behavior #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions docs/get-started/config/contexts/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,60 @@ This context targets Windows or Linux:
}
```

## Skipped tests

When Doc Detective runs tests, it checks if each context matches your current environment. If a context specifies a platform or browser that doesn't match the system running the test, Doc Detective skips that context and its associated test.

Doc Detective skips tests when:

- **Platform mismatch**: The context specifies a platform (Windows, macOS, or Linux) that doesn't match your current operating system.
- **Browser unavailable**: The context specifies a browser that isn't installed or available on your system.
- **All contexts skipped**: If all contexts for a test are skipped, the entire test is marked as SKIPPED (not PASS).

### Test results

When a test is skipped, the results show a SKIPPED status in the output JSON:

```json
{
"summary": {
"tests": {
"pass": 0,
"fail": 0,
"warning": 0,
"skipped": 1 // Test was skipped
},
"contexts": {
"pass": 0,
"fail": 0,
"warning": 0,
"skipped": 1 // Context didn't match environment
}
},
"specs": [
{
"result": "SKIPPED",
"tests": [
{
"result": "SKIPPED",
"contexts": [
{
"result": "SKIPPED",
"platform": "windows", // Current platform: linux
"browser": {
"name": "chrome"
}
}
]
}
]
}
]
}
```

This keeps your test reporting accurate and prevents skipped tests from incorrectly appearing as passing.

## Examples

### Basic Contexts
Expand Down
2 changes: 1 addition & 1 deletion docs/get-started/intro.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ See the [Installation](/docs/get-started/installation) guide to get started. Com

Doc Detective is a doc content testing framework that simplifies the process of keeping your docs accurate and up-to-date. You write tests, and Doc Detective runs them directly against your product to ensure your docs match your user experience. Whether it’s a UI-based process or a series of API calls, Doc Detective can help you find doc bugs before your users do.

Doc Detective ingests test specifications and text files, parses them for testable actions, then executes those actions in a browser. The results (PASS/FAIL and context) are output as a JSON object so that other pieces of infrastructure can parse and manipulate them as needed.
Doc Detective ingests test specifications and text files, parses them for testable actions, then executes those actions in a browser. The results (PASS, FAIL, or SKIPPED) are output as a JSON object so that other pieces of infrastructure can parse and manipulate them as needed. Tests are marked as SKIPPED when they can't run in the current environment—for example, when a test specifies a browser or platform that doesn't match the system running the test.

## What can Doc Detective do?

Expand Down
Loading