Skip to content

Commit 7834cc8

Browse files
promptless[bot]promptless[bot]
authored
Documentation updates from Promptless
1 parent 778dcb1 commit 7834cc8

File tree

5 files changed

+170
-6
lines changed

5 files changed

+170
-6
lines changed

‎docs/contribute/repos/doc-detective-core.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,6 @@ sidebar_position: 2
66

77
[`doc-detective-core`](https://github.com/doc-detective/doc-detective-core) is an NPM package that contains config and test logic and actually runs tests. It's installable via NPM (`npm i doc-detective-core`). This package contains the logic for performing each test action.
88

9+
This package includes the parallel execution engine with configurable worker pool pattern for improved performance when running multiple test contexts simultaneously.
10+
911
This repo depends on [`doc-detective-common`](doc-detective-common) for JSON schema definitions, schema validation logic, and path resolution logic.

‎docs/get-started/config/parallel-execution.md‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
sidebar_position: 4
3+
---
4+
5+
# Parallel Execution
6+
7+
Doc Detective supports parallel execution of test contexts using a configurable worker pool pattern. This feature can significantly improve performance for test suites with multiple contexts by running them concurrently instead of sequentially.
8+
9+
## How It Works
10+
11+
By default, Doc Detective executes test contexts sequentially (one after another). With parallel execution enabled, Doc Detective creates multiple worker processes that can execute different test contexts simultaneously while maintaining proper test isolation.
12+
13+
Each test context gets its own independent browser driver instance and execution environment, ensuring that tests don't interfere with each other even when running in parallel.
14+
15+
## Configuration
16+
17+
Enable parallel execution by setting the `concurrentRunners` field in your configuration:
18+
19+
```json
20+
{
21+
"input": "./tests",
22+
"concurrentRunners": 4
23+
}
24+
```
25+
26+
### Configuration Options
27+
28+
- **`concurrentRunners`**: Number of concurrent workers to use
29+
- Default: `1` (sequential execution)
30+
- Recommended: 2-4 workers for most systems
31+
- Maximum: Limited by your system's resources
32+
33+
## Performance Benefits
34+
35+
Parallel execution can provide significant performance improvements:
36+
37+
- **2 workers**: Typically 1.5-1.7x speedup
38+
- **4 workers**: Typically 2-2.5x speedup
39+
- **Higher worker counts**: Diminishing returns due to system overhead
40+
41+
The actual speedup depends on:
42+
- Number of test contexts in your test suite
43+
- System resources (CPU, memory)
44+
- Test complexity and duration
45+
- Browser startup overhead
46+
47+
## Best Practices
48+
49+
### Choosing Worker Count
50+
51+
- **Start with 2-4 workers** for most systems
52+
- **Monitor system resources** when increasing worker count
53+
- **Consider test suite size**: More workers than contexts provides no benefit
54+
- **Test different configurations** to find optimal performance
55+
56+
### Test Design Considerations
57+
58+
- **Ensure test isolation**: Tests should not depend on shared state
59+
- **Avoid resource conflicts**: Tests shouldn't compete for the same files or ports
60+
- **Use unique identifiers**: Generate unique test data to prevent conflicts
61+
62+
### System Requirements
63+
64+
- **Memory**: Each worker requires additional memory for browser instances
65+
- **CPU**: Multiple workers benefit from multi-core processors
66+
- **Browser resources**: Each context gets its own browser driver
67+
68+
## Examples
69+
70+
### Basic Parallel Configuration
71+
72+
```json
73+
{
74+
"input": "./tests",
75+
"concurrentRunners": 2,
76+
"logLevel": "info"
77+
}
78+
```
79+
80+
### Advanced Configuration with Multiple Contexts
81+
82+
```json
83+
{
84+
"input": "./tests",
85+
"concurrentRunners": 4,
86+
"runOn": [
87+
{
88+
"platform": "linux",
89+
"browser": "chrome"
90+
},
91+
{
92+
"platform": "linux",
93+
"browser": "firefox"
94+
}
95+
]
96+
}
97+
```
98+
99+
### Command Line Override
100+
101+
You can override the configuration from the command line:
102+
103+
```bash
104+
npx doc-detective --config .doc-detective.json --concurrentRunners 3
105+
```
106+
107+
## Troubleshooting
108+
109+
### Performance Issues
110+
111+
- **Reduce worker count** if system becomes unresponsive
112+
- **Monitor memory usage** - each worker uses additional RAM
113+
- **Check for resource contention** between parallel tests
114+
115+
### Test Failures
116+
117+
- **Verify test isolation** - ensure tests don't share state
118+
- **Check for timing issues** - parallel execution may expose race conditions
119+
- **Review error logs** - failed contexts are reported with detailed error information
120+
121+
### Browser Issues
122+
123+
- **Ensure sufficient browser resources** for multiple instances
124+
- **Consider headless mode** to reduce resource usage
125+
- **Check browser driver compatibility** with parallel execution
126+
127+
## Migration from Sequential Execution
128+
129+
Existing test suites work unchanged with parallel execution:
130+
131+
1. **No breaking changes** - all existing tests continue to work
132+
2. **Gradual adoption** - start with `concurrentRunners: 2`
133+
3. **Monitor results** - verify test outcomes remain consistent
134+
4. **Optimize incrementally** - increase worker count based on performance gains
135+
136+
## Error Handling
137+
138+
Parallel execution maintains robust error handling:
139+
140+
- **Isolated failures**: Error in one context doesn't terminate others
141+
- **Complete results**: All test results are collected regardless of execution order
142+
- **Detailed reporting**: Failed contexts include full error details
143+
- **Graceful cleanup**: Resources are properly cleaned up even on failures

‎docs/get-started/intro.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ Doc Detective is a doc content testing framework that simplifies the process of
2121

2222
Doc Detective ingests test specifications and text files, parses them for testable actions, then executes those actions in a browser. The results (PASS/FAIL and context) are output as a JSON object so that other pieces of infrastructure can parse and manipulate them as needed.
2323

24+
For improved performance, Doc Detective supports parallel execution of test contexts, allowing multiple tests to run simultaneously with configurable worker pools.
25+
2426
## What can Doc Detective do?
2527

2628
Doc Detective can do a lot, especially with a bit of creativity.

‎docs/get-started/sample-tests.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,16 @@ You can override config options with command-line arguments. For example, to run
3838
npx doc-detective --config .doc-detective.json --input tests.spec.json
3939
```
4040

41+
## Performance optimization
42+
43+
For test suites with multiple contexts, you can improve performance using parallel execution:
44+
45+
```bash
46+
npx doc-detective --concurrentRunners 4
47+
```
48+
49+
This runs up to 4 test contexts simultaneously instead of sequentially. See the [parallel execution guide](/docs/get-started/config/parallel-execution) for detailed configuration options and best practices.
50+
4151
<!-- ### Run remotely hosted tests
4252
4353
You can run tests hosted remotely by specifying the URL of the test file with the `--input` argument. For example, to run tests from a file hosted at `https://doc-detective.com/sample.spec.json`, run the following command:

‎docs/references/schemas/config.md‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ Configuration options for Doc Detective operations.
77

88
Field | Type | Description | Default
99
:-- | :-- | :-- | :--
10-
$schema | string | Optional. JSON Schema for this object.<br/><br/>Accepted values: `https://raw.githubusercontent.com/doc-detective/common/refs/heads/main/dist/schemas/config_v3.schema.json` |
1110
configId | string | Optional. Identifier for the configuration. |
1211
configPath | string | ReadOnly. Path to the configuration file. |
1312
input | one of:<br/>- string<br/>- array of string | Optional. Path(s) to test specifications and documentation source files. May be paths to specific files or to directories to scan for files. | `.`
@@ -19,13 +18,13 @@ origin | string | Optional. Default protocol and domain to use for relative URLs
1918
beforeAny | one of:<br/>- string<br/>- array of string | Optional. Path(s) to test specifications to perform before those specified by `input`. Useful for setting up testing environments. |
2019
afterAll | one of:<br/>- string<br/>- array of string | Optional. Path(s) to test specifications to perform after those specified by `input`. Useful for cleaning up testing environments. |
2120
detectSteps | boolean | Optional. Whether or not to detect steps in input files based on defined markup. | `true`
22-
allowUnsafeSteps | boolean | Optional. Whether or not to run potentially unsafe steps, such as those that might modify files or system state. |
2321
logLevel | string | Optional. Amount of detail to output when performing an operation.<br/><br/>Accepted values: `silent`, `error`, `warning`, `info`, `debug` | `info`
22+
concurrentRunners | number | Optional. Number of concurrent workers to use for parallel test context execution. Set to 1 for sequential execution (default behavior). Higher values can significantly improve performance for test suites with multiple contexts. | `1`
2423
runOn | array of object([context](/docs/references/schemas/context)) | Optional. Contexts to run the test in. Overrides contexts defined at the config and spec levels. |
25-
fileTypes | array of one of: string, object([File type (custom)](/docs/references/schemas/file-type-custom)), object([File type (executable)](/docs/references/schemas/file-type-executable)) | Optional. Configuration for file types and their markup detection. | ``["markdown","asciidoc","html"]``
26-
integrations | object([Integrations options](/docs/references/schemas/integrations-options)) | Optional. Options for connecting to external services. |
27-
telemetry | object([Telemetry options](/docs/references/schemas/telemetry-options)) | Optional. Options around sending telemetry for Doc Detective usage. | ``{"send":true}``
28-
environment | object([Environment details](/docs/references/schemas/environment-details)) | ReadOnly. Environment information for the system running Doc Detective. |
24+
fileTypes | array of one of: string, object([File type (custom)](/docs/references/schemas/File%20type%20(custom))), object([File type (executable)](/docs/references/schemas/File%20type%20(executable))) | Optional. Configuration for file types and their markup detection. | ``["markdown","asciidoc","html"]``
25+
integrations | object([Integrations options](/docs/references/schemas/Integrations%20options)) | Optional. Options for connecting to external services. |
26+
telemetry | object([Telemetry options](/docs/references/schemas/Telemetry%20options)) | Optional. Options around sending telemetry for Doc Detective usage. | ``{"send":true}``
27+
environment | object([Environment details](/docs/references/schemas/Environment%20details)) | ReadOnly. Environment information for the system running Doc Detective. |
2928

3029
## Examples
3130

@@ -45,6 +44,14 @@ environment | object([Environment details](/docs/references/schemas/environment-
4544
}
4645
```
4746

47+
```json
48+
{
49+
"input": "./tests",
50+
"concurrentRunners": 4,
51+
"logLevel": "info"
52+
}
53+
```
54+
4855
```json
4956
{
5057
"fileTypes": [

0 commit comments

Comments
 (0)