Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/contribute/repos/doc-detective-core.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,6 @@ sidebar_position: 2

[`doc-detective-core`](https://github.com/doc-detective/doc-detective-core) is an NPM package that contains config and test logic and actually runs tests. It's installable via NPM (`npm i doc-detective-core`). This package contains the logic for performing each test action.

This package includes the parallel execution engine with configurable worker pool pattern for improved performance when running multiple test contexts simultaneously.

This repo depends on [`doc-detective-common`](doc-detective-common) for JSON schema definitions, schema validation logic, and path resolution logic.
143 changes: 143 additions & 0 deletions docs/get-started/config/parallel-execution.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
---
sidebar_position: 4
---

# Parallel Execution

Doc Detective supports parallel execution of test contexts using a configurable worker pool pattern. This feature can significantly improve performance for test suites with multiple contexts by running them concurrently instead of sequentially.

## How It Works

By default, Doc Detective executes test contexts sequentially (one after another). With parallel execution enabled, Doc Detective creates multiple worker processes that can execute different test contexts simultaneously while maintaining proper test isolation.

Each test context gets its own independent browser driver instance and execution environment, ensuring that tests don't interfere with each other even when running in parallel.

## Configuration

Enable parallel execution by setting the `concurrentRunners` field in your configuration:

```json
{
"input": "./tests",
"concurrentRunners": 4
}
```

### Configuration Options

- **`concurrentRunners`**: Number of concurrent workers to use
- Default: `1` (sequential execution)
- Recommended: 2-4 workers for most systems
- Maximum: Limited by your system's resources

## Performance Benefits

Parallel execution can provide significant performance improvements:

- **2 workers**: Typically 1.5-1.7x speedup
- **4 workers**: Typically 2-2.5x speedup
- **Higher worker counts**: Diminishing returns due to system overhead

The actual speedup depends on:
- Number of test contexts in your test suite
- System resources (CPU, memory)
- Test complexity and duration
- Browser startup overhead

## Best Practices

### Choosing Worker Count

- **Start with 2-4 workers** for most systems
- **Monitor system resources** when increasing worker count
- **Consider test suite size**: More workers than contexts provides no benefit
- **Test different configurations** to find optimal performance

### Test Design Considerations

- **Ensure test isolation**: Tests should not depend on shared state
- **Avoid resource conflicts**: Tests shouldn't compete for the same files or ports
- **Use unique identifiers**: Generate unique test data to prevent conflicts

### System Requirements

- **Memory**: Each worker requires additional memory for browser instances
- **CPU**: Multiple workers benefit from multi-core processors
- **Browser resources**: Each context gets its own browser driver

## Examples

### Basic Parallel Configuration

```json
{
"input": "./tests",
"concurrentRunners": 2,
"logLevel": "info"
}
```

### Advanced Configuration with Multiple Contexts

```json
{
"input": "./tests",
"concurrentRunners": 4,
"runOn": [
{
"platform": "linux",
"browser": "chrome"
},
{
"platform": "linux",
"browser": "firefox"
}
]
}
```

### Command Line Override

You can override the configuration from the command line:

```bash
npx doc-detective --config .doc-detective.json --concurrentRunners 3
```

## Troubleshooting

### Performance Issues

- **Reduce worker count** if system becomes unresponsive
- **Monitor memory usage** - each worker uses additional RAM
- **Check for resource contention** between parallel tests

### Test Failures

- **Verify test isolation** - ensure tests don't share state
- **Check for timing issues** - parallel execution may expose race conditions
- **Review error logs** - failed contexts are reported with detailed error information

### Browser Issues

- **Ensure sufficient browser resources** for multiple instances
- **Consider headless mode** to reduce resource usage
- **Check browser driver compatibility** with parallel execution

## Migration from Sequential Execution

Existing test suites work unchanged with parallel execution:

1. **No breaking changes** - all existing tests continue to work
2. **Gradual adoption** - start with `concurrentRunners: 2`
3. **Monitor results** - verify test outcomes remain consistent
4. **Optimize incrementally** - increase worker count based on performance gains

## Error Handling

Parallel execution maintains robust error handling:

- **Isolated failures**: Error in one context doesn't terminate others
- **Complete results**: All test results are collected regardless of execution order
- **Detailed reporting**: Failed contexts include full error details
- **Graceful cleanup**: Resources are properly cleaned up even on failures
2 changes: 2 additions & 0 deletions docs/get-started/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ Doc Detective is a doc content testing framework that simplifies the process of

Doc Detective ingests test specifications and text files, parses them for testable actions, then executes those actions in a browser. The results (PASS/FAIL and context) are output as a JSON object so that other pieces of infrastructure can parse and manipulate them as needed.

For improved performance, Doc Detective supports parallel execution of test contexts, allowing multiple tests to run simultaneously with configurable worker pools.

## What can Doc Detective do?

Doc Detective can do a lot, especially with a bit of creativity.
Expand Down
10 changes: 10 additions & 0 deletions docs/get-started/sample-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,16 @@ You can override config options with command-line arguments. For example, to run
npx doc-detective --config .doc-detective.json --input tests.spec.json
```

## Performance optimization

For test suites with multiple contexts, you can improve performance using parallel execution:

```bash
npx doc-detective --concurrentRunners 4
```

This runs up to 4 test contexts simultaneously instead of sequentially. See the [parallel execution guide](/docs/get-started/config/parallel-execution) for detailed configuration options and best practices.

<!-- ### Run remotely hosted tests

You can run tests hosted remotely by specifying the URL of the test file with the `--input` argument. For example, to run tests from a file hosted at `https://doc-detective.com/sample.spec.json`, run the following command:
Expand Down
19 changes: 13 additions & 6 deletions docs/references/schemas/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ Configuration options for Doc Detective operations.

Field | Type | Description | Default
:-- | :-- | :-- | :--
$schema | string | Optional. JSON Schema for this object.<br/><br/>Accepted values: `https://raw.githubusercontent.com/doc-detective/common/refs/heads/main/dist/schemas/config_v3.schema.json` |
configId | string | Optional. Identifier for the configuration. |
configPath | string | ReadOnly. Path to the configuration file. |
input | one of:<br/>- string<br/>- array of string | Optional. Path(s) to test specifications and documentation source files. May be paths to specific files or to directories to scan for files. | `.`
Expand All @@ -19,13 +18,13 @@ origin | string | Optional. Default protocol and domain to use for relative URLs
beforeAny | one of:<br/>- string<br/>- array of string | Optional. Path(s) to test specifications to perform before those specified by `input`. Useful for setting up testing environments. |
afterAll | one of:<br/>- string<br/>- array of string | Optional. Path(s) to test specifications to perform after those specified by `input`. Useful for cleaning up testing environments. |
detectSteps | boolean | Optional. Whether or not to detect steps in input files based on defined markup. | `true`
allowUnsafeSteps | boolean | Optional. Whether or not to run potentially unsafe steps, such as those that might modify files or system state. |
logLevel | string | Optional. Amount of detail to output when performing an operation.<br/><br/>Accepted values: `silent`, `error`, `warning`, `info`, `debug` | `info`
concurrentRunners | number | Optional. Number of concurrent workers to use for parallel test context execution. Set to 1 for sequential execution (default behavior). Higher values can significantly improve performance for test suites with multiple contexts. | `1`
runOn | array of object([context](/docs/references/schemas/context)) | Optional. Contexts to run the test in. Overrides contexts defined at the config and spec levels. |
fileTypes | array of one of: string, object([File type (custom)](/docs/references/schemas/file-type-custom)), object([File type (executable)](/docs/references/schemas/file-type-executable)) | Optional. Configuration for file types and their markup detection. | ``["markdown","asciidoc","html"]``
integrations | object([Integrations options](/docs/references/schemas/integrations-options)) | Optional. Options for connecting to external services. |
telemetry | object([Telemetry options](/docs/references/schemas/telemetry-options)) | Optional. Options around sending telemetry for Doc Detective usage. | ``{"send":true}``
environment | object([Environment details](/docs/references/schemas/environment-details)) | ReadOnly. Environment information for the system running Doc Detective. |
fileTypes | array of one of: string, object([File type (custom)](/docs/references/schemas/File%20type%20(custom))), object([File type (executable)](/docs/references/schemas/File%20type%20(executable))) | Optional. Configuration for file types and their markup detection. | ``["markdown","asciidoc","html"]``
integrations | object([Integrations options](/docs/references/schemas/Integrations%20options)) | Optional. Options for connecting to external services. |
telemetry | object([Telemetry options](/docs/references/schemas/Telemetry%20options)) | Optional. Options around sending telemetry for Doc Detective usage. | ``{"send":true}``
environment | object([Environment details](/docs/references/schemas/Environment%20details)) | ReadOnly. Environment information for the system running Doc Detective. |

## Examples

Expand All @@ -45,6 +44,14 @@ environment | object([Environment details](/docs/references/schemas/environment-
}
```

```json
{
"input": "./tests",
"concurrentRunners": 4,
"logLevel": "info"
}
```

```json
{
"fileTypes": [
Expand Down
Loading