Fix scenario timeout error handling and add scenario-level timeout configuration #13
Conversation
Previously, when a scenario timeout occurred, AbortError was thrown instead of the documented ScenarioTimeoutError. This was caused by using deadline() which throws DeadlineError that wasn't detected by isTimeoutError(). This fix: - Removes deadline() usage, relying solely on AbortSignal.timeout() - Detects StepTimeoutError and converts it to ScenarioTimeoutError - Simplifies the timeout handling logic Fixes issue where scenario timeout was reported as AbortError instead of the expected ScenarioTimeoutError with proper context.
…-level timeout configuration
Users can now configure timeout per scenario using the scenario() options:
scenario("Quick Test", { timeout: 5000 })
.step("Call API", async () => { ... })
.build();
This timeout applies to the total execution time of all steps in the
scenario. When the scenario exceeds this limit, ScenarioTimeoutError is
thrown.
Priority: scenario.timeout > RunOptions.timeout
Changes:
- Add timeout?: number field to ScenarioOptions, ScenarioDefinition, and
ScenarioMetadata in @probitas/core
- Update ScenarioBuilder to include timeout in built definition
- Update Runner to respect scenario-level timeout over global RunOptions
timeout
This allows fine-grained timeout control for specific scenarios while
maintaining backward compatibility with global timeouts.
There was a problem hiding this comment.
Pull request overview
This pull request fixes scenario timeout error handling and adds per-scenario timeout configuration. The main change removes the deadline() dependency in favor of AbortSignal.timeout(), which directly throws the expected TimeoutError (DOMException). Additionally, scenarios can now specify their own timeout values that override the global RunOptions timeout.
Key changes:
- Simplified timeout handling by removing
deadline()and relying solely onAbortSignal.timeout() - Added
timeoutfield toScenarioOptions,ScenarioDefinition, andScenarioMetadatawith comprehensive documentation - Enhanced timeout error detection to recognize both
TimeoutError(DOMException) andStepTimeoutError, properly converting them toScenarioTimeoutError
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| packages/probitas-runner/runner.ts | Removed deadline() dependency, simplified timeout handling logic, added scenario-level timeout precedence (scenario.timeout ?? timeout), and enhanced isTimeoutError() to detect StepTimeoutError |
| packages/probitas-core/types/scenario.ts | Added timeout field to ScenarioOptions, ScenarioDefinition, and ScenarioMetadata interfaces with comprehensive JSDoc documentation |
| packages/probitas-builder/scenario_builder.ts | Updated build() method to include timeout field from scenario options in the generated definition |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Priority: scenario timeout > RunOptions timeout | ||
| const effectiveTimeout = scenario.timeout ?? timeout; | ||
| const scenarioResult = effectiveTimeout > 0 | ||
| ? await this.#runWithTimeout( | ||
| scenarioRunner, | ||
| scenario, | ||
| timeout, | ||
| effectiveTimeout, |
There was a problem hiding this comment.
The new scenario-level timeout feature lacks test coverage. While there are existing tests for RunOptions timeout, there are no tests verifying that scenario.timeout takes precedence over RunOptions.timeout, or that scenarios with different timeout values work correctly together.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Summary
ScenarioTimeoutErrorinstead ofAbortErrortimeoutoption to scenario configuration for per-scenario timeout controldeadline()dependencyWhy
Problem: When a scenario timed out, the runner was throwing a generic
AbortErrorinstead of the documented
ScenarioTimeoutError. This happened becausedeadline()threw
DeadlineErrorwhich wasn't detected by the timeout error check.Solution: Remove
deadline()and rely solely onAbortSignal.timeout(), whichdirectly throws the expected
TimeoutError(DOMException). The timeout detection nowalso recognizes
StepTimeoutErrorand properly converts it toScenarioTimeoutError.Enhancement: Users can now set timeouts per scenario using the
scenario()options, allowing fine-grained control over execution time limits. This is useful
for scenarios that need different timeout constraints than the global default.
Priority:
scenario.timeout>RunOptions.timeoutTest Plan
ScenarioTimeoutErroris thrown correctly