Improve timeout error messages with structured context#11
Conversation
Step timeout errors previously showed only generic "TimeoutError" messages without indicating which step failed, the configured timeout value, or total elapsed time. This made debugging timeout issues difficult in complex scenarios with multiple steps. The new StepTimeoutError includes: - stepName: identifies the failing step - timeoutMs: configured timeout value - attemptNumber: retry attempt when timeout occurred - elapsedMs: total execution time including retries This structured error information enables better error reporting and programmatic error handling in reporters and test frameworks.
Users can now set timeout limits for entire scenarios via RunOptions.timeout, providing control over scenario execution time independent of step-level timeouts. This addresses cases where the cumulative time of all steps in a scenario needs to be bounded, preventing long-running scenarios from blocking test suite completion. The implementation follows the same pattern as step timeout (deadline + AbortSignal + structured error), ensuring consistency and maintainability. Each scenario's timeout is enforced independently, so one timeout doesn't affect other scenarios' execution. - Add ScenarioTimeoutError with scenarioName, timeoutMs, and elapsedMs - Add timeout field to RunOptions interface - Implement timeout enforcement in Runner using deadline and AbortSignal - Add comprehensive tests for timeout behavior
There was a problem hiding this comment.
Pull request overview
This PR enhances timeout error handling by introducing structured error classes that provide detailed context when steps or scenarios time out. Previously, timeout errors showed only generic "TimeoutError" messages. Now they include the step/scenario name, configured timeout value, attempt number (for steps), and actual elapsed time.
Key Changes:
- Added
StepTimeoutErrorandScenarioTimeoutErrorclasses with structured fields for better debugging - Implemented scenario-level timeout configuration via
RunOptions.timeoutfield - Added retry attempt tracking to timeout errors using internal metadata
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/probitas-runner/errors.ts | New file defining StepTimeoutError and ScenarioTimeoutError with structured context |
| packages/probitas-runner/utils/retry.ts | Attaches retry attempt metadata to errors for context in timeout errors |
| packages/probitas-runner/step_runner.ts | Enriches timeout errors with retry context and elapsed time |
| packages/probitas-runner/runner.ts | Implements scenario-level timeout enforcement using deadline and AbortSignal |
| packages/probitas-runner/types/run.ts | Adds timeout field to RunOptions for scenario-level timeout configuration |
| packages/probitas-runner/step_runner_test.ts | Comprehensive tests for step timeout behavior with and without retry |
| packages/probitas-runner/runner_test.ts | Tests for scenario timeout with independent timeout handling and normal execution |
| packages/probitas-runner/mod.ts | Updates documentation to reference new error types and exports them |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /** | ||
| * Error with retry metadata attached | ||
| * @internal | ||
| */ | ||
| interface ErrorWithRetryMetadata extends Error { | ||
| __retryAttemptNumber?: number; | ||
| } |
There was a problem hiding this comment.
The ErrorWithRetryMetadata interface is defined identically in both this file and utils/retry.ts. Consider moving this to a shared location (e.g., a types file) or re-exporting it from one location to avoid duplication and ensure consistency.
| function isTimeoutError(error: unknown): boolean { | ||
| return error instanceof DOMException && error.name === "TimeoutError"; | ||
| } |
There was a problem hiding this comment.
The isTimeoutError function is duplicated in both runner.ts and step_runner.ts. Consider extracting this to a shared utility file to avoid duplication and ensure consistency.
lambdalisue
force-pushed
the
fix/timeout-error
branch
from
94155fb to
0655e4b
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
1 participant
Summary
StepTimeoutErrorclass with step name, timeout value, attempt number, and elapsed timeScenarioTimeoutErrorclass with scenario name, timeout value, and elapsed timetimeoutfield toRunOptionsfor scenario-level timeout configurationdeadlineandAbortSignalWhy
Previously, timeout errors showed only generic "TimeoutError" messages without context about which step or scenario failed, what the timeout limit was, or how long execution actually took. This made debugging timeout issues difficult in complex test scenarios.
These changes address two use cases:
Step-level timeouts: Users needed to know which specific step timed out, especially in scenarios with multiple steps and retry logic. The new
StepTimeoutErrorincludes the step name, configured timeout, retry attempt number, and total elapsed time.Scenario-level timeouts: Users needed to bound the cumulative execution time of all steps in a scenario to prevent long-running scenarios from blocking test suite completion. The new
RunOptions.timeoutprovides independent timeout control for each scenario, separate from step-level timeouts.The implementation follows a consistent pattern (deadline + AbortSignal + structured error) for maintainability. Each scenario's timeout is enforced independently, so one timeout doesn't affect other scenarios' execution.
Test Plan
deno task verify- all tests pass