Skip to content

Commit a3b26ad

Browse files
adam0whiteadam0white
committed
Update Sprint Status and Add New Stories for AI Test Agent Development
- Updated the sprint-status.yaml to reflect the current status of various epics and stories, marking several tasks as 'done' and 'ready-for-dev'. - Introduced new story files for the TestAgent Durable Object, including detailed context and acceptance criteria for stories 2.1 through 2.6, focusing on the implementation of the testing pipeline phases. - Added comprehensive documentation for error handling and user-friendly messaging in story 2.7, ensuring a consistent approach to managing failures across all phases. This commit addresses the ongoing development of the AI Test Agent and its associated testing framework.
1 parent 9634207 commit a3b26ad

15 files changed

+3672
-7
lines changed

‎docs/sprint-status.yaml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -45,13 +45,13 @@ development_status:
4545
epic-1-retrospective: completed
4646

4747
epic-2: contexted
48-
2-1-testagent-durable-object-skeleton: backlog
49-
2-2-browser-rendering-integration-and-stagehand-setup: backlog
50-
2-3-phase-1-load-validation: backlog
51-
2-4-phase-2-control-discovery: backlog
52-
2-5-phase-3-gameplay-exploration-with-computer-use: backlog
53-
2-6-phase-4-evaluation-scoring: backlog
54-
2-7-graceful-error-handling-and-user-friendly-messages: backlog
48+
2-1-testagent-durable-object-skeleton: done
49+
2-2-browser-rendering-integration-and-stagehand-setup: ready-for-dev
50+
2-3-phase-1-load-validation: ready-for-dev
51+
2-4-phase-2-control-discovery: ready-for-dev
52+
2-5-phase-3-gameplay-exploration-with-computer-use: ready-for-dev
53+
2-6-phase-4-evaluation-scoring: ready-for-dev
54+
2-7-graceful-error-handling-and-user-friendly-messages: ready-for-dev
5555
epic-2-retrospective: optional
5656

5757
epic-3: backlog

‎docs/stories/2-1-testagent-durable-object-skeleton.context.xml‎

Lines changed: 268 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/stories/2-1-testagent-durable-object-skeleton.md‎

Lines changed: 329 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/stories/2-2-browser-rendering-integration-and-stagehand-setup.context.xml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
<story-context id="bmad/bmm/workflows/4-implementation/story-context/template" v="1.0">
2+
<metadata>
3+
<epicId>2</epicId>
4+
<storyId>2.2</storyId>
5+
<title>Browser Rendering Integration and Stagehand Setup</title>
6+
<status>drafted</status>
7+
<generatedAt>2025-01-27</generatedAt>
8+
<generator>BMAD Story Context Workflow</generator>
9+
<sourceStoryPath>docs/stories/2-2-browser-rendering-integration-and-stagehand-setup.md</sourceStoryPath>
10+
</metadata>
11+
12+
<story>
13+
<asA>developer</asA>
14+
<iWant>Stagehand integrated with Cloudflare Browser Rendering</iWant>
15+
<soThat>the TestAgent can control a browser session.</soThat>
16+
<tasks>
17+
<task id="1">Configure Browser Rendering Service Binding (AC: 1)</task>
18+
<task id="2">Install and Initialize Stagehand Library (AC: 2)</task>
19+
<task id="3">Implement launchBrowser() Helper Method (AC: 3, 6)</task>
20+
<task id="4">Implement Browser Session Persistence (AC: 4)</task>
21+
<task id="5">Implement closeBrowser() Helper Method (AC: 5)</task>
22+
<task id="6">Enable Console Log Capture (AC: 7)</task>
23+
<task id="7">Enable Network Request Monitoring (AC: 8)</task>
24+
<task id="8">Implement captureScreenshot() Method (AC: 9)</task>
25+
<task id="9">Update TypeScript Types and Interfaces</task>
26+
</tasks>
27+
</story>
28+
29+
<acceptanceCriteria>
30+
<ac id="1">Browser Rendering service binding configured: TestAgent DO has access to env.BROWSER binding for Browser Rendering service</ac>
31+
<ac id="2">Stagehand library initialized: Stagehand instance created with Browser Rendering connection and configured for Computer Use mode</ac>
32+
<ac id="3">Helper method launchBrowser(): Creates browser session, configures viewport and user agent, returns Stagehand instance</ac>
33+
<ac id="4">Browser session persists in DO state: Browser session handle stored in DO state and reused across phases 1-3</ac>
34+
<ac id="5">Helper method closeBrowser(): Cleanly terminates browser session, releases resources</ac>
35+
<ac id="6">Browser configuration: Headless mode enabled, viewport 1280x720, user agent string set to 'GameEval TestAgent/1.0'</ac>
36+
<ac id="7">Console log capture enabled: Console logs from browser session streamed to DO state, accumulated in memory</ac>
37+
<ac id="8">Network request monitoring enabled: Failed network requests (status >= 400 or connection failures) tracked in DO state</ac>
38+
<ac id="9">Screenshot capture function: captureScreenshot(description: string) method saves screenshots to R2 using existing helper functions</ac>
39+
</acceptanceCriteria>
40+
41+
<artifacts>
42+
<docs>
43+
<doc path="docs/epics/epic-2-ai-test-agent-browser-automation.md" title="Epic 2: AI Test Agent & Browser Automation" section="Story 2.2">
44+
Story 2.2 defines browser integration requirements: Browser Rendering service binding, Stagehand initialization, browser session management, console log capture, network monitoring, and screenshot capture. Browser session persists across phases 1-3 for efficiency.
45+
</doc>
46+
<doc path="docs/epic-2-tech-context.md" title="Epic 2 Technical Specification" section="Services and Modules, Acceptance Criteria">
47+
Technical specification details Browser Rendering integration pattern, Stagehand Computer Use mode configuration, browser session persistence in DO state, and evidence capture requirements. Includes data models for browser session handles, console logs, and network errors.
48+
</doc>
49+
<doc path="docs/architecture/novel-pattern-designs.md" title="Novel Pattern Designs" section="Pattern 1">
50+
Pattern 1 describes TestAgent as Durable Object with browser session persistence. Browser session stored in DO state, reused across phases 1-3 to maintain game state and reduce overhead. Evidence accumulates naturally throughout test execution.
51+
</doc>
52+
<doc path="docs/architecture/technology-stack-details.md" title="Technology Stack Details" section="Browser Automation">
53+
Browser Rendering service provides serverless browser sessions. Viewport 1280x720, console log and network request capture enabled. Stagehand library provides Computer Use mode for autonomous gameplay and observe() for control discovery.
54+
</doc>
55+
<doc path="docs/prd/6-technical-architecture.md" title="Technical Architecture" section="6.5 Stagehand Integration">
56+
Stagehand integration pattern: Browser Rendering service binding launches browser, Stagehand initialized with Computer Use mode. Evidence capture pattern: screenshots stored incrementally to R2, console logs captured continuously, network errors tracked.
57+
</doc>
58+
<doc path="docs/prd/11b-references-resources.md" title="References & Resources" section="Cloudflare Documentation, Stagehand Resources">
59+
Cloudflare Browser Rendering documentation: https://developers.cloudflare.com/browser-rendering/. Stagehand Integration Guide: https://developers.cloudflare.com/browser-rendering/platform/stagehand/. Stagehand Computer Use Guide: https://docs.stagehand.dev/v3/best-practices/computer-use.
60+
</doc>
61+
<doc path="docs/architecture/architecture-decision-records-adrs.md" title="Architecture Decision Records" section="ADR-002">
62+
ADR-002: Single TestAgent Durable Object per test run (DO ID = test UUID). Browser session persists in DO state, survives workflow retries. State management uses built-in Agent SQL database for per-test reasoning.
63+
</doc>
64+
</docs>
65+
<code>
66+
<artifact path="src/agents/TestAgent.ts" kind="durable-object" symbol="TestAgent" lines="1-396" reason="TestAgent class exists with DO foundation. Need to add browser session management methods (launchBrowser, closeBrowser, captureScreenshot), console log listeners, network error tracking, and browser session persistence logic." />
67+
<artifact path="src/shared/helpers/r2.ts" kind="helper" symbol="uploadScreenshot" lines="138-162" reason="uploadScreenshot() function available for screenshot storage. Takes r2 bucket, testId, phase, action, and buffer. Returns DbResult with R2 object key. Use in captureScreenshot() method." />
68+
<artifact path="src/shared/helpers/r2.ts" kind="helper" symbol="uploadLog" lines="191-228" reason="uploadLog() function available for log file storage with append functionality. Uses fetch-modify-put pattern. Set up for console log and network error log flushing in Phase 4." />
69+
<artifact path="src/shared/types.ts" kind="type-definition" symbol="TestAgentState" lines="1-50" reason="TestAgentState interface needs updates: add browserSession property, consoleLogs array, networkErrors array. Add ConsoleLogEntry and NetworkError interfaces." />
70+
<artifact path="wrangler.toml" kind="configuration" symbol="browser" lines="30-32" reason="Browser Rendering binding configured: [browser] binding = 'BROWSER'. Verify binding accessible in TestAgent via env.BROWSER." />
71+
<artifact path="worker-configuration.d.ts" kind="type-definition" symbol="Env" lines="1-100" reason="Worker environment types need BROWSER binding type: BROWSER: BrowserRenderingService. Add BrowserRenderingService type definition if not present." />
72+
<artifact path="package.json" kind="manifest" symbol="dependencies" lines="12-14" reason="Stagehand package already installed: 'stagehand': 'latest'. Import Stagehand in TestAgent.ts: import { Stagehand } from 'stagehand'." />
73+
</code>
74+
<dependencies>
75+
<ecosystem name="node">
76+
<package name="stagehand" version="latest">Browser automation library with Computer Use mode and observe() for control discovery</package>
77+
<package name="@cloudflare/workers-types" version="^4.0.0">TypeScript types for Workers APIs and Browser Rendering service</package>
78+
</ecosystem>
79+
</dependencies>
80+
</artifacts>
81+
82+
<constraints>
83+
<constraint>Browser session must persist in DO state across phases 1-3 to maintain game state and reduce overhead</constraint>
84+
<constraint>Browser session handle stored in DO state, survives workflow retries (Pattern 1 from novel-pattern-designs.md)</constraint>
85+
<constraint>Console logs accumulated in memory during phases 1-3, flushed to R2 at end of Phase 3 (Story 2.5)</constraint>
86+
<constraint>Network errors tracked in DO state, included in Phase 4 evaluation (Story 2.6)</constraint>
87+
<constraint>Browser configuration: headless mode, viewport 1280x720, user agent 'GameEval TestAgent/1.0'</constraint>
88+
<constraint>Stagehand initialized with Computer Use mode: mode: 'computer-use', model: 'gpt-4o' or via AI Gateway (reference Story 2.5)</constraint>
89+
<constraint>R2 screenshot upload uses existing uploadScreenshot() helper from src/shared/helpers/r2.ts</constraint>
90+
<constraint>Error handling: user-friendly error messages, no stack traces exposed</constraint>
91+
<constraint>RPC-only architecture: No exposed HTTP APIs, all communication via service bindings</constraint>
92+
</constraints>
93+
94+
<interfaces>
95+
<interface name="BrowserRenderingService" kind="service-binding" signature="env.BROWSER.launch(options: BrowserLaunchOptions): Promise&lt;Browser&gt;" path="worker-configuration.d.ts">Browser Rendering service binding for launching browser sessions. Configured in wrangler.toml as BROWSER binding.</interface>
96+
<interface name="Stagehand" kind="library-class" signature="new Stagehand(page: Page, options: { mode: 'computer-use', model: string }): Stagehand" path="node_modules/stagehand">Stagehand library class for browser automation. Initialize with browser page and Computer Use mode configuration.</interface>
97+
<interface name="uploadScreenshot" kind="helper-function" signature="uploadScreenshot(r2: R2Bucket, testId: string, phase: Phase, action: string, buffer: ArrayBuffer): Promise&lt;DbResult&lt;string&gt;&gt;" path="src/shared/helpers/r2.ts">R2 screenshot upload helper. Returns R2 object key on success. Use in captureScreenshot() method.</interface>
98+
<interface name="TestAgentState" kind="interface" signature="interface TestAgentState { browserSession?: BrowserSessionHandle; consoleLogs: ConsoleLogEntry[]; networkErrors: NetworkError[]; ... }" path="src/shared/types.ts">TestAgent state interface needs updates: add browserSession, consoleLogs, networkErrors properties for browser session management.</interface>
99+
</interfaces>
100+
101+
<tests>
102+
<standards>Integration tests verify browser session launch, Stagehand initialization, console log capture, network error tracking, and screenshot capture. Use wrangler dev for local testing. Test browser session persistence by running Phase 1, then Phase 2, verify session reused. Test error handling with invalid browser launch options.</standards>
103+
<locations>tests/ directory for integration tests. Manual testing via wrangler dev local environment. Cloudflare dashboard for deployed testing.</locations>
104+
<ideas>
105+
<test ac="1">Verify env.BROWSER accessible in TestAgent constructor. Log binding availability in TestAgent initialization.</test>
106+
<test ac="2">Call launchBrowser(), verify Stagehand instance returned with Computer Use mode configured.</test>
107+
<test ac="3">Call launchBrowser(), verify browser session created with viewport 1280x720, headless mode, user agent 'GameEval TestAgent/1.0'.</test>
108+
<test ac="4">Run Phase 1, then Phase 2, verify browser session reused from DO state (check session handle matches).</test>
109+
<test ac="5">Call closeBrowser(), verify browser session terminated, resources released, browser session cleared from DO state.</test>
110+
<test ac="6">Verify browser configuration: headless=true, viewport={width: 1280, height: 720}, userAgent='GameEval TestAgent/1.0'.</test>
111+
<test ac="7">Navigate to page with console.log(), console.warn(), console.error(), verify logs captured in DO state with timestamps and levels.</test>
112+
<test ac="8">Navigate to page with failed request (404), verify network error tracked in DO state with URL, status code, error message.</test>
113+
<test ac="9">Call captureScreenshot('test-action'), verify screenshot saved to R2 using uploadScreenshot() helper, R2 key returned or public URL generated.</test>
114+
</ideas>
115+
</tests>
116+
</story-context>
117+

0 commit comments

Comments
 (0)