Skip to content

Commit 1cea41d

Browse files
authored
Merge pull request #943 from RooVetGit/feature/gradingTestResponse
Feature/grading vscode test response
2 parents b933ce4 + 4886cb7 commit 1cea41d

File tree

3 files changed

+206
-45
lines changed

3 files changed

+206
-45
lines changed
Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# VSCode Integration Tests
2+
3+
This document describes the integration test setup for the Roo Code VSCode extension.
4+
5+
## Overview
6+
7+
The integration tests use the `@vscode/test-electron` package to run tests in a real VSCode environment. These tests verify that the extension works correctly within VSCode, including features like mode switching, webview interactions, and API communication.
8+
9+
## Test Setup
10+
11+
### Directory Structure
12+
13+
```
14+
src/test/
15+
├── runTest.ts # Main test runner
16+
├── suite/
17+
│ ├── index.ts # Test suite configuration
18+
│ ├── modes.test.ts # Mode switching tests
19+
│ ├── tasks.test.ts # Task execution tests
20+
│ └── extension.test.ts # Extension activation tests
21+
```
22+
23+
### Test Runner Configuration
24+
25+
The test runner (`runTest.ts`) is responsible for:
26+
27+
- Setting up the extension development path
28+
- Configuring the test environment
29+
- Running the integration tests using `@vscode/test-electron`
30+
31+
### Environment Setup
32+
33+
1. Create a `.env.integration` file in the root directory with required environment variables:
34+
35+
```
36+
OPENROUTER_API_KEY=sk-or-v1-...
37+
```
38+
39+
2. The test suite (`suite/index.ts`) configures:
40+
41+
- Mocha test framework with TDD interface
42+
- 10-minute timeout for LLM communication
43+
- Global extension API access
44+
- WebView panel setup
45+
- OpenRouter API configuration
46+
47+
## Test Suite Structure
48+
49+
Tests are organized using Mocha's TDD interface (`suite` and `test` functions). The main test files are:
50+
51+
- `modes.test.ts`: Tests mode switching functionality
52+
- `tasks.test.ts`: Tests task execution
53+
- `extension.test.ts`: Tests extension activation
54+
55+
### Global Objects
56+
57+
The following global objects are available in tests:
58+
59+
```typescript
60+
declare global {
61+
var api: ClineAPI
62+
var provider: ClineProvider
63+
var extension: vscode.Extension<ClineAPI>
64+
var panel: vscode.WebviewPanel
65+
}
66+
```
67+
68+
## Running Tests
69+
70+
1. Ensure you have the required environment variables set in `.env.integration`
71+
72+
2. Run the integration tests:
73+
74+
```bash
75+
npm run test:integration
76+
```
77+
78+
3. If you want to run a specific test, you can use the `test.only` function in the test file. This will run only the test you specify and ignore the others. Be sure to remove the `test.only` function before committing your changes.
79+
80+
The tests will:
81+
82+
- Download and launch a clean VSCode instance
83+
- Install the extension
84+
- Execute the test suite
85+
- Report results
86+
87+
## Writing New Tests
88+
89+
When writing new integration tests:
90+
91+
1. Create a new test file in `src/test/suite/` with the `.test.ts` extension
92+
93+
2. Structure your tests using the TDD interface:
94+
95+
```typescript
96+
import * as assert from "assert"
97+
import * as vscode from "vscode"
98+
99+
suite("Your Test Suite Name", () => {
100+
test("Should do something specific", async function () {
101+
// Your test code here
102+
})
103+
})
104+
```
105+
106+
3. Use the global objects (`api`, `provider`, `extension`, `panel`) to interact with the extension
107+
108+
### Best Practices
109+
110+
1. **Timeouts**: Use appropriate timeouts for async operations:
111+
112+
```typescript
113+
const timeout = 30000
114+
const interval = 1000
115+
```
116+
117+
2. **State Management**: Reset extension state before/after tests:
118+
119+
```typescript
120+
await globalThis.provider.updateGlobalState("mode", "Ask")
121+
await globalThis.provider.updateGlobalState("alwaysAllowModeSwitch", true)
122+
```
123+
124+
3. **Assertions**: Use clear assertions with meaningful messages:
125+
126+
```typescript
127+
assert.ok(condition, "Descriptive message about what failed")
128+
```
129+
130+
4. **Error Handling**: Wrap test code in try/catch blocks and clean up resources:
131+
132+
```typescript
133+
try {
134+
// Test code
135+
} finally {
136+
// Cleanup code
137+
}
138+
```
139+
140+
5. **Wait for Operations**: Use polling when waiting for async operations:
141+
142+
```typescript
143+
let startTime = Date.now()
144+
while (Date.now() - startTime < timeout) {
145+
if (condition) break
146+
await new Promise((resolve) => setTimeout(resolve, interval))
147+
}
148+
```
149+
150+
6. **Grading**: When grading tests, use the `Grade:` format to ensure the test is graded correctly (See modes.test.ts for an example).
151+
152+
```typescript
153+
await globalThis.api.startNewTask(
154+
`Given this prompt: ${testPrompt} grade the response from 1 to 10 in the format of "Grade: (1-10)": ${output} \n Be sure to say 'I AM DONE GRADING' after the task is complete`,
155+
)
156+
```

src/test/suite/modes.test.ts

Lines changed: 44 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@ suite("Roo Code Modes", () => {
55
test("Should handle switching modes correctly", async function () {
66
const timeout = 30000
77
const interval = 1000
8-
8+
const testPrompt =
9+
"For each mode (Code, Architect, Ask) respond with the mode name and what it specializes in after switching to that mode, do not start with the current mode, be sure to say 'I AM DONE' after the task is complete"
910
if (!globalThis.extension) {
1011
assert.fail("Extension not found")
1112
}
@@ -27,9 +28,7 @@ suite("Roo Code Modes", () => {
2728
await globalThis.provider.updateGlobalState("autoApprovalEnabled", true)
2829

2930
// Start a new task.
30-
await globalThis.api.startNewTask(
31-
"For each mode (Code, Architect, Ask) respond with the mode name and what it specializes in after switching to that mode, do not start with the current mode, be sure to say 'I AM DONE' after the task is complete",
32-
)
31+
await globalThis.api.startNewTask(testPrompt)
3332

3433
// Wait for task to appear in history with tokens.
3534
startTime = Date.now()
@@ -52,47 +51,50 @@ suite("Roo Code Modes", () => {
5251
assert.fail("No messages received")
5352
}
5453

55-
assert.ok(
56-
globalThis.provider.messages.some(
57-
({ type, text }) => type === "say" && text?.includes(`"request":"[switch_mode to 'code' because:`),
58-
),
59-
"Did not receive expected response containing 'Roo wants to switch to code mode'",
60-
)
61-
assert.ok(
62-
globalThis.provider.messages.some(
63-
({ type, text }) => type === "say" && text?.includes("software engineer"),
64-
),
65-
"Did not receive expected response containing 'I am Roo in Code mode, specializing in software engineering'",
66-
)
54+
//Log the messages to the console
55+
globalThis.provider.messages.forEach(({ type, text }) => {
56+
if (type === "say") {
57+
console.log(text)
58+
}
59+
})
6760

68-
assert.ok(
69-
globalThis.provider.messages.some(
70-
({ type, text }) =>
71-
type === "say" && text?.includes(`"request":"[switch_mode to 'architect' because:`),
72-
),
73-
"Did not receive expected response containing 'Roo wants to switch to architect mode'",
74-
)
75-
assert.ok(
76-
globalThis.provider.messages.some(
77-
({ type, text }) =>
78-
type === "say" && (text?.includes("technical planning") || text?.includes("technical leader")),
79-
),
80-
"Did not receive expected response containing 'I am Roo in Architect mode, specializing in analyzing codebases'",
61+
//Start Grading Portion of test to grade the response from 1 to 10
62+
await globalThis.provider.updateGlobalState("mode", "Ask")
63+
let output = globalThis.provider.messages.map(({ type, text }) => (type === "say" ? text : "")).join("\n")
64+
await globalThis.api.startNewTask(
65+
`Given this prompt: ${testPrompt} grade the response from 1 to 10 in the format of "Grade: (1-10)": ${output} \n Be sure to say 'I AM DONE GRADING' after the task is complete`,
8166
)
8267

83-
assert.ok(
84-
globalThis.provider.messages.some(
85-
({ type, text }) => type === "say" && text?.includes(`"request":"[switch_mode to 'ask' because:`),
86-
),
87-
"Did not receive expected response containing 'Roo wants to switch to ask mode'",
88-
)
89-
assert.ok(
90-
globalThis.provider.messages.some(
91-
({ type, text }) =>
92-
type === "say" && (text?.includes("technical knowledge") || text?.includes("technical assist")),
93-
),
94-
"Did not receive expected response containing 'I am Roo in Ask mode, specializing in answering questions'",
95-
)
68+
startTime = Date.now()
69+
70+
while (Date.now() - startTime < timeout) {
71+
const messages = globalThis.provider.messages
72+
73+
if (
74+
messages.some(
75+
({ type, text }) =>
76+
type === "say" && text?.includes("I AM DONE GRADING") && !text?.includes("be sure to say"),
77+
)
78+
) {
79+
break
80+
}
81+
82+
await new Promise((resolve) => setTimeout(resolve, interval))
83+
}
84+
if (globalThis.provider.messages.length === 0) {
85+
assert.fail("No messages received")
86+
}
87+
globalThis.provider.messages.forEach(({ type, text }) => {
88+
if (type === "say" && text?.includes("Grade:")) {
89+
console.log(text)
90+
}
91+
})
92+
const gradeMessage = globalThis.provider.messages.find(
93+
({ type, text }) => type === "say" && !text?.includes("Grade: (1-10)") && text?.includes("Grade:"),
94+
)?.text
95+
const gradeMatch = gradeMessage?.match(/Grade: (\d+)/)
96+
const gradeNum = gradeMatch ? parseInt(gradeMatch[1]) : undefined
97+
assert.ok(gradeNum !== undefined && gradeNum >= 7 && gradeNum <= 10, "Grade must be between 7 and 10")
9698
} finally {
9799
}
98100
})

src/test/suite/task.test.ts

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,16 +22,19 @@ suite("Roo Code Task", () => {
2222
await new Promise((resolve) => setTimeout(resolve, interval))
2323
}
2424

25+
await globalThis.provider.updateGlobalState("mode", "Code")
26+
await globalThis.provider.updateGlobalState("alwaysAllowModeSwitch", true)
27+
await globalThis.provider.updateGlobalState("autoApprovalEnabled", true)
28+
2529
await globalThis.api.startNewTask("Hello world, what is your name? Respond with 'My name is ...'")
2630

2731
// Wait for task to appear in history with tokens.
2832
startTime = Date.now()
2933

3034
while (Date.now() - startTime < timeout) {
31-
const state = await globalThis.provider.getState()
32-
const task = state.taskHistory?.[0]
35+
const messages = globalThis.provider.messages
3336

34-
if (task && task.tokensOut > 0) {
37+
if (messages.some(({ type, text }) => type === "say" && text?.includes("My name is Roo"))) {
3538
break
3639
}
3740

0 commit comments

Comments
 (0)