Skip to content

Commit febd317

Browse files
committed
Update qa.md with new changes
> Add new oagi sample app > Update the paths > Update the deployed app names referenced > Add a step 7 that will automatically try to invoke the deployed apps to automate testing for changes introduced.
1 parent 7acb017 commit febd317

File tree

1 file changed

+84
-34
lines changed

1 file changed

+84
-34
lines changed

.cursor/commands/qa.md

Lines changed: 84 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -48,20 +48,21 @@ Use the built CLI binary with non-interactive flags. The command format is:
4848

4949
Here are all valid language + template combinations:
5050

51-
| Language | Template | Folder Name | Needs Env File | Required Env Vars |
52-
| ---------- | -------------- | ----------------- | -------------- | ------------------------------ |
53-
| typescript | sample-app | ts-sample-app | No | - |
54-
| typescript | captcha-solver | ts-captcha-solver | No | - |
55-
| typescript | stagehand | ts-stagehand | Yes | OPENAI_API_KEY |
56-
| typescript | computer-use | ts-computer-use | Yes | ANTHROPIC_API_KEY |
57-
| typescript | magnitude | ts-magnitude | Yes | ANTHROPIC_API_KEY |
58-
| typescript | cua | ts-cua | Yes | OPENAI_API_KEY |
59-
| typescript | gemini-cua | ts-gemini-cua | Yes | GOOGLE_API_KEY, OPENAI_API_KEY |
60-
| python | sample-app | py-sample-app | No | - |
61-
| python | captcha-solver | py-captcha-solver | No | - |
62-
| python | browser-use | py-browser-use | Yes | OPENAI_API_KEY |
63-
| python | computer-use | py-computer-use | Yes | ANTHROPIC_API_KEY |
64-
| python | cua | py-cua | Yes | OPENAI_API_KEY |
51+
| Language | Template | Folder Name | Deployed App Name | Needs Env File | Required Env Vars |
52+
| ---------- | ---------------------- | ----------------- | --------------------- | -------------- | ------------------------------ |
53+
| typescript | sample-app | ts-sample-app | ts-basic | No | - |
54+
| typescript | captcha-solver | ts-captcha-solver | ts-captcha-solver | No | - |
55+
| typescript | stagehand | ts-stagehand | ts-stagehand | Yes | OPENAI_API_KEY |
56+
| typescript | anthropic-computer-use | ts-anthropic-cua | ts-anthropic-cua | Yes | ANTHROPIC_API_KEY |
57+
| typescript | magnitude | ts-magnitude | ts-magnitude | Yes | ANTHROPIC_API_KEY |
58+
| typescript | openai-computer-use | ts-openai-cua | ts-openai-cua | Yes | OPENAI_API_KEY |
59+
| typescript | gemini-computer-use | ts-gemini-cua | ts-gemini-cua | Yes | GOOGLE_API_KEY, OPENAI_API_KEY |
60+
| python | sample-app | py-sample-app | python-basic | No | - |
61+
| python | captcha-solver | py-captcha-solver | python-captcha-solver | No | - |
62+
| python | browser-use | py-browser-use | python-bu | Yes | OPENAI_API_KEY |
63+
| python | anthropic-computer-use | py-anthropic-cua | python-anthropic-cua | Yes | ANTHROPIC_API_KEY |
64+
| python | openai-computer-use | py-openai-cua | python-openai-cua | Yes | OPENAI_API_KEY |
65+
| python | openagi-computer-use | py-openagi-cua | python-openagi-cua | Yes | OAGI_API_KEY |
6566

6667
### Create Commands
6768

@@ -72,17 +73,18 @@ Run each of these (they are non-interactive when all flags are provided):
7273
../bin/kernel create -n ts-sample-app -l typescript -t sample-app
7374
../bin/kernel create -n ts-captcha-solver -l typescript -t captcha-solver
7475
../bin/kernel create -n ts-stagehand -l typescript -t stagehand
75-
../bin/kernel create -n ts-computer-use -l typescript -t computer-use
76+
../bin/kernel create -n ts-anthropic-cua -l typescript -t anthropic-computer-use
7677
../bin/kernel create -n ts-magnitude -l typescript -t magnitude
77-
../bin/kernel create -n ts-cua -l typescript -t cua
78-
../bin/kernel create -n ts-gemini-cua -l typescript -t gemini-cua
78+
../bin/kernel create -n ts-openai-cua -l typescript -t openai-computer-use
79+
../bin/kernel create -n ts-gemini-cua -l typescript -t gemini-computer-use
7980

8081
# Python templates
8182
../bin/kernel create -n py-sample-app -l python -t sample-app
8283
../bin/kernel create -n py-captcha-solver -l python -t captcha-solver
8384
../bin/kernel create -n py-browser-use -l python -t browser-use
84-
../bin/kernel create -n py-computer-use -l python -t computer-use
85-
../bin/kernel create -n py-cua -l python -t cua
85+
../bin/kernel create -n py-anthropic-cua -l python -t anthropic-computer-use
86+
../bin/kernel create -n py-openai-cua -l python -t openai-computer-use
87+
../bin/kernel create -n py-openagi-cua -l python -t openagi-computer-use
8688
```
8789

8890
## Step 5: Deploy Each Template
@@ -125,10 +127,10 @@ echo "OPENAI_API_KEY=<value from human>" > .env
125127
cd ..
126128
```
127129

128-
**ts-computer-use** (needs ANTHROPIC_API_KEY):
130+
**ts-anthropic-cua** (needs ANTHROPIC_API_KEY):
129131

130132
```bash
131-
cd ts-computer-use
133+
cd ts-anthropic-cua
132134
echo "ANTHROPIC_API_KEY=<value from human>" > .env
133135
../bin/kernel deploy index.ts --env-file .env
134136
cd ..
@@ -143,10 +145,10 @@ echo "ANTHROPIC_API_KEY=<value from human>" > .env
143145
cd ..
144146
```
145147

146-
**ts-cua** (needs OPENAI_API_KEY):
148+
**ts-openai-cua** (needs OPENAI_API_KEY):
147149

148150
```bash
149-
cd ts-cua
151+
cd ts-openai-cua
150152
echo "OPENAI_API_KEY=<value from human>" > .env
151153
../bin/kernel deploy index.ts --env-file .env
152154
cd ..
@@ -173,24 +175,33 @@ echo "OPENAI_API_KEY=<value from human>" > .env
173175
cd ..
174176
```
175177

176-
**py-computer-use** (needs ANTHROPIC_API_KEY):
178+
**py-anthropic-cua** (needs ANTHROPIC_API_KEY):
177179

178180
```bash
179-
cd py-computer-use
181+
cd py-anthropic-cua
180182
echo "ANTHROPIC_API_KEY=<value from human>" > .env
181183
../bin/kernel deploy main.py --env-file .env
182184
cd ..
183185
```
184186

185-
**py-cua** (needs OPENAI_API_KEY):
187+
**py-openai-cua** (needs OPENAI_API_KEY):
186188

187189
```bash
188-
cd py-cua
190+
cd py-openai-cua
189191
echo "OPENAI_API_KEY=<value from human>" > .env
190192
../bin/kernel deploy main.py --env-file .env
191193
cd ..
192194
```
193195

196+
**py-openagi-cua** (needs OAGI_API_KEY):
197+
198+
```bash
199+
cd py-openagi-cua
200+
echo "OAGI_API_KEY=<value from human>" > .env
201+
../bin/kernel deploy main.py --env-file .env
202+
cd ..
203+
```
204+
194205
## Step 6: Provide Invoke Commands
195206

196207
Once all deployments are complete, present the human with these invoke commands to test manually:
@@ -200,28 +211,67 @@ Once all deployments are complete, present the human with these invoke commands
200211
kernel invoke ts-basic get-page-title --payload '{"url": "https://www.google.com"}'
201212
kernel invoke ts-captcha-solver test-captcha-solver
202213
kernel invoke ts-stagehand teamsize-task --payload '{"company": "Kernel"}'
203-
kernel invoke ts-cu cu-task --payload '{"query": "Return the first url of a search result for NYC restaurant reviews Pete Wells"}'
214+
kernel invoke ts-anthropic-cua cua-task --payload '{"query": "Return the first url of a search result for NYC restaurant reviews Pete Wells"}'
204215
kernel invoke ts-magnitude mag-url-extract --payload '{"url": "https://en.wikipedia.org/wiki/Special:Random"}'
205-
kernel invoke ts-cua cua-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 5 articles"}'
216+
kernel invoke ts-openai-cua cua-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 5 articles"}'
206217
kernel invoke ts-gemini-cua gemini-cua-task
207218

208219
# Python apps
209220
kernel invoke python-basic get-page-title --payload '{"url": "https://www.google.com"}'
210221
kernel invoke python-captcha-solver test-captcha-solver
211222
kernel invoke python-bu bu-task --payload '{"task": "Compare the price of gpt-4o and DeepSeek-V3"}'
212-
kernel invoke python-cu cu-task --payload '{"query": "Return the first url of a search result for NYC restaurant reviews Pete Wells"}'
213-
kernel invoke python-cua cua-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 5 articles"}'
223+
kernel invoke python-anthropic-cua cua-task --payload '{"query": "Return the first url of a search result for NYC restaurant reviews Pete Wells"}'
224+
kernel invoke python-openai-cua cua-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 5 articles"}'
225+
kernel invoke python-openagi-cua openagi-default-task -p '{"instruction": "Navigate to https://agiopen.org and click the What is Computer Use? button"}'
214226
```
215227

228+
## Step 7: Automated Runtime Testing (Optional)
229+
230+
**STOP and ask the human:** "Would you like me to automatically invoke all 13 templates and report back on their runtime status?"
231+
232+
If the human agrees, invoke each template and collect results. Present findings in this format:
233+
234+
### Testing Guidelines
235+
236+
- **Timeout:** Cancel each invocation after 90 seconds if it has not completed. Mark the status as `TIMEOUT` in the results table.
237+
- **Parallel execution:** You may run multiple invocations in parallel to speed up testing.
238+
- **Error handling:** Capture any runtime errors and include them in the Notes column.
239+
240+
### Test Results
241+
242+
| Template | App Name | Status | Notes |
243+
| ----------------- | --------------------- | ------- | ----- |
244+
| ts-sample-app | ts-basic | | |
245+
| ts-captcha-solver | ts-captcha-solver | | |
246+
| ts-stagehand | ts-stagehand | | |
247+
| ts-anthropic-cua | ts-anthropic-cua | | |
248+
| ts-magnitude | ts-magnitude | | |
249+
| ts-openai-cua | ts-openai-cua | | |
250+
| ts-gemini-cua | ts-gemini-cua | | |
251+
| py-sample-app | python-basic | | |
252+
| py-captcha-solver | python-captcha-solver | | |
253+
| py-browser-use | python-bu | | |
254+
| py-anthropic-cua | python-anthropic-cua | | |
255+
| py-openai-cua | python-openai-cua | | |
256+
| py-openagi-cua | python-openagi-cua | | |
257+
258+
Status values:
259+
- **SUCCESS**: App started and returned a result
260+
- **FAILED**: App encountered a runtime error
261+
- **TIMEOUT**: App did not complete within 90 seconds (cancelled)
262+
263+
Notes should include brief error messages for failures or confirmation of successful output.
264+
216265
## Summary Checklist
217266

218267
- [ ] Built CLI with `make build`
219268
- [ ] Created QA directory
220269
- [ ] Got KERNEL_API_KEY from human
221-
- [ ] Created all 12 template variations
222-
- [ ] Got required API keys from human (OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY)
223-
- [ ] Deployed all 12 apps
270+
- [ ] Created all 13 template variations
271+
- [ ] Got required API keys from human (OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, OAGI_API_KEY)
272+
- [ ] Deployed all 13 apps
224273
- [ ] Provided invoke commands to human for manual testing
274+
- [ ] (Optional) Ran automated runtime testing and reviewed results
225275

226276
## Cleanup
227277

0 commit comments

Comments
 (0)