Skip to content

Commit 9dccb82

Browse files
maxkorpmaxkorp
committed
Merge branch 'main' into mme/dojo-samples
2 parents 711026f + 19334f3 commit 9dccb82

File tree

140 files changed

+14380
-11299
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

140 files changed

+14380
-11299
lines changed

‎.github/workflows/dojo-e2e.yml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: e2e
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
e2e:
11+
name: E2E Tests
12+
runs-on: ubuntu-latest
13+
14+
steps:
15+
- name: Checkout code
16+
uses: actions/checkout@v4
17+
18+
- name: Set up Node.js
19+
uses: actions/setup-node@v4
20+
with:
21+
node-version: '22'
22+
23+
- name: Install pnpm
24+
uses: pnpm/action-setup@v4
25+
with:
26+
version: 10.13.1
27+
28+
- name: Install Poetry
29+
uses: snok/install-poetry@v1
30+
with:
31+
version: latest
32+
virtualenvs-create: true
33+
virtualenvs-in-project: true
34+
35+
- name: Install uv
36+
uses: astral-sh/setup-uv@v6
37+
38+
- name: Setup pnpm cache
39+
uses: actions/cache@v4
40+
with:
41+
path: ~/.local/share/pnpm/store
42+
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
43+
restore-keys: |
44+
${{ runner.os }}-pnpm-store-
45+
46+
- name: Install dependencies
47+
working-directory: typescript-sdk
48+
run: pnpm install --frozen-lockfile
49+
50+
- name: Prepare dojo for e2e
51+
working-directory: typescript-sdk/apps/dojo
52+
run: node ./scripts/prep-dojo-everything.js -e2e
53+
54+
- name: Install e2e dependencies
55+
working-directory: typescript-sdk/apps/dojo/e2e2
56+
run: |
57+
pnpm install --frozen-lockfile
58+
pnpm dlx playwright install --with-deps
59+
60+
- name: write langgraph env files
61+
working-directory: typescript-sdk/integrations/langgraph
62+
env:
63+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
64+
LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
65+
run: |
66+
echo "OPENAI_API_KEY=${OPENAI_API_KEY}" > examples/python/.env
67+
echo "LANGSMITH_API_KEY=${LANGSMITH_API_KEY}" >> examples/python/.env
68+
echo "OPENAI_API_KEY=${OPENAI_API_KEY}" > examples/typescript/.env
69+
echo "LANGSMITH_API_KEY=${LANGSMITH_API_KEY}" >> examples/typescript/.env
70+
echo "OPENAI_API_KEY=${OPENAI_API_KEY}" > python/ag_ui_langgraph/.env
71+
echo "LANGSMITH_API_KEY=${LANGSMITH_API_KEY}" >> python/ag_ui_langgraph/.env
72+
73+
- name: Run dojo+agents and tests
74+
working-directory: typescript-sdk/apps/dojo/e2e2
75+
env:
76+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
77+
LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
78+
run: |
79+
node ../scripts/run-dojo-everything.js &
80+
npx wait-port 9999
81+
sleep 10
82+
pnpm exec playwright test --reporter=dot
83+
84+
- name: Upload traces
85+
if: always() # Uploads artifacts even if tests fail
86+
uses: actions/upload-artifact@v4
87+
with:
88+
name: playwright-traces
89+
path: typescript-sdk/apps/dojo/e2e2/test-results/
90+
retention-days: 7

‎.github/workflows/test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ jobs:
6969
- name: Install pnpm
7070
uses: pnpm/action-setup@v4
7171
with:
72-
version: 9.0.0
72+
version: 10.13.1
7373

7474
- name: Setup pnpm cache
7575
uses: actions/cache@v4

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Built for simplicity and flexibility, it enables seamless integration between AI
66

77
---
88

9-
[📅 Upcoming Event: July 24 - San Francisco AG-UI workshop with AI Tinkerers in person (Feat. CopilotKit & Mastra)](https://lu.ma/ai-tinkerers-workshop?tk=JeUCY5)
9+
[📅 Upcoming Event: August 6th - AG-UI + Mastra: Build a Project Management Canvas](https://lu.ma/94688z7e)
1010

1111
<br>
1212

@@ -29,7 +29,7 @@ Built for simplicity and flexibility, it enables seamless integration between AI
2929

3030
## 🚀 Getting Started
3131
Create a new AG-UI application in seconds:
32-
```ts
32+
```bash
3333
npx create-ag-ui-app my-agent-app
3434
```
3535
<h3>Building AG-UI Integrations (new frameworks):</h3>

‎docs/quickstart/clients.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ AG-UI protocol.
1515

1616
Building your own client is useful if you want to explore/hack on the AG-UI
1717
protocol. For production use, use a full-featured client like
18-
[CopilotKit ](https://copilokit.ai).
18+
[CopilotKit](https://copilotkit.ai).
1919

2020
## What you'll build
2121

‎typescript-sdk/apps/dojo/e2e/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
playwright-report/
2+
test-results/

‎typescript-sdk/apps/dojo/e2e/README.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# CopilotKit Demo Smoke Tests
2+
3+
This repository houses Playwright-based smoke tests that run on a 6-hour schedule to make sure CopilotKit demo apps remain live and functional.
4+
5+
## 🔧 Local development
6+
7+
```bash
8+
# Install deps
9+
npm install
10+
11+
# Install browsers once
12+
npx playwright install --with-deps
13+
14+
# Run the full suite
15+
npm test
16+
```
17+
18+
Playwright HTML reports are saved to `./playwright-report`.
19+
20+
## ➕ Adding a new smoke test
21+
22+
1. Duplicate an existing file in `tests/` or create `tests/<demo>.spec.ts`.
23+
2. Use Playwright's `test` API—keep the test short (<30 s).
24+
3. Commit and push—GitHub Actions will pick it up on the next scheduled run.
25+
26+
## 🚦 CI / CD
27+
28+
- `.github/workflows/scheduled-tests.yml` executes the suite every 6 hours and on manual trigger.
29+
- Failing runs surface in the Actions tab; the HTML report is uploaded as an artifact.
30+
- (Optional) Slack notifications can be wired by adding a step after the tests.
31+
- Slack alert on failure is baked into the workflow. Just add `SLACK_WEBHOOK_URL` (Incoming Webhook) in repo secrets.

‎typescript-sdk/apps/dojo/e2e/VIDEO_SETUP.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
# 📹 S3 Video Upload System
2+
3+
This system automatically uploads videos of failed Playwright tests to S3 and embeds clickable links in Slack notifications.
4+
5+
## ✅ **Setup Complete Checklist**
6+
7+
- [x] AWS infrastructure created (`setup-aws.sh`)
8+
- [x] Dependencies installed (`@aws-sdk/client-s3`, `json2md`)
9+
- [x] S3 video uploader created (`lib/upload-video.ts`)
10+
- [x] Custom reporter created (`reporters/s3-video-reporter.ts`)
11+
- [x] Playwright config updated (video recording enabled)
12+
- [x] Slack layout updated (video links embedded)
13+
- [x] GitHub Actions updated (AWS credentials)
14+
15+
## 🔧 **Required GitHub Secrets**
16+
17+
Add these secrets to your repository:
18+
19+
```
20+
AWS_ACCESS_KEY_ID=AKIA...
21+
AWS_SECRET_ACCESS_KEY=...
22+
AWS_S3_BUCKET_NAME=copilotkit-e2e-smoke-test-recordings-abc123
23+
AWS_S3_REGION=us-east-1
24+
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
25+
```
26+
27+
## 🎯 **How It Works**
28+
29+
### **1. Video Recording**
30+
31+
- Videos recorded only for **failed tests** (`retain-on-failure`)
32+
- 1280x720 resolution, WebM format
33+
- Stored temporarily in `test-results/`
34+
35+
### **2. S3 Upload Process**
36+
37+
```
38+
Failed Test → Video Recorded → S3 Upload → Slack Notification
39+
```
40+
41+
### **3. S3 File Organization**
42+
43+
```
44+
copilotkit-e2e-smoke-test-recordings-{random}/
45+
└── github-runs/
46+
└── {GITHUB_RUN_ID}/
47+
└── cpk-demos-smoke-tests/
48+
└── {SUITE_NAME}/
49+
└── {TEST_NAME}/
50+
└── video.webm
51+
```
52+
53+
### **4. Slack Integration**
54+
55+
Videos appear as clickable links in categorized failure notifications:
56+
57+
```
58+
🤖 AI Response Issues (2 failures)
59+
• Human in the Loop Feature: Chat interaction steps
60+
→ No AI response - Expected: /Travel Guide/i
61+
📹 [Watch Video](https://bucket.s3.amazonaws.com/path/video.webm)
62+
63+
🔧 Action: Check API keys and AI service status
64+
```
65+
66+
## 🛠 **Local Development**
67+
68+
### **Test Video Upload Locally**
69+
70+
```bash
71+
# Set environment variables
72+
export AWS_S3_BUCKET_NAME="your-bucket-name"
73+
export AWS_S3_REGION="us-east-1"
74+
export AWS_ACCESS_KEY_ID="your-key"
75+
export AWS_SECRET_ACCESS_KEY="your-secret"
76+
77+
# Run tests with video upload enabled
78+
CI=true pnpm exec playwright test --reporter=./reporters/s3-video-reporter.ts
79+
```
80+
81+
### **Disable Video Upload Locally**
82+
83+
Videos are automatically disabled in local runs. To force enable:
84+
85+
```bash
86+
# Edit playwright.config.ts
87+
uploadVideos: true // In local reporter config
88+
```
89+
90+
## 📊 **Monitoring & Debugging**
91+
92+
### **Check Upload Status**
93+
94+
- Videos upload logs appear in GitHub Actions output
95+
- Failed uploads are logged but don't fail the workflow
96+
- Video URLs written to `test-results/video-urls.json`
97+
98+
### **Common Issues**
99+
100+
**❌ No videos in Slack**
101+
102+
- Check AWS credentials in GitHub secrets
103+
- Verify S3 bucket permissions
104+
- Look for upload errors in Actions logs
105+
106+
**❌ Videos not accessible**
107+
108+
- Verify S3 bucket has public read access
109+
- Check bucket policy and CORS settings
110+
111+
**❌ Upload timeouts**
112+
113+
- Large video files may timeout
114+
- Check network connectivity to S3
115+
- Consider video compression settings
116+
117+
## 🧹 **Maintenance**
118+
119+
### **Automatic Cleanup**
120+
121+
- Videos automatically deleted after **30 days**
122+
- Lifecycle policy configured in S3 bucket
123+
- No manual cleanup required
124+
125+
### **Cost Management**
126+
127+
- Only failed tests generate videos (~5-10 MB each)
128+
- 30-day retention keeps costs low
129+
- Monitor S3 usage in AWS console
130+
131+
## 🚀 **Next Steps**
132+
133+
1. **Run `setup-aws.sh`** to create infrastructure ✅
134+
2. **Add GitHub secrets** from script output ⏳
135+
3. **Test the system** by running a failing test ⏳
136+
4. **Check Slack notifications** for video links ⏳
137+
138+
## 🔗 **File Structure**
139+
140+
```
141+
cpk-demos-smoke-tests/
142+
├── lib/
143+
│ └── upload-video.ts # S3 upload functionality
144+
├── reporters/
145+
│ └── s3-video-reporter.ts # Playwright reporter
146+
├── .github/workflows/
147+
│ └── scheduled-tests.yml # AWS credentials setup
148+
├── playwright.config.ts # Video recording config
149+
├── slack-layout.ts # Video links in notifications
150+
├── setup-aws.sh # AWS infrastructure script
151+
└── VIDEO_SETUP.md # This file
152+
```
153+
154+
## 📹 **Video URL Format**
155+
156+
```
157+
https://{bucket}.s3.{region}.amazonaws.com/github-runs/{run-id}/{project}/{suite}/{test}/video-{timestamp}.webm
158+
```
159+
160+
Example:
161+
162+
```
163+
https://copilotkit-e2e-recordings.s3.us-east-1.amazonaws.com/github-runs/1234567890/cpk-demos-smoke-tests/Human-in-the-Loop-Feature/Chat-interaction-steps/video-20240115-143022.webm
164+
```
165+
166+
**🎉 Your failed test videos are now automatically uploaded to S3 and linked in Slack!**

0 commit comments

Comments
 (0)