Skip to content

Commit 2719a02

Browse files
ranst91ranst91
authored
Merge pull request #189 from Abubakar-01/dojo-abubakar-e2e
Added the e2e test cases for dojo application
2 parents 5390b98 + 12b75bf commit 2719a02

File tree

89 files changed

+11795
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

89 files changed

+11795
-0
lines changed

‎typescript-sdk/apps/dojo/e2e/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
playwright-report/
2+
test-results/

‎typescript-sdk/apps/dojo/e2e/README.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# CopilotKit Demo Smoke Tests
2+
3+
This repository houses Playwright-based smoke tests that run on a 6-hour schedule to make sure CopilotKit demo apps remain live and functional.
4+
5+
## 🔧 Local development
6+
7+
```bash
8+
# Install deps
9+
npm install
10+
11+
# Install browsers once
12+
npx playwright install --with-deps
13+
14+
# Run the full suite
15+
npm test
16+
```
17+
18+
Playwright HTML reports are saved to `./playwright-report`.
19+
20+
## ➕ Adding a new smoke test
21+
22+
1. Duplicate an existing file in `tests/` or create `tests/<demo>.spec.ts`.
23+
2. Use Playwright's `test` API—keep the test short (<30 s).
24+
3. Commit and push—GitHub Actions will pick it up on the next scheduled run.
25+
26+
## 🚦 CI / CD
27+
28+
- `.github/workflows/scheduled-tests.yml` executes the suite every 6 hours and on manual trigger.
29+
- Failing runs surface in the Actions tab; the HTML report is uploaded as an artifact.
30+
- (Optional) Slack notifications can be wired by adding a step after the tests.
31+
- Slack alert on failure is baked into the workflow. Just add `SLACK_WEBHOOK_URL` (Incoming Webhook) in repo secrets.

‎typescript-sdk/apps/dojo/e2e/VIDEO_SETUP.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
# 📹 S3 Video Upload System
2+
3+
This system automatically uploads videos of failed Playwright tests to S3 and embeds clickable links in Slack notifications.
4+
5+
## ✅ **Setup Complete Checklist**
6+
7+
- [x] AWS infrastructure created (`setup-aws.sh`)
8+
- [x] Dependencies installed (`@aws-sdk/client-s3`, `json2md`)
9+
- [x] S3 video uploader created (`lib/upload-video.ts`)
10+
- [x] Custom reporter created (`reporters/s3-video-reporter.ts`)
11+
- [x] Playwright config updated (video recording enabled)
12+
- [x] Slack layout updated (video links embedded)
13+
- [x] GitHub Actions updated (AWS credentials)
14+
15+
## 🔧 **Required GitHub Secrets**
16+
17+
Add these secrets to your repository:
18+
19+
```
20+
AWS_ACCESS_KEY_ID=AKIA...
21+
AWS_SECRET_ACCESS_KEY=...
22+
AWS_S3_BUCKET_NAME=copilotkit-e2e-smoke-test-recordings-abc123
23+
AWS_S3_REGION=us-east-1
24+
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
25+
```
26+
27+
## 🎯 **How It Works**
28+
29+
### **1. Video Recording**
30+
31+
- Videos recorded only for **failed tests** (`retain-on-failure`)
32+
- 1280x720 resolution, WebM format
33+
- Stored temporarily in `test-results/`
34+
35+
### **2. S3 Upload Process**
36+
37+
```
38+
Failed Test → Video Recorded → S3 Upload → Slack Notification
39+
```
40+
41+
### **3. S3 File Organization**
42+
43+
```
44+
copilotkit-e2e-smoke-test-recordings-{random}/
45+
└── github-runs/
46+
└── {GITHUB_RUN_ID}/
47+
└── cpk-demos-smoke-tests/
48+
└── {SUITE_NAME}/
49+
└── {TEST_NAME}/
50+
└── video.webm
51+
```
52+
53+
### **4. Slack Integration**
54+
55+
Videos appear as clickable links in categorized failure notifications:
56+
57+
```
58+
🤖 AI Response Issues (2 failures)
59+
• Human in the Loop Feature: Chat interaction steps
60+
→ No AI response - Expected: /Travel Guide/i
61+
📹 [Watch Video](https://bucket.s3.amazonaws.com/path/video.webm)
62+
63+
🔧 Action: Check API keys and AI service status
64+
```
65+
66+
## 🛠 **Local Development**
67+
68+
### **Test Video Upload Locally**
69+
70+
```bash
71+
# Set environment variables
72+
export AWS_S3_BUCKET_NAME="your-bucket-name"
73+
export AWS_S3_REGION="us-east-1"
74+
export AWS_ACCESS_KEY_ID="your-key"
75+
export AWS_SECRET_ACCESS_KEY="your-secret"
76+
77+
# Run tests with video upload enabled
78+
CI=true pnpm exec playwright test --reporter=./reporters/s3-video-reporter.ts
79+
```
80+
81+
### **Disable Video Upload Locally**
82+
83+
Videos are automatically disabled in local runs. To force enable:
84+
85+
```bash
86+
# Edit playwright.config.ts
87+
uploadVideos: true // In local reporter config
88+
```
89+
90+
## 📊 **Monitoring & Debugging**
91+
92+
### **Check Upload Status**
93+
94+
- Videos upload logs appear in GitHub Actions output
95+
- Failed uploads are logged but don't fail the workflow
96+
- Video URLs written to `test-results/video-urls.json`
97+
98+
### **Common Issues**
99+
100+
**❌ No videos in Slack**
101+
102+
- Check AWS credentials in GitHub secrets
103+
- Verify S3 bucket permissions
104+
- Look for upload errors in Actions logs
105+
106+
**❌ Videos not accessible**
107+
108+
- Verify S3 bucket has public read access
109+
- Check bucket policy and CORS settings
110+
111+
**❌ Upload timeouts**
112+
113+
- Large video files may timeout
114+
- Check network connectivity to S3
115+
- Consider video compression settings
116+
117+
## 🧹 **Maintenance**
118+
119+
### **Automatic Cleanup**
120+
121+
- Videos automatically deleted after **30 days**
122+
- Lifecycle policy configured in S3 bucket
123+
- No manual cleanup required
124+
125+
### **Cost Management**
126+
127+
- Only failed tests generate videos (~5-10 MB each)
128+
- 30-day retention keeps costs low
129+
- Monitor S3 usage in AWS console
130+
131+
## 🚀 **Next Steps**
132+
133+
1. **Run `setup-aws.sh`** to create infrastructure ✅
134+
2. **Add GitHub secrets** from script output ⏳
135+
3. **Test the system** by running a failing test ⏳
136+
4. **Check Slack notifications** for video links ⏳
137+
138+
## 🔗 **File Structure**
139+
140+
```
141+
cpk-demos-smoke-tests/
142+
├── lib/
143+
│ └── upload-video.ts # S3 upload functionality
144+
├── reporters/
145+
│ └── s3-video-reporter.ts # Playwright reporter
146+
├── .github/workflows/
147+
│ └── scheduled-tests.yml # AWS credentials setup
148+
├── playwright.config.ts # Video recording config
149+
├── slack-layout.ts # Video links in notifications
150+
├── setup-aws.sh # AWS infrastructure script
151+
└── VIDEO_SETUP.md # This file
152+
```
153+
154+
## 📹 **Video URL Format**
155+
156+
```
157+
https://{bucket}.s3.{region}.amazonaws.com/github-runs/{run-id}/{project}/{suite}/{test}/video-{timestamp}.webm
158+
```
159+
160+
Example:
161+
162+
```
163+
https://copilotkit-e2e-recordings.s3.us-east-1.amazonaws.com/github-runs/1234567890/cpk-demos-smoke-tests/Human-in-the-Loop-Feature/Chat-interaction-steps/video-20240115-143022.webm
164+
```
165+
166+
**🎉 Your failed test videos are now automatically uploaded to S3 and linked in Slack!**

‎typescript-sdk/apps/dojo/e2e/clean-reporter.js‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
class CleanReporter {
2+
onBegin(config, suite) {
3+
console.log(`\n🎭 Running ${suite.allTests().length} tests...\n`);
4+
}
5+
6+
onTestEnd(test, result) {
7+
const suiteName = test.parent?.title || "Unknown";
8+
const testName = test.title;
9+
10+
// Clean up suite name
11+
const cleanSuite = suiteName
12+
.replace(/Tests?$/i, "")
13+
.replace(/Page$/i, "")
14+
.replace(/([a-z])([A-Z])/g, "$1 $2")
15+
.trim();
16+
17+
if (result.status === "passed") {
18+
console.log(`✅ ${cleanSuite}: ${testName}`);
19+
} else if (result.status === "failed") {
20+
console.log(`❌ ${cleanSuite}: ${testName}`);
21+
22+
// Extract the most relevant error info
23+
const error = result.error || result.errors?.[0];
24+
if (error) {
25+
let errorMsg = error.message || "Unknown error";
26+
27+
// Clean up common error patterns to make them more readable
28+
if (errorMsg.includes("None of the expected patterns matched")) {
29+
const patterns = errorMsg.match(/patterns matched[^:]*: ([^`]+)/);
30+
errorMsg = `AI response timeout - Expected: ${
31+
patterns?.[1] || "AI response"
32+
}`;
33+
} else if (
34+
errorMsg.includes("Timed out") &&
35+
errorMsg.includes("toBeVisible")
36+
) {
37+
const element = errorMsg.match(/locator\('([^']+)'\)/);
38+
errorMsg = `Element not found: ${element?.[1] || "UI element"}`;
39+
} else if (errorMsg.includes("toBeGreaterThan")) {
40+
errorMsg = "Expected content not generated (count was 0)";
41+
}
42+
43+
// Show just the key error info
44+
console.log(` 💥 ${errorMsg.split("\n")[0]}`);
45+
46+
// If it's an AI/API issue, make it clear
47+
if (
48+
errorMsg.includes("AI") ||
49+
errorMsg.includes("patterns") ||
50+
errorMsg.includes("timeout")
51+
) {
52+
console.log(` 🔑 Likely cause: AI service down or API key issue`);
53+
}
54+
}
55+
console.log(""); // Extra spacing after failures
56+
} else if (result.status === "skipped") {
57+
console.log(`⏭ ${cleanSuite}: ${testName} (skipped)`);
58+
}
59+
}
60+
61+
onEnd(result) {
62+
console.log("\n" + "=".repeat(60));
63+
console.log(`📊 TEST SUMMARY`);
64+
console.log("=".repeat(60));
65+
66+
console.log(`\n🔍 FAILURE ANALYSIS:`);
67+
console.log(`• Most failures appear to be AI service related`);
68+
console.log(`• Check API keys and service availability`);
69+
console.log(
70+
`• Run 'pnpm exec playwright show-report' for detailed HTML report`
71+
);
72+
73+
console.log("=".repeat(60) + "\n");
74+
}
75+
}
76+
77+
module.exports = CleanReporter;

0 commit comments

Comments
 (0)