Skip to content

Commit 467a3c7

Browse files
XunzhuoXunzhuo
committed
CI
Signed-off-by: bitliu <[email protected]>
1 parent 5071db1 commit 467a3c7

File tree

17 files changed

+1993
-1
lines changed

17 files changed

+1993
-1
lines changed

‎.github/workflows/integration-test-ai-gateway.yml‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
name: Integration Test [AI Gateway]
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
push:
8+
branches:
9+
- main
10+
workflow_dispatch: # Allow manual triggering
11+
12+
jobs:
13+
test-ai-gateway:
14+
runs-on: ubuntu-latest
15+
timeout-minutes: 60
16+
17+
steps:
18+
- name: Check out the repo
19+
uses: actions/checkout@v4
20+
21+
- name: Set up Go
22+
uses: actions/setup-go@v5
23+
with:
24+
go-version: '1.24'
25+
26+
- name: Set up Rust
27+
uses: actions-rust-lang/setup-rust-toolchain@v1
28+
with:
29+
toolchain: 1.90
30+
31+
- name: Install system dependencies
32+
run: |
33+
sudo apt-get update
34+
sudo apt-get install -y \
35+
make \
36+
curl \
37+
build-essential \
38+
pkg-config
39+
40+
- name: Install Kind
41+
run: |
42+
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
43+
chmod +x ./kind
44+
sudo mv ./kind /usr/local/bin/kind
45+
46+
- name: Install kubectl
47+
run: |
48+
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
49+
chmod +x kubectl
50+
sudo mv kubectl /usr/local/bin/kubectl
51+
52+
- name: Install Helm
53+
run: |
54+
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
55+
56+
- name: Download E2E test dependencies
57+
run: |
58+
cd e2e && go mod download
59+
60+
- name: Build E2E test binary
61+
run: |
62+
make build-e2e
63+
64+
- name: Run AI Gateway E2E tests
65+
id: e2e-test
66+
run: |
67+
make e2e-test PROFILE=ai-gateway
68+
env:
69+
E2E_VERBOSE: "true"
70+
71+
- name: Show cluster logs on failure
72+
if: failure()
73+
run: |
74+
echo "=== Kind Cluster Info ==="
75+
kind get clusters || true
76+
kubectl cluster-info --context kind-semantic-router-e2e || true
77+
78+
echo "=== All Pods ==="
79+
kubectl get pods --all-namespaces -o wide || true
80+
81+
echo "=== Semantic Router Logs ==="
82+
kubectl logs -n vllm-semantic-router-system deployment/semantic-router --tail=100 || true
83+
84+
echo "=== Envoy Gateway Logs ==="
85+
kubectl logs -n envoy-gateway-system deployment/envoy-gateway --tail=100 || true
86+
87+
echo "=== AI Gateway Controller Logs ==="
88+
kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller --tail=100 || true
89+
90+
echo "=== Gateway Resources ==="
91+
kubectl get gateway -A || true
92+
kubectl get httproute -A || true
93+
94+
echo "=== Events ==="
95+
kubectl get events --all-namespaces --sort-by='.lastTimestamp' || true
96+
97+
- name: Clean up
98+
if: always()
99+
run: |
100+
make e2e-cleanup || true
101+

‎.github/workflows/integration-test-helm.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Helm Chart CI
1+
name: Integration Test [Helm]
22

33
on:
44
push:

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ _run:
1919
-f tools/make/helm.mk \
2020
-f tools/make/observability.mk \
2121
-f tools/make/openshift.mk \
22+
-f tools/make/e2e.mk \
2223
$(MAKECMDGOALS)
2324

2425
.PHONY: _run

‎e2e/.gitignore‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Binaries
2+
bin/
3+
*.exe
4+
*.dll
5+
*.so
6+
*.dylib
7+
8+
# Test binary
9+
e2e.test
10+
*.test
11+
12+
# Output
13+
*.out
14+
15+
# Go workspace file
16+
go.work
17+

‎e2e/README.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
# E2E Test Framework
2+
3+
A comprehensive end-to-end testing framework for Semantic Router with support for multiple deployment profiles.
4+
5+
## Architecture
6+
7+
The framework is designed to be extensible and supports multiple test profiles:
8+
9+
- **ai-gateway**: Tests Semantic Router with Envoy AI Gateway integration
10+
- **istio**: Tests Semantic Router with Istio Gateway (future)
11+
- **production-stack**: Tests vLLM Production Stack configurations (future)
12+
- **llm-d**: Tests with LLM-D (future)
13+
- **dynamo**: Tests with Nvidia Dynamo (future)
14+
- **aibrix**: Tests with vLLM AIBrix (future)
15+
16+
## Directory Structure
17+
18+
```
19+
e2e/
20+
├── cmd/
21+
│ └── e2e/ # Main test runner
22+
├── pkg/
23+
│ ├── framework/ # Core test framework
24+
│ ├── cluster/ # Kind cluster management
25+
│ ├── docker/ # Docker image operations
26+
│ ├── helm/ # Helm deployment utilities
27+
│ └── testcases/ # Test case definitions
28+
├── profiles/
29+
│ ├── ai-gateway/ # AI Gateway test profile
30+
│ ├── istio/ # Istio test profile (future)
31+
│ └── ...
32+
└── README.md
33+
```
34+
35+
## Quick Start
36+
37+
### Run all tests with default profile (ai-gateway)
38+
39+
```bash
40+
make e2e-test
41+
```
42+
43+
### Run specific profile
44+
45+
```bash
46+
make e2e-test PROFILE=ai-gateway
47+
```
48+
49+
### Run with custom options
50+
51+
```bash
52+
# Keep cluster after test
53+
make e2e-test KEEP_CLUSTER=true
54+
55+
# Use existing cluster
56+
make e2e-test USE_EXISTING_CLUSTER=true
57+
58+
# Verbose output
59+
make e2e-test VERBOSE=true
60+
```
61+
62+
## Adding New Test Profiles
63+
64+
1. Create a new directory under `profiles/`
65+
2. Implement the `Profile` interface
66+
3. Register test cases using the test case registry
67+
4. Add profile-specific deployment configurations
68+
69+
See `profiles/ai-gateway/` for a complete example.
70+
71+
## Test Case Registration
72+
73+
Test cases are registered using a simple function-based approach:
74+
75+
```go
76+
func init() {
77+
testcases.Register("my-test", testcases.TestCase{
78+
Name: "My Test",
79+
Description: "Description of what this test does",
80+
Fn: func(ctx context.Context, client *kubernetes.Clientset) error {
81+
// Test implementation
82+
return nil
83+
},
84+
})
85+
}
86+
```
87+
88+
## Framework Features
89+
90+
- **Automatic cluster lifecycle management**: Creates and cleans up Kind clusters
91+
- **Docker image building and loading**: Builds images and loads them into Kind
92+
- **Helm deployment automation**: Deploys required Helm charts
93+
- **Parallel test execution**: Runs independent tests in parallel
94+
- **Detailed logging**: Provides comprehensive test output
95+
- **Resource cleanup**: Ensures proper cleanup even on failures
96+
97+
## Prerequisites
98+
99+
Before running E2E tests, ensure you have the following tools installed:
100+
101+
- [Go](https://golang.org/doc/install) 1.24 or later
102+
- [Docker](https://docs.docker.com/get-docker/)
103+
- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)
104+
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
105+
- [Helm](https://helm.sh/docs/intro/install/)
106+
107+
## Getting Started
108+
109+
### 1. Install dependencies
110+
111+
```bash
112+
make e2e-deps
113+
```
114+
115+
### 2. Build the E2E test binary
116+
117+
```bash
118+
make build-e2e
119+
```
120+
121+
### 3. Run tests
122+
123+
```bash
124+
# Run all tests with default profile (ai-gateway)
125+
make e2e-test
126+
127+
# Run with verbose output
128+
make e2e-test E2E_VERBOSE=true
129+
130+
# Run and keep cluster for debugging
131+
make e2e-test-debug
132+
133+
# Run specific test cases
134+
make e2e-test-specific E2E_TESTS="basic-health-check,chat-completions-request"
135+
```
136+
137+
## CI Integration
138+
139+
The E2E tests are automatically run in GitHub Actions on:
140+
141+
- Pull requests to `main` branch
142+
- Pushes to `main` branch
143+
144+
See `.github/workflows/integration-test-ai-gateway.yml` for the CI configuration.
145+
146+
## Troubleshooting
147+
148+
### Cluster creation fails
149+
150+
```bash
151+
# Clean up any existing cluster
152+
make e2e-cleanup
153+
154+
# Try again
155+
make e2e-test
156+
```
157+
158+
### Tests fail with timeout
159+
160+
Increase the timeout in the test case or check if the cluster has enough resources:
161+
162+
```bash
163+
# Check cluster status
164+
kubectl get nodes
165+
kubectl get pods --all-namespaces
166+
```
167+
168+
### Port forward fails
169+
170+
Make sure no other process is using port 8080:
171+
172+
```bash
173+
# Check what's using port 8080
174+
lsof -i :8080
175+
176+
# Kill the process if needed
177+
kill -9 <PID>
178+
```
179+
180+
## Development
181+
182+
### Adding a new test case
183+
184+
1. Create a new test function in `profiles/<profile>/testcases.go`
185+
2. Register it in the `init()` function
186+
3. Add the test case name to the profile's `GetTestCases()` method
187+
188+
Example:
189+
190+
```go
191+
func init() {
192+
testcases.Register("my-new-test", testcases.TestCase{
193+
Description: "My new test description",
194+
Tags: []string{"ai-gateway", "functional"},
195+
Fn: testMyNewFeature,
196+
})
197+
}
198+
199+
func testMyNewFeature(ctx context.Context, client *kubernetes.Clientset, opts testcases.TestCaseOptions) error {
200+
// Test implementation
201+
return nil
202+
}
203+
```
204+
205+
### Adding a new profile
206+
207+
1. Create a new directory under `profiles/`
208+
2. Implement the `Profile` interface
209+
3. Register test cases
210+
4. Update `cmd/e2e/main.go` to include the new profile
211+
212+
See `profiles/ai-gateway/` for a complete example.

0 commit comments

Comments
 (0)