Skip to content

Commit fb52631

Browse files
XunzhuoXunzhuo
authored
[Feat] Add automate e2e test framework for extensible integration tests (vllm-project#655)
* feat: add Golang-based E2E test framework for integration tests Signed-off-by: bitliu <[email protected]> * lint Signed-off-by: bitliu <[email protected]> --------- Signed-off-by: bitliu <[email protected]>
1 parent c5f55f8 commit fb52631

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+8235
-803
lines changed

‎.github/workflows/quickstart-integration-test.yml‎ renamed to ‎.github/workflows/integration-test-docker.yml‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,9 @@
1-
name: Quickstart Integration Test
1+
name: Integration Test [Docker Compose]
22

33
on:
44
pull_request:
55
branches:
66
- main
7-
paths:
8-
- 'scripts/quickstart.sh'
9-
- 'deploy/docker-compose/**'
10-
- 'config/config.yaml'
11-
- 'tools/make/common.mk'
12-
- 'tools/make/models.mk'
13-
- 'tools/make/docker.mk'
147
workflow_dispatch: # Allow manual triggering
158

169
jobs:

‎.github/workflows/helm-ci.yml‎ renamed to ‎.github/workflows/integration-test-helm.yml‎

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,12 @@
1-
name: Helm Chart CI
1+
name: Integration Test [Helm]
22

33
on:
44
push:
55
branches:
66
- main
7-
paths:
8-
- 'deploy/helm/**'
9-
- '.github/workflows/helm-ci.yml'
107
pull_request:
118
branches:
129
- main
13-
paths:
14-
- 'deploy/helm/**'
15-
- '.github/workflows/helm-ci.yml'
1610
workflow_dispatch:
1711

1812
env:
@@ -161,9 +155,6 @@ jobs:
161155
# CI environment: Download only essential model to avoid OOM
162156
# Only download all-MiniLM-L12-v2 (smallest model ~120MB)
163157
helm install semantic-router ${{ env.CHART_PATH }} \
164-
--set initContainer.resources.limits.memory=2Gi \
165-
--set initContainer.resources.requests.memory=1Gi \
166-
--set-json 'initContainer.models=[{"name":"all-MiniLM-L12-v2","repo":"sentence-transformers/all-MiniLM-L12-v2"}]' \
167158
--namespace vllm-semantic-router-system \
168159
--wait \
169160
--timeout 10m \
@@ -272,9 +263,6 @@ jobs:
272263
echo "::group::Upgrade Chart"
273264
# Use same minimal config for upgrade test
274265
helm upgrade semantic-router ${{ env.CHART_PATH }} \
275-
--set initContainer.resources.limits.memory=2Gi \
276-
--set initContainer.resources.requests.memory=1Gi \
277-
--set-json 'initContainer.models=[{"name":"all-MiniLM-L12-v2","repo":"sentence-transformers/all-MiniLM-L12-v2"}]' \
278266
--namespace vllm-semantic-router-system \
279267
--wait \
280268
--timeout 10m

‎.github/workflows/integration-test-k8s.yml‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
name: Integration Test [Kubernetes]
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
push:
8+
branches:
9+
- main
10+
workflow_dispatch: # Allow manual triggering
11+
12+
jobs:
13+
integration-test:
14+
runs-on: ubuntu-latest
15+
timeout-minutes: 60
16+
17+
steps:
18+
- name: Check out the repo
19+
uses: actions/checkout@v4
20+
21+
- name: Set up Go
22+
uses: actions/setup-go@v5
23+
with:
24+
go-version: '1.24'
25+
26+
- name: Set up Rust
27+
uses: actions-rust-lang/setup-rust-toolchain@v1
28+
with:
29+
toolchain: 1.90
30+
31+
- name: Install system dependencies
32+
run: |
33+
sudo apt-get update
34+
sudo apt-get install -y \
35+
make \
36+
curl \
37+
build-essential \
38+
pkg-config
39+
40+
- name: Install Kind
41+
run: |
42+
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
43+
chmod +x ./kind
44+
sudo mv ./kind /usr/local/bin/kind
45+
46+
- name: Install kubectl
47+
run: |
48+
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
49+
chmod +x kubectl
50+
sudo mv kubectl /usr/local/bin/kubectl
51+
52+
- name: Install Helm
53+
run: |
54+
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
55+
56+
- name: Download E2E test dependencies
57+
run: |
58+
cd e2e && go mod download
59+
60+
- name: Build E2E test binary
61+
run: |
62+
make build-e2e
63+
64+
- name: Run Integration E2E tests
65+
id: e2e-test
66+
run: |
67+
set +e # Don't exit on error, we want to capture the result
68+
make e2e-test E2E_PROFILE=ai-gateway E2E_VERBOSE=true E2E_KEEP_CLUSTER=false
69+
TEST_EXIT_CODE=$?
70+
echo "test_exit_code=${TEST_EXIT_CODE}" >> $GITHUB_OUTPUT
71+
exit ${TEST_EXIT_CODE}
72+
73+
- name: Upload test reports
74+
if: always()
75+
uses: actions/upload-artifact@v4
76+
with:
77+
name: test-reports
78+
path: |
79+
test-report.json
80+
test-report.md
81+
semantic-router-logs.txt
82+
retention-days: 30
83+
84+
- name: Create test summary from report
85+
if: always()
86+
run: |
87+
if [ -f "test-report.md" ]; then
88+
echo "=== Reading test report from test-report.md ==="
89+
cat test-report.md >> $GITHUB_STEP_SUMMARY
90+
91+
# Add semantic-router logs section if available
92+
if [ -f "semantic-router-logs.txt" ]; then
93+
cat >> $GITHUB_STEP_SUMMARY << 'EOF'
94+
95+
---
96+
97+
### 📝 Semantic Router Logs
98+
99+
<details>
100+
<summary>Click to view semantic-router logs</summary>
101+
102+
```
103+
EOF
104+
# Add first 500 lines of logs to summary (to avoid exceeding GitHub limits)
105+
head -n 500 semantic-router-logs.txt >> $GITHUB_STEP_SUMMARY
106+
107+
# Check if there are more lines
108+
TOTAL_LINES=$(wc -l < semantic-router-logs.txt)
109+
if [ "$TOTAL_LINES" -gt 500 ]; then
110+
cat >> $GITHUB_STEP_SUMMARY << EOF
111+
112+
... (showing first 500 lines of $TOTAL_LINES total lines)
113+
114+
📦 Full logs are available in the workflow artifacts: semantic-router-logs.txt
115+
EOF
116+
fi
117+
118+
cat >> $GITHUB_STEP_SUMMARY << 'EOF'
119+
```
120+
121+
</details>
122+
EOF
123+
fi
124+
125+
# Add additional context
126+
cat >> $GITHUB_STEP_SUMMARY << 'EOF'
127+
128+
---
129+
130+
### 📚 Additional Resources
131+
132+
- **Trigger:** ${{ github.event_name }}
133+
- **Branch:** `${{ github.ref_name }}`
134+
- **Commit:** `${{ github.sha }}`
135+
- **Workflow Run:** [${{ github.run_id }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
136+
- [E2E Test Framework Documentation](https://github.com/${{ github.repository }}/tree/main/e2e)
137+
- [AI Gateway Profile](https://github.com/${{ github.repository }}/tree/main/e2e/profiles/ai-gateway)
138+
139+
### 📦 Artifacts
140+
141+
- **test-report.json** - Detailed test results in JSON format
142+
- **test-report.md** - Human-readable test report
143+
- **semantic-router-logs.txt** - Complete semantic-router pod logs
144+
- All artifacts are retained for 30 days
145+
EOF
146+
else
147+
echo "⚠️ Test report file not found!" >> $GITHUB_STEP_SUMMARY
148+
echo "" >> $GITHUB_STEP_SUMMARY
149+
echo "The E2E test framework did not generate a report file." >> $GITHUB_STEP_SUMMARY
150+
echo "This might indicate that the test failed before report generation." >> $GITHUB_STEP_SUMMARY
151+
fi
152+
153+
- name: Clean up
154+
if: always()
155+
run: |
156+
make e2e-cleanup || true
157+

‎.github/workflows/k8s-config-test.yml‎

Lines changed: 0 additions & 116 deletions
This file was deleted.

0 commit comments

Comments
 (0)