[Feat] Add automate e2e test framework for extensible integration tests

[Xunzhuo]

[Feat] Add Golang-based E2E Test Framework for Integration Tests

This PR introduces a comprehensive end-to-end testing framework written in Go to replace the existing shell-based integration tests, providing better maintainability, extensibility, and test coverage for Semantic Router.

What Changed

New E2E Test Framework (e2e/)

• Implemented a modular, profile-based testing architecture with separation of concerns
• Added support for multiple deployment profiles (currently: ai-gateway, with extensibility for istio, production-stack, etc.)
• Created 7 comprehensive test cases covering core functionality:
  • chat-completions-request: Basic API validation
  • chat-completions-stress-request: Sequential stress testing (1000 requests)
  • chat-completions-progressive-stress: Progressive QPS testing (10/20/50/100 QPS)
  • domain-classify: Domain classification accuracy (65 test cases)
  • semantic-cache: Cache hit rate validation (5 groups)
  • pii-detection: PII detection and blocking (10 PII types)
  • jailbreak-detection: Jailbreak attack detection (10 attack types)

Framework Features

• Automatic Kind cluster lifecycle management
• Docker image building and loading into Kind
• Helm deployment automation with health checks
• Automatic port forwarding with proper cleanup
• Detailed test reporting (JSON + Markdown formats)
• Support for parallel test execution
• Advanced workflows: setup-once, run-multiple-times for rapid development

Test Infrastructure

• Generated 2,352 lines of realistic test data across 4 JSON files
• Implemented comprehensive test case registry system
• Added reusable helper functions for Kubernetes operations
• Created detailed logging and debugging capabilities

CI/CD Improvements

• Consolidated and renamed GitHub Actions workflows for clarity:
  • quickstart-integration-test.yml → integration-test-docker.yml
  • helm-ci.yml → integration-test-helm.yml
  • New: integration-test-k8s.yml (replaces 5 separate K8s test workflows)
• Removed 803 lines of redundant workflow configurations
• Improved test organization and maintainability

Bug Fixes & Enhancements

• Fixed PII detection policy to properly handle block actions
• Added VSR decision headers to classification responses for better observability
• Updated AI Gateway configurations for better test compatibility
• Added comprehensive documentation (e2e/README.md - 473 lines)

Makefile Targets (tools/make/e2e.mk)

make e2e-test                    # Run all tests
make e2e-test-debug              # Run with debugging enabled
make e2e-test-specific           # Run specific test cases
make e2e-setup                   # Setup environment only
make e2e-test-only               # Run tests without setup
make e2e-cleanup                 # Clean up test cluster
make e2e-help                    # Show detailed help

Why These Changes

Problems with Old Approach:

• Shell-based tests were hard to maintain and extend
• Limited test coverage and reporting capabilities
• Difficult to debug failures
• No support for different deployment profiles
• Tests were tightly coupled to specific environments

Benefits of New Framework:

• ✅ Maintainable: Clear separation of concerns, reusable test cases
• ✅ Extensible: Easy to add new profiles and test cases
• ✅ Comprehensive: Detailed metrics and reporting
• ✅ Developer-friendly: Fast iteration with setup-once workflows
• ✅ Production-ready: Supports multiple deployment scenarios
• ✅ Well-documented: 473-line README with examples

Testing Done

• ✅ All 7 test cases pass successfully with the ai-gateway profile
• ✅ Verified cluster lifecycle management (create/cleanup)
• ✅ Tested parallel execution mode
• ✅ Validated setup-once, run-multiple workflows
• ✅ Confirmed proper port forwarding cleanup
• ✅ Generated and validated test reports (JSON + Markdown)
• ✅ Pre-commit checks pass

Documentation

• Added comprehensive e2e/README.md with:
  • Architecture overview
  • Quick start guide
  • Advanced workflows
  • Development guide for adding new tests/profiles
  • Complete environment variable reference
  • Test data format documentation

Related Issues

FIX #xxxx (if applicable)

---

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW

---

• Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

[github-actions]

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 Root Directory

Owners: @rootfs, @Xunzhuo
Files changed:

• .github/workflows/integration-test-k8s.yml
• Makefile
• .github/workflows/integration-test-docker.yml
• .github/workflows/integration-test-helm.yml

📁 e2e

Owners: @Xunzhuo
Files changed:

• e2e/.gitignore
• e2e/OWNER
• e2e/README.md
• e2e/cmd/e2e/main.go
• e2e/go.mod
• e2e/go.sum
• e2e/pkg/cluster/kind.go
• e2e/pkg/docker/builder.go
• e2e/pkg/framework/debug.go
• e2e/pkg/framework/report.go
• e2e/pkg/framework/runner.go
• e2e/pkg/framework/types.go
• e2e/pkg/helm/deployer.go
• e2e/pkg/helpers/kubernetes.go
• e2e/pkg/testcases/registry.go
• e2e/profiles/ai-gateway/profile.go
• e2e/scripts/generate_test_data.py
• e2e/testcases/cache.go
• e2e/testcases/chat_completions_progressive_stress.go
• e2e/testcases/chat_completions_request.go
• e2e/testcases/chat_completions_stress_request.go
• e2e/testcases/common.go
• e2e/testcases/domain_classify.go
• e2e/testcases/jailbreak_detection.go
• e2e/testcases/pii_detection.go
• e2e/testcases/testdata/cache_cases.json
• e2e/testcases/testdata/domain_classify_cases.json
• e2e/testcases/testdata/jailbreak_detection_cases.json
• e2e/testcases/testdata/pii_detection_cases.json

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

• src/semantic-router/pkg/utils/pii/policy_test.go
• src/semantic-router/pkg/extproc/processor_req_body.go
• src/semantic-router/pkg/extproc/req_filter_classification.go
• src/semantic-router/pkg/extproc/req_filter_pii.go
• src/semantic-router/pkg/utils/pii/policy.go

📁 tools

Owners: @yuluo-yx, @rootfs, @Xunzhuo
Files changed:

• tools/make/e2e.mk
• tools/linter/codespell/.codespell.skip

📁 deploy

Owners: @rootfs, @Xunzhuo
Files changed:

• deploy/kubernetes/ai-gateway/aigw-resources/gwapi-resources.yaml
• deploy/kubernetes/ai-gateway/semantic-router-values/values.yaml
• deploy/kubernetes/ai-gateway/semantic-router/config.yaml
• deploy/kubernetes/aibrix/aigw-resources/gwapi-resources.yaml

---

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	9b93ef7
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/69185b4e5c66210008468caa
😎 Deploy Preview	https://deploy-preview-655--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[rootfs]

This is really cool! Thanks for making this happen!