update

Signed-off-by: bitliu <[email protected]>

Author: Xunzhuo

.github/workflows/integration-test-k8s.yml

@@ -10,7 +10,7 @@ on:
   workflow_dispatch: # Allow manual triggering
 
 jobs:
-  test-ai-gateway:
+  integration-test:
     runs-on: ubuntu-latest
     timeout-minutes: 60
 

deploy/kubernetes/ai-gateway/aigw-resources/gwapi-resources.yaml

@@ -69,13 +69,6 @@ spec:
         value: math-expert
     backendRefs:
     - name: vllm-llama3-8b-instruct
-  - matches:
-    - headers:
-      - type: Exact
-        name: x-ai-eg-model
-        value: math-expert
-    backendRefs:
-    - name: vllm-llama3-8b-instruct
   - matches:
     - headers:
       - type: Exact

e2e/README.md

@@ -4,7 +4,13 @@ A comprehensive end-to-end testing framework for Semantic Router with support fo
 
 ## Architecture
 
-The framework is designed to be extensible and supports multiple test profiles:
+The framework follows a **separation of concerns** design:
+
+- **Profiles**: Define deployment environments and configurations
+- **Test Cases**: Reusable test logic that can be shared across profiles
+- **Framework**: Core infrastructure for test execution and reporting
+
+### Supported Profiles
 
 - **ai-gateway**: Tests Semantic Router with Envoy AI Gateway integration
 - **istio**: Tests Semantic Router with Istio Gateway (future)
@@ -24,14 +30,41 @@ e2e/
 │   ├── cluster/          # Kind cluster management
 │   ├── docker/           # Docker image operations
 │   ├── helm/             # Helm deployment utilities
-│   └── testcases/        # Test case definitions
+│   ├── helpers/          # Kubernetes helper functions
+│   └── testcases/        # Test case registry
+├── testcases/            # Reusable test cases (shared across profiles)
+│   ├── testdata/         # Test data files
+│   ├── common.go         # Common helper functions
+│   ├── chat_completions_request.go
+│   ├── domain_classify.go
+│   ├── cache.go
+│   ├── pii_detection.go
+│   └── jailbreak_detection.go
 ├── profiles/
-│   ├── ai-gateway/       # AI Gateway test profile
-│   ├── istio/            # Istio test profile (future)
-│   └── ...
+│   └── ai-gateway/       # AI Gateway test profile
+│       └── profile.go    # Profile definition and environment setup
 └── README.md
 ```
 
+## Available Test Cases
+
+The framework includes the following test cases (all in `e2e/testcases/`):
+
+| Test Case | Description | Metrics |
+|-----------|-------------|---------|
+| `chat-completions-request` | Basic chat completions API test | Response validation |
+| `domain-classify` | Domain classification accuracy | 65 cases, accuracy rate |
+| `cache` | Semantic cache hit rate | 5 groups, cache hit rate |
+| `pii-detection` | PII detection and blocking | 10 PII types, detection rate, block rate |
+| `jailbreak-detection` | Jailbreak attack detection | 10 attack types, detection rate, block rate |
+
+All test cases:
+
+- Use model name `"MoM"`
+- Automatically clean up port forwarding
+- Generate detailed reports with statistics
+- Support verbose logging
+
 ## Quick Start
 
 ### Install dependencies (optional)
@@ -65,6 +98,15 @@ make e2e-test USE_EXISTING_CLUSTER=true
 make e2e-test VERBOSE=true
 ```
 
+### Test Reports
+
+After running tests, reports are generated:
+
+- `test-report.json`: Structured test results
+- `test-report.md`: Human-readable Markdown report
+
+Each test case also prints detailed statistics to the console.
+
 ## Adding New Test Profiles
 
 1. Create a new directory under `profiles/`
@@ -74,32 +116,109 @@ make e2e-test VERBOSE=true
 
 See `profiles/ai-gateway/` for a complete example.
 
+## Key Concepts
+
+### Profile vs Test Case Separation
+
+**Profiles** are responsible for:
+
+- Deploying the test environment (Helm charts, Kubernetes resources)
+- Verifying environment health
+- Providing service configuration (namespace, labels, port mappings)
+
+**Test Cases** are responsible for:
+
+- Executing test logic
+- Validating functionality
+- Reporting results
+
+This separation allows test cases to be **reused across different profiles** by simply providing different service configurations.
+
+**Benefits:**
+
+- ✅ Test cases are independent of deployment details
+- ✅ Easy to add new profiles without duplicating test logic
+- ✅ Profiles can share common test cases
+- ✅ Test cases can be maintained in one place
+- ✅ Clear separation of concerns
+
+### Service Configuration
+
+Profiles provide service configuration to test cases via `ServiceConfig`:
+
+```go
+type ServiceConfig struct {
+    LabelSelector string  // e.g., "gateway.envoyproxy.io/owning-gateway-namespace=default,..."
+    Namespace     string  // Service namespace
+    Name          string  // Service name (optional, if empty uses LabelSelector)
+    PortMapping   string  // e.g., "8080:80" (localPort:servicePort)
+}
+```
+
+Test cases use this configuration to connect to the deployed service without knowing the specific deployment details.
+
 ## Test Case Registration
 
 Test cases are registered using a simple function-based approach:
 
 ```go
 func init() {
-    testcases.Register("my-test", testcases.TestCase{
-        Name:        "My Test",
+    pkgtestcases.Register("my-test", pkgtestcases.TestCase{
         Description: "Description of what this test does",
-        Fn: func(ctx context.Context, client *kubernetes.Clientset) error {
-            // Test implementation
-            return nil
-        },
+        Tags:        []string{"functional", "llm"},
+        Fn:          testMyFeature,
     })
 }
+
+func testMyFeature(ctx context.Context, client *kubernetes.Clientset, opts pkgtestcases.TestCaseOptions) error {
+    // Setup connection to service
+    localPort, stopPortForward, err := setupServiceConnection(ctx, client, opts)
+    if err != nil {
+        return err
+    }
+    defer stopPortForward() // Always clean up port forwarding
+
+    // Test implementation using localPort
+    // ...
+    return nil
+}
 ```
 
 ## Framework Features
 
 - **Automatic cluster lifecycle management**: Creates and cleans up Kind clusters
 - **Docker image building and loading**: Builds images and loads them into Kind
 - **Helm deployment automation**: Deploys required Helm charts
-- **Parallel test execution**: Runs independent tests in parallel
+- **Automatic port forwarding cleanup**: Each test case cleans up its port forwarding
 - **Detailed logging**: Provides comprehensive test output
+- **Test reporting**: Generates JSON and Markdown reports
 - **Resource cleanup**: Ensures proper cleanup even on failures
 
+## Test Data
+
+Test data is stored in `e2e/testcases/testdata/` as JSON files. Each test case loads its own test data.
+
+**Available Test Data:**
+
+- `domain_classify_cases.json`: 65 test cases across 13 categories
+- `cache_cases.json`: 5 groups of similar questions for cache testing
+- `pii_detection_cases.json`: 10 PII types (email, phone, SSN, etc.)
+- `jailbreak_detection_cases.json`: 10 attack types (prompt injection, DAN, etc.)
+
+**Test Data Format Example:**
+
+```json
+{
+  "cases": [
+    {
+      "question": "What is 2+2?",
+      "expected_category": "math",
+      "expected_reasoning": "Basic arithmetic question"
+    }
+  ]
+}
+```
+
 ## Prerequisites
 
 Before running E2E tests, ensure you have the following tools installed:
@@ -112,34 +231,138 @@ Before running E2E tests, ensure you have the following tools installed:
 
 ## Development
 
-### Adding a new test case
+### Adding a New Test Case
 
-1. Create a new test function in `profiles/<profile>/testcases.go`
-2. Register it in the `init()` function
-3. Add the test case name to the profile's `GetTestCases()` method
+Test cases are created in the `e2e/testcases/` directory and can be reused across multiple profiles.
 
-Example:
+**Steps:**
+
+1. Create a new file in `e2e/testcases/` (e.g., `my_feature.go`)
+2. Implement the test function with proper cleanup
+3. Register it in the `init()` function
+4. Add test data to `e2e/testcases/testdata/` if needed
+5. Add the test case name to any profile's `GetTestCases()` method
+
+**Example:**
 
 ```go
+package testcases
+
+import (
+    "context"
+    "fmt"
+
+    "k8s.io/client-go/kubernetes"
+    pkgtestcases "github.com/vllm-project/semantic-router/e2e/pkg/testcases"
+)
+
 func init() {
-    testcases.Register("my-new-test", testcases.TestCase{
-        Description: "My new test description",
-        Tags:        []string{"ai-gateway", "functional"},
-        Fn:          testMyNewFeature,
+    pkgtestcases.Register("my-feature", pkgtestcases.TestCase{
+        Description: "Test my new feature",
+        Tags:        []string{"functional", "llm"},
+        Fn:          testMyFeature,
     })
 }
 
-func testMyNewFeature(ctx context.Context, client *kubernetes.Clientset, opts testcases.TestCaseOptions) error {
-    // Test implementation
+func testMyFeature(ctx context.Context, client *kubernetes.Clientset, opts pkgtestcases.TestCaseOptions) error {
+    if opts.Verbose {
+        fmt.Println("[Test] Testing my feature")
+    }
+
+    // Setup service connection and get local port
+    localPort, stopPortForward, err := setupServiceConnection(ctx, client, opts)
+    if err != nil {
+        return err
+    }
+    defer stopPortForward() // IMPORTANT: Always clean up port forwarding
+
+    // Test implementation using localPort
+    url := fmt.Sprintf("http://localhost:%s/v1/chat/completions", localPort)
+    // ... send requests and validate responses
+
     return nil
 }
 ```
 
-### Adding a new profile
+**Important Notes:**
 
-1. Create a new directory under `profiles/`
-2. Implement the `Profile` interface
-3. Register test cases
-4. Update `cmd/e2e/main.go` to include the new profile
+- Always use `defer stopPortForward()` to clean up port forwarding
+- Use `opts.ServiceConfig` to get service connection details
+- Use `opts.Verbose` for debug logging
+- Load test data from `e2e/testcases/testdata/`
+- Use model name `"MoM"` in all requests
+
+### Adding a New Profile
+
+Profiles define deployment environments and can reuse existing test cases.
+
+**Steps:**
+
+1. Create a new directory under `profiles/` (e.g., `profiles/istio/`)
+2. Create `profile.go` implementing the `Profile` interface
+3. Implement required methods:
+   - `Setup()`: Deploy environment
+   - `Teardown()`: Clean up resources
+   - `GetTestCases()`: Return list of test case names to run
+   - `GetServiceConfig()`: Provide service configuration
+4. Import the `testcases` package to register test cases
+5. Update `cmd/e2e/main.go` to include the new profile
+
+**Example:**
+
+```go
+package myprofile
+
+import (
+    "context"
+
+    "github.com/vllm-project/semantic-router/e2e/pkg/framework"
+
+    // Import testcases to register them
+    _ "github.com/vllm-project/semantic-router/e2e/testcases"
+)
+
+type Profile struct {
+    verbose bool
+}
+
+func NewProfile(verbose bool) *Profile {
+    return &Profile{verbose: verbose}
+}
+
+func (p *Profile) Name() string {
+    return "my-profile"
+}
+
+func (p *Profile) Description() string {
+    return "My custom deployment profile"
+}
+
+func (p *Profile) Setup(ctx context.Context, opts *framework.SetupOptions) error {
+    // Deploy your environment
+    return nil
+}
+
+func (p *Profile) Teardown(ctx context.Context, opts *framework.TeardownOptions) error {
+    // Clean up resources
+    return nil
+}
+
+func (p *Profile) GetTestCases() []string {
+    return []string{
+        "chat-completions-request",
+        "domain-classify",
+        // ... other test cases
+    }
+}
+
+func (p *Profile) GetServiceConfig() framework.ServiceConfig {
+    return framework.ServiceConfig{
+        LabelSelector: "app=my-service",
+        Namespace:     "default",
+        PortMapping:   "8080:80",
+    }
+}
+```
 
 See `profiles/ai-gateway/` for a complete example.

e2e/pkg/framework/report.go

@@ -291,6 +291,17 @@ func (rg *ReportGenerator) generateMarkdown() string {
 			errorMsg = fmt.Sprintf(" - Error: `%s`", result.Error.Error())
 		}
 		md += fmt.Sprintf("- %s **%s** (%s)%s\n", status, result.Name, result.Duration, errorMsg)
+
+		// Add details if available
+		if len(result.Details) > 0 {
+			md += "  <details>\n  <summary>Details</summary>\n\n"
+			md += "  | Metric | Value |\n"
+			md += "  |--------|-------|\n"
+			for key, value := range result.Details {
+				md += fmt.Sprintf("  | %s | %v |\n", key, value)
+			}
+			md += "\n  </details>\n"
+		}
 	}
 
 	// Add pod details by namespace