feat: Add istio profile for E2E testing framework

This PR implements the Istio profile for the E2E testing framework as requested in issue vllm-project#656.

## What Changed

### Core Implementation (Issue vllm-project#656 Requirements)
- **New Istio Profile** (`e2e/profiles/istio/`)
  - Implements 5-stage deployment: Istio control plane, namespace configuration,
    Semantic Router with sidecar injection, Istio Gateway/VirtualService/DestinationRule,
    and environment verification
  - Robust error handling with panic recovery and defer-based cleanup
  - Configurable Istio version via `ISTIO_VERSION` environment variable (default: 1.28.0)
  - Comprehensive service health verification with 60-second stabilization period

- **4 Istio-Specific Test Cases** (100% passing)
  1. `istio-sidecar-health-check` - Verify Envoy sidecar injection and health
  2. `istio-traffic-routing` - Test routing through Istio ingress gateway
  3. `istio-mtls-verification` - Verify mutual TLS configuration and certificates
  4. `istio-tracing-observability` - Validate distributed tracing and metrics

- **Integration & Documentation**
  - Registered profile in `e2e/cmd/e2e/main.go`
  - Added comprehensive documentation to `e2e/README.md`
  - Integrated into CI matrix (`.github/workflows/integration-test-k8s.yml`)
  - Updated Make targets help text (`tools/make/e2e.mk`)

### Out-of-Scope Changes (Justified)

#### Helper Function Enhancement (`e2e/pkg/helpers/kubernetes.go`)
- **Change**: Added `GetServiceByLabelInNamespace()` with backward-compatible wrapper
- **Why**: Original `GetEnvoyServiceName()` was hardcoded to `"envoy-gateway-system"`,
  preventing reuse for Istio which uses `"istio-system"` namespace
- **Approach**: Backward-compatible wrapper pattern
  - Old function `GetEnvoyServiceName(ctx, client, labelSelector, verbose)` still works
  - Calls new generic function `GetServiceByLabelInNamespace()` with hardcoded namespace
  - Istio uses the new flexible function directly
- **Impact**:
  - **ZERO changes** to existing profiles (ai-gateway, aibrix, dynamic-config)
  - Makes helper function generic and reusable for future profiles
  - Marked old function as deprecated for future cleanup
- **Safety**: Existing profiles continue to work unchanged, Istio gets flexibility it needs

## Test Coverage

This PR implements the **4 Istio-specific tests** required by issue vllm-project#656:
- ✅ Basic health check with Istio sidecar
- ✅ Traffic routing through Istio gateway
- ✅ mTLS verification
- ✅ Request tracing and observability

**Test Results**: 4/4 passing (100%)

**Note on Signal-Decision Engine Tests**:
Signal-decision engine tests (introduced in PR vllm-project#695) are intentionally excluded
to maintain focus on Istio integration validation. These tests validate semantic
router logic (already covered by ai-gateway, aibrix, and dynamic-config profiles)
rather than Istio mesh functionality. Following the established pattern where
PR vllm-project#695 retroactively added signal-decision tests to existing profiles, these
can be added to the Istio profile in a follow-up PR.

**Note on Kubernetes Version**:
CI uses Kind v0.22.0 which defaults to Kubernetes 1.29.2, meeting Istio 1.28+
compatibility requirements without requiring explicit version pinning.

## Implementation Highlights

- 5-stage deployment with comprehensive error handling
- Panic recovery with defer-based cleanup
- Service health verification before tests (prevents 503 errors)
- 60-second stabilization period for mesh readiness
- Complete Istio resource cleanup in teardown
- All Kubernetes resources properly namespaced
- Configurable Istio version for testing flexibility

## Testing Done

✅ All 4 Istio test cases pass successfully
✅ Verified cluster lifecycle management (create/cleanup)
✅ Confirmed Istio sidecar injection and health
✅ Validated traffic routing through Istio gateway
✅ Verified mTLS configuration and certificates
✅ Confirmed distributed tracing headers and metrics
✅ Tested with Istio 1.28.0 and CI's default Kubernetes version

## Makefile Targets

```bash
make e2e-test E2E_PROFILE=istio              # Run all Istio tests
make e2e-test E2E_PROFILE=istio E2E_VERBOSE=1 # Run with verbose output
ISTIO_VERSION=1.28.0 make e2e-test E2E_PROFILE=istio  # Custom Istio version
```

Resolves vllm-project#656

Signed-off-by: Asaad Balum <[email protected]>

Author: asaadbalum

.github/workflows/integration-test-k8s.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       fail-fast: false  # Continue testing other profiles even if one fails
       matrix:
-        profile: [ai-gateway, aibrix]
+        profile: [ai-gateway, aibrix, istio]
 
     steps:
       - name: Check out the repo

e2e/README.md

@@ -14,7 +14,7 @@ The framework follows a **separation of concerns** design:
 
 - **ai-gateway**: Tests Semantic Router with Envoy AI Gateway integration
 - **aibrix**: Tests Semantic Router with vLLM AIBrix integration
-- **istio**: Tests Semantic Router with Istio Gateway (future)
+- **istio**: Tests Semantic Router with Istio service mesh integration
 - **production-stack**: Tests vLLM Production Stack configurations (future)
 - **llm-d**: Tests with LLM-D (future)
 - **dynamo**: Tests with Nvidia Dynamo (future)
@@ -517,3 +517,136 @@ func (p *Profile) GetServiceConfig() framework.ServiceConfig {
 ```
 
 See `profiles/ai-gateway/` for a complete example.
+
+## Profile Details
+
+### Istio Profile
+
+The Istio profile tests Semantic Router deployment and functionality in an Istio service mesh environment.
+
+**What it Tests:**
+
+- Istio sidecar injection and health
+- Traffic routing through Istio ingress gateway
+- Mutual TLS (mTLS) between services
+- Distributed tracing and observability
+
+**Prerequisites:**
+
+- `istioctl` must be installed on the system
+- Docker and Kind (managed by E2E framework)
+
+**Components Deployed:**
+
+1. **Istio Control Plane** (`istio-system` namespace):
+   - `istiod` - Istio control plane
+   - `istio-ingressgateway` - Ingress gateway for external traffic
+
+2. **Semantic Router** (`semantic-router` namespace):
+   - Deployed via Helm with Istio sidecar injection enabled
+   - Namespace labeled with `istio-injection=enabled`
+
+3. **Istio Resources**:
+   - `Gateway` - Configures ingress gateway on port 80
+   - `VirtualService` - Routes traffic to Semantic Router service
+   - `DestinationRule` - Enables mTLS with `ISTIO_MUTUAL` mode
+
+**Test Cases:**
+
+| Test Case | Description | What it Validates |
+|-----------|-------------|-------------------|
+| `istio-sidecar-health-check` | Verify Envoy sidecar injection | - Istio-proxy container exists<br>- Sidecar is healthy and ready<br>- Namespace has `istio-injection=enabled` label |
+| `istio-traffic-routing` | Test routing through Istio gateway | - Gateway and VirtualService exist<br>- Requests route correctly to Semantic Router<br>- Istio/Envoy headers present in responses |
+| `istio-mtls-verification` | Verify mutual TLS configuration | - DestinationRule has `ISTIO_MUTUAL` mode<br>- mTLS certificates present in istio-proxy<br>- PeerAuthentication policy (if configured) |
+| `istio-tracing-observability` | Check distributed tracing and metrics | - Trace headers propagated<br>- Envoy metrics exposed<br>- Telemetry configuration<br>- Access logs enabled |
+
+**Usage:**
+
+```bash
+# Run all Istio tests
+make e2e-test E2E_PROFILE=istio
+
+# Run specific Istio tests
+make e2e-test-specific E2E_PROFILE=istio E2E_TESTS="istio-sidecar-health-check,istio-mtls-verification"
+
+# Run with verbose output
+./bin/e2e -profile istio -verbose
+
+# Keep cluster for debugging
+make e2e-test E2E_PROFILE=istio E2E_KEEP_CLUSTER=true
+```
+
+**Architecture:**
+
+```
+┌─────────────────────────────────────────┐
+│         Istio Ingress Gateway            │
+│      (istio-system namespace)            │
+│   Port 80 → semantic-router service      │
+└────────────┬────────────────────────────┘
+             │
+             ▼
+┌─────────────────────────────────────────┐
+│    Semantic Router Pod                   │
+│  (semantic-router namespace)             │
+│  ┌─────────────┐  ┌──────────────────┐  │
+│  │   Main      │  │  Istio-Proxy     │  │
+│  │ Container   │◄─┤  (Envoy Sidecar) │  │
+│  │             │  │                  │  │
+│  │  :8801      │  │  mTLS, Tracing   │  │
+│  └─────────────┘  └──────────────────┘  │
+└─────────────────────────────────────────┘
+             │
+             ▼
+┌─────────────────────────────────────────┐
+│         Istiod (Control Plane)           │
+│  - Config distribution                   │
+│  - Certificate management (mTLS)         │
+│  - Sidecar injection                     │
+└─────────────────────────────────────────┘
+```
+
+**Key Features Tested:**
+
+- ✅ **Automatic Sidecar Injection**: Istio automatically injects Envoy proxy sidecars into pods
+- ✅ **Traffic Management**: Requests route through Istio Gateway → VirtualService → Semantic Router
+- ✅ **Security (mTLS)**: Automatic mutual TLS encryption and authentication between services
+- ✅ **Observability**: Distributed tracing, metrics collection, and access logs
+- ✅ **Service Mesh Integration**: Semantic Router operates correctly within Istio mesh
+
+**Setup Steps (Automated by Profile):**
+
+1. Install Istio control plane using `istioctl install`
+2. Create namespace with `istio-injection=enabled` label
+3. Deploy Semantic Router via Helm (sidecar auto-injected)
+4. Create Istio Gateway and VirtualService for traffic routing
+5. Create DestinationRule for mTLS configuration
+6. Verify all components are ready
+
+**Troubleshooting:**
+
+If tests fail, check:
+
+```bash
+# Check Istio installation
+kubectl get pods -n istio-system
+
+# Check sidecar injection
+kubectl get pods -n semantic-router -o jsonpath='{.items[*].spec.containers[*].name}'
+
+# Check Istio resources
+kubectl get gateway,virtualservice,destinationrule -n semantic-router
+
+# Check mTLS configuration
+kubectl get destinationrule semantic-router -n semantic-router -o yaml
+
+# View Istio proxy logs
+kubectl logs -n semantic-router <pod-name> -c istio-proxy
+```
+
+**Related Resources:**
+
+- [Istio Documentation](https://istio.io/latest/docs/)
+- [Istio Traffic Management](https://istio.io/latest/docs/concepts/traffic-management/)
+- [Istio Security (mTLS)](https://istio.io/latest/docs/concepts/security/)
+- [Istio Observability](https://istio.io/latest/docs/concepts/observability/)

e2e/cmd/e2e/main.go

@@ -12,10 +12,12 @@ import (
 	aigateway "github.com/vllm-project/semantic-router/e2e/profiles/ai-gateway"
 	aibrix "github.com/vllm-project/semantic-router/e2e/profiles/aibrix"
 	dynamicconfig "github.com/vllm-project/semantic-router/e2e/profiles/dynamic-config"
+	istio "github.com/vllm-project/semantic-router/e2e/profiles/istio"
 
 	// Import profiles to register test cases
 	_ "github.com/vllm-project/semantic-router/e2e/profiles/ai-gateway"
 	_ "github.com/vllm-project/semantic-router/e2e/profiles/aibrix"
+	_ "github.com/vllm-project/semantic-router/e2e/profiles/istio"
 )
 
 const version = "v1.0.0"
@@ -103,9 +105,8 @@ func getProfile(name string) (framework.Profile, error) {
 		return dynamicconfig.NewProfile(), nil
 	case "aibrix":
 		return aibrix.NewProfile(), nil
-	// Add more profiles here as they are implemented
-	// case "istio":
-	//     return istio.NewProfile(), nil
+	case "istio":
+		return istio.NewProfile(), nil
 	default:
 		return nil, fmt.Errorf("unknown profile: %s", name)
 	}

e2e/pkg/helpers/kubernetes.go

@@ -38,22 +38,28 @@ func CheckDeployment(ctx context.Context, client *kubernetes.Clientset, namespac
 
 // GetEnvoyServiceName finds the Envoy service name in the envoy-gateway-system namespace
 // using label selectors to match the Gateway-owned service
+// Deprecated: Use GetServiceByLabelInNamespace for more flexibility
 func GetEnvoyServiceName(ctx context.Context, client *kubernetes.Clientset, labelSelector string, verbose bool) (string, error) {
-	services, err := client.CoreV1().Services("envoy-gateway-system").List(ctx, metav1.ListOptions{
+	return GetServiceByLabelInNamespace(ctx, client, "envoy-gateway-system", labelSelector, verbose)
+}
+
+// GetServiceByLabelInNamespace finds a service by label selector in a specific namespace
+func GetServiceByLabelInNamespace(ctx context.Context, client *kubernetes.Clientset, namespace string, labelSelector string, verbose bool) (string, error) {
+	services, err := client.CoreV1().Services(namespace).List(ctx, metav1.ListOptions{
 		LabelSelector: labelSelector,
 	})
 	if err != nil {
 		return "", fmt.Errorf("failed to list services with selector %s: %w", labelSelector, err)
 	}
 
 	if len(services.Items) == 0 {
-		return "", fmt.Errorf("no service found with selector %s in envoy-gateway-system namespace", labelSelector)
+		return "", fmt.Errorf("no service found with selector %s in %s namespace", labelSelector, namespace)
 	}
 
 	// Return the first matching service (should only be one)
 	serviceName := services.Items[0].Name
 	if verbose {
-		fmt.Printf("[Helper] Found Envoy service: %s (matched by labels: %s)\n", serviceName, labelSelector)
+		fmt.Printf("[Helper] Found service: %s (matched by labels: %s in namespace: %s)\n", serviceName, labelSelector, namespace)
 	}
 
 	return serviceName, nil