feat: Add istio profile for E2E testing framework

This PR implements the Istio profile for the E2E testing framework as requested in issue vllm-project#656.

## What Changed

### Core Implementation (Issue vllm-project#656 Requirements)
- **New Istio Profile** (`e2e/profiles/istio/`)
  - Implements 5-stage deployment: Istio control plane, namespace configuration,
    Semantic Router with sidecar injection, Istio Gateway/VirtualService/DestinationRule,
    and environment verification
  - Robust error handling with panic recovery and defer-based cleanup
  - Configurable Istio version via `ISTIO_VERSION` environment variable (default: 1.28.0)
  - Comprehensive service health verification with 60-second stabilization period

- **4 Istio-Specific Test Cases** (100% passing)
  1. `istio-sidecar-health-check` - Verify Envoy sidecar injection and health
  2. `istio-traffic-routing` - Test routing through Istio ingress gateway
  3. `istio-mtls-verification` - Verify mutual TLS configuration and certificates
  4. `istio-tracing-observability` - Validate distributed tracing and metrics

- **Integration & Documentation**
  - Registered profile in `e2e/cmd/e2e/main.go`
  - Added comprehensive documentation to `e2e/README.md`
  - Integrated into CI matrix (`.github/workflows/integration-test-k8s.yml`)
  - Updated Make targets help text (`tools/make/e2e.mk`)

### Out-of-Scope Changes (Justified)

#### 1. Helper Function Fix (`e2e/pkg/helpers/kubernetes.go`)
- **Change**: Added `namespace` parameter to `GetEnvoyServiceName()`
- **Why**: Function was hardcoded to `"envoy-gateway-system"`, preventing reuse for Istio (`"istio-system"`)
- **Impact**: Makes helper function generic and reusable across all profiles
- **Safety**: Updated all 4 call sites (ai-gateway, aibrix, dynamic-config, istio)
- **How AIBrix didn't need it**: AIBrix uses `envoy-gateway-system`, matching the hardcoded value

#### 2. Kubernetes 1.29 Upgrade (`e2e/pkg/cluster/kind.go`)
- **Change**: Explicitly specify Kubernetes 1.29 for Kind clusters
- **Why**: Istio 1.28+ requires Kubernetes 1.29+ ([docs](https://istio.io/latest/docs/releases/supported-releases/))
- **Impact**: Benefits ALL E2E profiles (more stable, newer K8s APIs)
- **Safety**:
  - Shared code path - all profiles use same Kind cluster creation
  - CI matrix tests all profiles (ai-gateway, aibrix, istio) in parallel
  - K8s 1.29 maintains N-2 backward compatibility
  - This is an upgrade (1.27→1.29), not a breaking change
  - CI will catch issues before merge

## Test Coverage

This PR implements the **4 Istio-specific tests** required by issue vllm-project#656:
- ✅ Basic health check with Istio sidecar
- ✅ Traffic routing through Istio gateway
- ✅ mTLS verification
- ✅ Request tracing and observability

**Test Results**: 4/4 passing (100%)

**Note on Signal-Decision Engine Tests**:
Signal-decision engine tests (introduced in PR vllm-project#695) are intentionally excluded
to maintain focus on Istio integration validation. These tests validate semantic
router logic (already covered by ai-gateway, aibrix, and dynamic-config profiles)
rather than Istio mesh functionality. Following the established pattern where
PR vllm-project#695 retroactively added signal-decision tests to existing profiles, these
can be added to the Istio profile in a follow-up PR.

## Implementation Highlights

- 5-stage deployment with comprehensive error handling
- Panic recovery with defer-based cleanup
- Service health verification before tests (prevents 503 errors)
- 60-second stabilization period for mesh readiness
- Complete Istio resource cleanup in teardown
- All Kubernetes resources properly namespaced
- Configurable Istio version for testing flexibility

## Testing Done

✅ All 4 Istio test cases pass successfully
✅ Verified cluster lifecycle management (create/cleanup)
✅ Confirmed Istio sidecar injection and health
✅ Validated traffic routing through Istio gateway
✅ Verified mTLS configuration and certificates
✅ Confirmed distributed tracing headers and metrics
✅ Tested with Istio 1.28.0 and Kubernetes 1.29

## Makefile Targets

```bash
make e2e-test E2E_PROFILE=istio              # Run all Istio tests
make e2e-test E2E_PROFILE=istio E2E_VERBOSE=1 # Run with verbose output
ISTIO_VERSION=1.28.0 make e2e-test E2E_PROFILE=istio  # Custom Istio version
```

Resolves vllm-project#656

Signed-off-by: Asaad Balum <[email protected]>

Author: asaadbalum

.github/workflows/integration-test-k8s.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       fail-fast: false  # Continue testing other profiles even if one fails
       matrix:
-        profile: [ai-gateway, aibrix]
+        profile: [ai-gateway, aibrix, istio]
 
     steps:
       - name: Check out the repo

e2e/README.md

@@ -14,7 +14,7 @@ The framework follows a **separation of concerns** design:
 
 - **ai-gateway**: Tests Semantic Router with Envoy AI Gateway integration
 - **aibrix**: Tests Semantic Router with vLLM AIBrix integration
-- **istio**: Tests Semantic Router with Istio Gateway (future)
+- **istio**: Tests Semantic Router with Istio service mesh integration
 - **production-stack**: Tests vLLM Production Stack configurations (future)
 - **llm-d**: Tests with LLM-D (future)
 - **dynamo**: Tests with Nvidia Dynamo (future)
@@ -517,3 +517,136 @@ func (p *Profile) GetServiceConfig() framework.ServiceConfig {
 ```
 
 See `profiles/ai-gateway/` for a complete example.
+
+## Profile Details
+
+### Istio Profile
+
+The Istio profile tests Semantic Router deployment and functionality in an Istio service mesh environment.
+
+**What it Tests:**
+
+- Istio sidecar injection and health
+- Traffic routing through Istio ingress gateway
+- Mutual TLS (mTLS) between services
+- Distributed tracing and observability
+
+**Prerequisites:**
+
+- `istioctl` must be installed on the system
+- Docker and Kind (managed by E2E framework)
+
+**Components Deployed:**
+
+1. **Istio Control Plane** (`istio-system` namespace):
+   - `istiod` - Istio control plane
+   - `istio-ingressgateway` - Ingress gateway for external traffic
+
+2. **Semantic Router** (`semantic-router` namespace):
+   - Deployed via Helm with Istio sidecar injection enabled
+   - Namespace labeled with `istio-injection=enabled`
+
+3. **Istio Resources**:
+   - `Gateway` - Configures ingress gateway on port 80
+   - `VirtualService` - Routes traffic to Semantic Router service
+   - `DestinationRule` - Enables mTLS with `ISTIO_MUTUAL` mode
+
+**Test Cases:**
+
+| Test Case | Description | What it Validates |
+|-----------|-------------|-------------------|
+| `istio-sidecar-health-check` | Verify Envoy sidecar injection | - Istio-proxy container exists<br>- Sidecar is healthy and ready<br>- Namespace has `istio-injection=enabled` label |
+| `istio-traffic-routing` | Test routing through Istio gateway | - Gateway and VirtualService exist<br>- Requests route correctly to Semantic Router<br>- Istio/Envoy headers present in responses |
+| `istio-mtls-verification` | Verify mutual TLS configuration | - DestinationRule has `ISTIO_MUTUAL` mode<br>- mTLS certificates present in istio-proxy<br>- PeerAuthentication policy (if configured) |
+| `istio-tracing-observability` | Check distributed tracing and metrics | - Trace headers propagated<br>- Envoy metrics exposed<br>- Telemetry configuration<br>- Access logs enabled |
+
+**Usage:**
+
+```bash
+# Run all Istio tests
+make e2e-test E2E_PROFILE=istio
+
+# Run specific Istio tests
+make e2e-test-specific E2E_PROFILE=istio E2E_TESTS="istio-sidecar-health-check,istio-mtls-verification"
+
+# Run with verbose output
+./bin/e2e -profile istio -verbose
+
+# Keep cluster for debugging
+make e2e-test E2E_PROFILE=istio E2E_KEEP_CLUSTER=true
+```
+
+**Architecture:**
+
+```
+┌─────────────────────────────────────────┐
+│         Istio Ingress Gateway            │
+│      (istio-system namespace)            │
+│   Port 80 → semantic-router service      │
+└────────────┬────────────────────────────┘
+             │
+             ▼
+┌─────────────────────────────────────────┐
+│    Semantic Router Pod                   │
+│  (semantic-router namespace)             │
+│  ┌─────────────┐  ┌──────────────────┐  │
+│  │   Main      │  │  Istio-Proxy     │  │
+│  │ Container   │◄─┤  (Envoy Sidecar) │  │
+│  │             │  │                  │  │
+│  │  :8801      │  │  mTLS, Tracing   │  │
+│  └─────────────┘  └──────────────────┘  │
+└─────────────────────────────────────────┘
+             │
+             ▼
+┌─────────────────────────────────────────┐
+│         Istiod (Control Plane)           │
+│  - Config distribution                   │
+│  - Certificate management (mTLS)         │
+│  - Sidecar injection                     │
+└─────────────────────────────────────────┘
+```
+
+**Key Features Tested:**
+
+- ✅ **Automatic Sidecar Injection**: Istio automatically injects Envoy proxy sidecars into pods
+- ✅ **Traffic Management**: Requests route through Istio Gateway → VirtualService → Semantic Router
+- ✅ **Security (mTLS)**: Automatic mutual TLS encryption and authentication between services
+- ✅ **Observability**: Distributed tracing, metrics collection, and access logs
+- ✅ **Service Mesh Integration**: Semantic Router operates correctly within Istio mesh
+
+**Setup Steps (Automated by Profile):**
+
+1. Install Istio control plane using `istioctl install`
+2. Create namespace with `istio-injection=enabled` label
+3. Deploy Semantic Router via Helm (sidecar auto-injected)
+4. Create Istio Gateway and VirtualService for traffic routing
+5. Create DestinationRule for mTLS configuration
+6. Verify all components are ready
+
+**Troubleshooting:**
+
+If tests fail, check:
+
+```bash
+# Check Istio installation
+kubectl get pods -n istio-system
+
+# Check sidecar injection
+kubectl get pods -n semantic-router -o jsonpath='{.items[*].spec.containers[*].name}'
+
+# Check Istio resources
+kubectl get gateway,virtualservice,destinationrule -n semantic-router
+
+# Check mTLS configuration
+kubectl get destinationrule semantic-router -n semantic-router -o yaml
+
+# View Istio proxy logs
+kubectl logs -n semantic-router <pod-name> -c istio-proxy
+```
+
+**Related Resources:**
+
+- [Istio Documentation](https://istio.io/latest/docs/)
+- [Istio Traffic Management](https://istio.io/latest/docs/concepts/traffic-management/)
+- [Istio Security (mTLS)](https://istio.io/latest/docs/concepts/security/)
+- [Istio Observability](https://istio.io/latest/docs/concepts/observability/)

e2e/cmd/e2e/main.go

@@ -12,10 +12,12 @@ import (
 	aigateway "github.com/vllm-project/semantic-router/e2e/profiles/ai-gateway"
 	aibrix "github.com/vllm-project/semantic-router/e2e/profiles/aibrix"
 	dynamicconfig "github.com/vllm-project/semantic-router/e2e/profiles/dynamic-config"
+	istio "github.com/vllm-project/semantic-router/e2e/profiles/istio"
 
 	// Import profiles to register test cases
 	_ "github.com/vllm-project/semantic-router/e2e/profiles/ai-gateway"
 	_ "github.com/vllm-project/semantic-router/e2e/profiles/aibrix"
+	_ "github.com/vllm-project/semantic-router/e2e/profiles/istio"
 )
 
 const version = "v1.0.0"
@@ -103,9 +105,8 @@ func getProfile(name string) (framework.Profile, error) {
 		return dynamicconfig.NewProfile(), nil
 	case "aibrix":
 		return aibrix.NewProfile(), nil
-	// Add more profiles here as they are implemented
-	// case "istio":
-	//     return istio.NewProfile(), nil
+	case "istio":
+		return istio.NewProfile(), nil
 	default:
 		return nil, fmt.Errorf("unknown profile: %s", name)
 	}

e2e/pkg/cluster/kind.go

@@ -38,8 +38,10 @@ func (k *KindCluster) Create(ctx context.Context) error {
 		return nil
 	}
 
-	// Create cluster
-	cmd := exec.CommandContext(ctx, "kind", "create", "cluster", "--name", k.Name)
+	// Create cluster with Kubernetes 1.29 (required by Istio 1.28+)
+	cmd := exec.CommandContext(ctx, "kind", "create", "cluster",
+		"--name", k.Name,
+		"--image", "kindest/node:v1.29.0")
 	if k.Verbose {
 		cmd.Stdout = os.Stdout
 		cmd.Stderr = os.Stderr

e2e/pkg/helpers/kubernetes.go

@@ -36,18 +36,17 @@ func CheckDeployment(ctx context.Context, client *kubernetes.Clientset, namespac
 	return nil
 }
 
-// GetEnvoyServiceName finds the Envoy service name in the envoy-gateway-system namespace
-// using label selectors to match the Gateway-owned service
-func GetEnvoyServiceName(ctx context.Context, client *kubernetes.Clientset, labelSelector string, verbose bool) (string, error) {
-	services, err := client.CoreV1().Services("envoy-gateway-system").List(ctx, metav1.ListOptions{
+// GetEnvoyServiceName finds the service name using label selectors
+func GetEnvoyServiceName(ctx context.Context, client *kubernetes.Clientset, namespace string, labelSelector string, verbose bool) (string, error) {
+	services, err := client.CoreV1().Services(namespace).List(ctx, metav1.ListOptions{
 		LabelSelector: labelSelector,
 	})
 	if err != nil {
 		return "", fmt.Errorf("failed to list services with selector %s: %w", labelSelector, err)
 	}
 
 	if len(services.Items) == 0 {
-		return "", fmt.Errorf("no service found with selector %s in envoy-gateway-system namespace", labelSelector)
+		return "", fmt.Errorf("no service found with selector %s in %s namespace", labelSelector, namespace)
 	}
 
 	// Return the first matching service (should only be one)

e2e/profiles/ai-gateway/profile.go

@@ -260,7 +260,7 @@ func (p *Profile) verifyEnvironment(ctx context.Context, opts *framework.SetupOp
 	var envoyService string
 	for {
 		// Try to get Envoy service name
-		envoyService, err = helpers.GetEnvoyServiceName(ctx, client, labelSelector, p.verbose)
+		envoyService, err = helpers.GetEnvoyServiceName(ctx, client, "envoy-gateway-system", labelSelector, p.verbose)
 		if err == nil {
 			// Verify that the service has exactly 1 pod running with all containers ready
 			podErr := helpers.VerifyServicePodsRunning(ctx, client, "envoy-gateway-system", envoyService, p.verbose)

e2e/profiles/aibrix/profile.go

@@ -333,7 +333,7 @@ func (p *Profile) verifyEnvironment(ctx context.Context, opts *framework.SetupOp
 	var envoyService string
 	for {
 		// Try to get Envoy service name
-		envoyService, err = helpers.GetEnvoyServiceName(ctx, client, labelSelectorAIBrixGateway, p.verbose)
+		envoyService, err = helpers.GetEnvoyServiceName(ctx, client, namespaceEnvoyGateway, labelSelectorAIBrixGateway, p.verbose)
 		if err == nil {
 			// Verify that the service has exactly 1 pod running with all containers ready
 			podErr := helpers.VerifyServicePodsRunning(ctx, client, namespaceEnvoyGateway, envoyService, p.verbose)

e2e/profiles/dynamic-config/profile.go

@@ -280,7 +280,7 @@ func (p *Profile) verifyEnvironment(ctx context.Context, opts *framework.SetupOp
 	var envoyService string
 	for {
 		// Try to get Envoy service name
-		envoyService, err = helpers.GetEnvoyServiceName(ctx, client, labelSelector, p.verbose)
+		envoyService, err = helpers.GetEnvoyServiceName(ctx, client, "envoy-gateway-system", labelSelector, p.verbose)
 		if err == nil {
 			// Verify that the service has exactly 1 pod running with all containers ready
 			podErr := helpers.VerifyServicePodsRunning(ctx, client, "envoy-gateway-system", envoyService, p.verbose)