feat(e2e): Add comprehensive signal-decision engine test coverage #695

yehudit1987 · 2025-11-19T13:13:55Z

Implements 6 new E2E test suites to validate the signal-decision
engine :

New Test Files:

decision_priority.go: Priority-based routing (4 cases)
plugin_chain_execution.go: Plugin ordering & blocking (4 cases)
rule_condition_logic.go: AND/OR operators (6 cases)
decision_fallback.go: Fallback behavior (5 cases)
keyword_routing.go: Keyword matching (6 cases)
plugin_config_variations.go: Plugin configs (6 cases)

Key Fixes:

Fix keyword_rules field: 'category' → 'name'
Fix rule condition field: 'rule_name' → 'name'
Add keyword conditions to thinking_decision
Standardize fallback decision name: 'general_decision' → 'other_decision'

Additional Changes:

Add signal-decision tests to all 3 profiles (ai-gateway, aibrix, dynamic-config)
Fix markdown formatting in e2e/README.md and website/docs/api/crd-reference.md

Resolve #692

netlify · 2025-11-19T13:14:01Z

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	`0ba3985`
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/691efd92eab26f00084c1a36
😎 Deploy Preview	https://deploy-preview-695--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

github-actions · 2025-11-19T13:14:08Z

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 `e2e`

Owners: @Xunzhuo
Files changed:

e2e/testcases/decision_fallback.go
e2e/testcases/decision_priority.go
e2e/testcases/keyword_routing.go
e2e/testcases/plugin_chain_execution.go
e2e/testcases/plugin_config_variations.go
e2e/testcases/rule_condition_logic.go
e2e/README.md
e2e/profiles/ai-gateway/profile.go
e2e/profiles/ai-gateway/values.yaml
e2e/profiles/aibrix/profile.go
e2e/profiles/dynamic-config/crds/intelligentroute.yaml
e2e/profiles/dynamic-config/profile.go
e2e/testcases/common.go

📁 `deploy`

Owners: @rootfs, @Xunzhuo
Files changed:

deploy/kubernetes/aibrix/semantic-router-values/values.yaml

📁 `website`

Owners: @Xunzhuo, @rootfs, @yuluo-yx
Files changed:

website/docs/api/crd-reference.md

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

Xunzhuo

hopefully you can catch some bugs after adding the testcases, happy hacking! thanks

Xunzhuo · 2025-11-20T01:26:58Z

We need to add this to dynamic config profile as well, plz check the relevant files

…cture Signed-off-by: Yehudit Kerido <[email protected]>

e2e/profiles/dynamic-config/crds/intelligentroute.yaml

Signed-off-by: Yehudit Kerido <[email protected]>

yehudit1987 · 2025-11-20T12:08:33Z

Hi @Xunzhuo.
I've added 6 new signal-decision engine tests across all 3 profiles. While implementing them, I discovered some (probably) backend issues:

Test Accuracy:

• ✅ decision-priority-selection: 100% (working perfectly)
• ⚠️ plugin-chain-execution: 75% (US_SSN detection works, EMAIL_ADDRESS doesn't)
• 🔴 keyword-routing: 17% (only "ASAP" keyword works)
• 🔴 rule-condition-logic: 33% (keywords work sporadically)
• 🔴 decision-fallback-behavior: 40% (classifier unreliable)
• ⚠️ plugin-config-variations: 50% (same classifier issues)

Quick Summary of Issues:

1. PII: EMAIL_ADDRESS not blocking (returns 200 instead of 403), but US_SSN works fine
2. Keywords: Only "ASAP" matches consistently, others ("urgent", "think", "careful") fail
3. Domain Classifier: Returns empty/wrong classifications for obvious queries

I hope I have verified correctly the test implementations (proper headers, config, model parameter). I suggest the issues are backend bugs, but feel free to double check this conclusion. Should I open separate issues for these backend bugs, or would you prefer to fix them before merging? Tests currently "pass" in CI (0% threshold) so they won't block anything. Anyway I will be happy to take part in those future investigations.

Xunzhuo · 2025-11-20T13:04:53Z

Thanks!

Xunzhuo · 2025-11-20T15:11:37Z

@yehudit1987 would you like to try adding test cases around embedding signals?

Xunzhuo · 2025-11-20T15:13:38Z

And plz create issues for your findings, if u can dig more that will be helpful

yehudit1987 · 2025-11-20T15:35:54Z

And plz create issues for your findings, if u can dig more that will be helpful

I will, please leave it for me. Will create those next week and ask for your approval.

This PR implements the Istio profile for the E2E testing framework as requested in issue vllm-project#656. ## What Changed ### Core Implementation (Issue vllm-project#656 Requirements) - **New Istio Profile** (`e2e/profiles/istio/`) - Implements 5-stage deployment: Istio control plane, namespace configuration, Semantic Router with sidecar injection, Istio Gateway/VirtualService/DestinationRule, and environment verification - Robust error handling with panic recovery and defer-based cleanup - Configurable Istio version via `ISTIO_VERSION` environment variable (default: 1.28.0) - Comprehensive service health verification with 60-second stabilization period - **4 Istio-Specific Test Cases** (100% passing) 1. `istio-sidecar-health-check` - Verify Envoy sidecar injection and health 2. `istio-traffic-routing` - Test routing through Istio ingress gateway 3. `istio-mtls-verification` - Verify mutual TLS configuration and certificates 4. `istio-tracing-observability` - Validate distributed tracing and metrics - **Integration & Documentation** - Registered profile in `e2e/cmd/e2e/main.go` - Added comprehensive documentation to `e2e/README.md` - Integrated into CI matrix (`.github/workflows/integration-test-k8s.yml`) - Updated Make targets help text (`tools/make/e2e.mk`) ### Out-of-Scope Changes (Justified) #### 1. Helper Function Fix (`e2e/pkg/helpers/kubernetes.go`) - **Change**: Added `namespace` parameter to `GetEnvoyServiceName()` - **Why**: Function was hardcoded to `"envoy-gateway-system"`, preventing reuse for Istio (`"istio-system"`) - **Impact**: Makes helper function generic and reusable across all profiles - **Safety**: Updated all 4 call sites (ai-gateway, aibrix, dynamic-config, istio) - **How AIBrix didn't need it**: AIBrix uses `envoy-gateway-system`, matching the hardcoded value #### 2. Kubernetes 1.29 Upgrade (`e2e/pkg/cluster/kind.go`) - **Change**: Explicitly specify Kubernetes 1.29 for Kind clusters - **Why**: Istio 1.28+ requires Kubernetes 1.29+ ([docs](https://istio.io/latest/docs/releases/supported-releases/)) - **Impact**: Benefits ALL E2E profiles (more stable, newer K8s APIs) - **Safety**: - Shared code path - all profiles use same Kind cluster creation - CI matrix tests all profiles (ai-gateway, aibrix, istio) in parallel - K8s 1.29 maintains N-2 backward compatibility - This is an upgrade (1.27→1.29), not a breaking change - CI will catch issues before merge ## Test Coverage This PR implements the **4 Istio-specific tests** required by issue vllm-project#656: - ✅ Basic health check with Istio sidecar - ✅ Traffic routing through Istio gateway - ✅ mTLS verification - ✅ Request tracing and observability **Test Results**: 4/4 passing (100%) **Note on Signal-Decision Engine Tests**: Signal-decision engine tests (introduced in PR vllm-project#695) are intentionally excluded to maintain focus on Istio integration validation. These tests validate semantic router logic (already covered by ai-gateway, aibrix, and dynamic-config profiles) rather than Istio mesh functionality. Following the established pattern where PR vllm-project#695 retroactively added signal-decision tests to existing profiles, these can be added to the Istio profile in a follow-up PR. ## Implementation Highlights - 5-stage deployment with comprehensive error handling - Panic recovery with defer-based cleanup - Service health verification before tests (prevents 503 errors) - 60-second stabilization period for mesh readiness - Complete Istio resource cleanup in teardown - All Kubernetes resources properly namespaced - Configurable Istio version for testing flexibility ## Testing Done ✅ All 4 Istio test cases pass successfully ✅ Verified cluster lifecycle management (create/cleanup) ✅ Confirmed Istio sidecar injection and health ✅ Validated traffic routing through Istio gateway ✅ Verified mTLS configuration and certificates ✅ Confirmed distributed tracing headers and metrics ✅ Tested with Istio 1.28.0 and Kubernetes 1.29 ## Makefile Targets ```bash make e2e-test E2E_PROFILE=istio # Run all Istio tests make e2e-test E2E_PROFILE=istio E2E_VERBOSE=1 # Run with verbose output ISTIO_VERSION=1.28.0 make e2e-test E2E_PROFILE=istio # Custom Istio version ``` Resolves vllm-project#656 Signed-off-by: Asaad Balum <[email protected]>

This PR implements the Istio profile for the E2E testing framework as requested in issue vllm-project#656. ## What Changed ### Core Implementation (Issue vllm-project#656 Requirements) - **New Istio Profile** (`e2e/profiles/istio/`) - Implements 5-stage deployment: Istio control plane, namespace configuration, Semantic Router with sidecar injection, Istio Gateway/VirtualService/DestinationRule, and environment verification - Robust error handling with panic recovery and defer-based cleanup - Configurable Istio version via `ISTIO_VERSION` environment variable (default: 1.28.0) - Comprehensive service health verification with 60-second stabilization period - **4 Istio-Specific Test Cases** (100% passing) 1. `istio-sidecar-health-check` - Verify Envoy sidecar injection and health 2. `istio-traffic-routing` - Test routing through Istio ingress gateway 3. `istio-mtls-verification` - Verify mutual TLS configuration and certificates 4. `istio-tracing-observability` - Validate distributed tracing and metrics - **Integration & Documentation** - Registered profile in `e2e/cmd/e2e/main.go` - Added comprehensive documentation to `e2e/README.md` - Integrated into CI matrix (`.github/workflows/integration-test-k8s.yml`) - Updated Make targets help text (`tools/make/e2e.mk`) ### Out-of-Scope Changes (Justified) #### Helper Function Enhancement (`e2e/pkg/helpers/kubernetes.go`) - **Change**: Added `GetServiceByLabelInNamespace()` with backward-compatible wrapper - **Why**: Original `GetEnvoyServiceName()` was hardcoded to `"envoy-gateway-system"`, preventing reuse for Istio which uses `"istio-system"` namespace - **Approach**: Backward-compatible wrapper pattern - Old function `GetEnvoyServiceName(ctx, client, labelSelector, verbose)` still works - Calls new generic function `GetServiceByLabelInNamespace()` with hardcoded namespace - Istio uses the new flexible function directly - **Impact**: - **ZERO changes** to existing profiles (ai-gateway, aibrix, dynamic-config) - Makes helper function generic and reusable for future profiles - Marked old function as deprecated for future cleanup - **Safety**: Existing profiles continue to work unchanged, Istio gets flexibility it needs ## Test Coverage This PR implements the **4 Istio-specific tests** required by issue vllm-project#656: - ✅ Basic health check with Istio sidecar - ✅ Traffic routing through Istio gateway - ✅ mTLS verification - ✅ Request tracing and observability **Test Results**: 4/4 passing (100%) **Note on Signal-Decision Engine Tests**: Signal-decision engine tests (introduced in PR vllm-project#695) are intentionally excluded to maintain focus on Istio integration validation. These tests validate semantic router logic (already covered by ai-gateway, aibrix, and dynamic-config profiles) rather than Istio mesh functionality. Following the established pattern where PR vllm-project#695 retroactively added signal-decision tests to existing profiles, these can be added to the Istio profile in a follow-up PR. **Note on Kubernetes Version**: CI uses Kind v0.22.0 which defaults to Kubernetes 1.29.2, meeting Istio 1.28+ compatibility requirements without requiring explicit version pinning. ## Implementation Highlights - 5-stage deployment with comprehensive error handling - Panic recovery with defer-based cleanup - Service health verification before tests (prevents 503 errors) - 60-second stabilization period for mesh readiness - Complete Istio resource cleanup in teardown - All Kubernetes resources properly namespaced - Configurable Istio version for testing flexibility ## Testing Done ✅ All 4 Istio test cases pass successfully ✅ Verified cluster lifecycle management (create/cleanup) ✅ Confirmed Istio sidecar injection and health ✅ Validated traffic routing through Istio gateway ✅ Verified mTLS configuration and certificates ✅ Confirmed distributed tracing headers and metrics ✅ Tested with Istio 1.28.0 and CI's default Kubernetes version ## Makefile Targets ```bash make e2e-test E2E_PROFILE=istio # Run all Istio tests make e2e-test E2E_PROFILE=istio E2E_VERBOSE=1 # Run with verbose output ISTIO_VERSION=1.28.0 make e2e-test E2E_PROFILE=istio # Custom Istio version ``` Resolves vllm-project#656 Signed-off-by: Asaad Balum <[email protected]>

github-actions bot assigned rootfs and Xunzhuo Nov 19, 2025

yehudit1987 force-pushed the test_coverage_sd branch 3 times, most recently from be04d47 to bb8d3d2 Compare November 19, 2025 14:43

Xunzhuo reviewed Nov 19, 2025

View reviewed changes

yehudit1987 force-pushed the test_coverage_sd branch 2 times, most recently from 674043b to bc5f724 Compare November 19, 2025 18:10

yehudit1987 force-pushed the test_coverage_sd branch from bc5f724 to 3f607e3 Compare November 20, 2025 06:39

[E2E] Add Signal-Decision Engine Test Coverage for New Plugin Archite…

03b5f9c

…cture Signed-off-by: Yehudit Kerido <[email protected]>

yehudit1987 force-pushed the test_coverage_sd branch from 3f607e3 to 03b5f9c Compare November 20, 2025 07:38

Xunzhuo reviewed Nov 20, 2025

View reviewed changes

e2e/profiles/dynamic-config/crds/intelligentroute.yaml Outdated Show resolved Hide resolved

fix wrong header in test

0ba3985

Signed-off-by: Yehudit Kerido <[email protected]>

yehudit1987 force-pushed the test_coverage_sd branch from c7d133f to 0ba3985 Compare November 20, 2025 11:37

yehudit1987 marked this pull request as ready for review November 20, 2025 12:07

yehudit1987 requested a review from rootfs as a code owner November 20, 2025 12:07

Xunzhuo approved these changes Nov 20, 2025

View reviewed changes

Xunzhuo merged commit a149800 into vllm-project:main Nov 20, 2025
19 checks passed

yehudit1987 deleted the test_coverage_sd branch November 23, 2025 06:12

asaadbalum mentioned this pull request Nov 24, 2025

feat: Add istio profile for E2E testing framework #728

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(e2e): Add comprehensive signal-decision engine test coverage #695

feat(e2e): Add comprehensive signal-decision engine test coverage #695

Uh oh!

yehudit1987 commented Nov 19, 2025 •

edited

Loading

Uh oh!

netlify bot commented Nov 19, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 19, 2025 •

edited

Loading

Uh oh!

Xunzhuo left a comment

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

Uh oh!

yehudit1987 commented Nov 20, 2025 •

edited

Loading

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

yehudit1987 commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(e2e): Add comprehensive signal-decision engine test coverage #695

feat(e2e): Add comprehensive signal-decision engine test coverage #695

Uh oh!

Conversation

yehudit1987 commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vllm-semantic-router ready!

Uh oh!

github-actions bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👥 vLLM Semantic Team Notification

📁 e2e

📁 deploy

📁 website

🎉 Thanks for your contributions!

Uh oh!

Xunzhuo left a comment

Choose a reason for hiding this comment

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

Uh oh!

yehudit1987 commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Accuracy:

Quick Summary of Issues:

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

Xunzhuo commented Nov 20, 2025

Uh oh!

yehudit1987 commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yehudit1987 commented Nov 19, 2025 •

edited

Loading

netlify bot commented Nov 19, 2025 •

edited

Loading

github-actions bot commented Nov 19, 2025 •

edited

Loading

📁 `e2e`

📁 `deploy`

📁 `website`

yehudit1987 commented Nov 20, 2025 •

edited

Loading