Skip to content

Conversation

@Xunzhuo
Copy link
Member

@Xunzhuo Xunzhuo commented Nov 17, 2025

Description

This PR adds error handling to domain classification, PII detection, and jailbreak detection e2e tests to ensure they fail explicitly when accuracy is 0%.

Motivation

Previously, these tests would return nil (success) even when all test cases failed, which could mask critical issues in the routing and detection logic. This change makes test failures more visible and actionable by returning a descriptive error when correctTests == 0.

Changes

  • Added zero-accuracy check in testDomainClassify() function
  • Added zero-accuracy check in testPIIDetection() function
  • Added zero-accuracy check in testJailbreakDetection() function
  • All three tests now return an error with format: "<test-name> test failed: 0% accuracy (0/<total> correct)"

Testing

The changes ensure that:

  • Tests fail explicitly when accuracy is 0%
  • Error messages clearly indicate which test failed and the total number of test cases
  • Existing behavior is preserved when accuracy > 0%

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE


  • Make sure the code changes pass the pre-commit checks.
  • Sign-off your commit by using -s when doing git commit
  • Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

Pull Request opened by Augment Code with guidance from the PR author

Add error handling to domain classification, PII detection, and jailbreak
detection tests to ensure they fail explicitly when accuracy is 0%.

Previously, these tests would return nil (success) even when all test cases
failed, which could mask critical issues. Now they return a descriptive error
when correctTests == 0, making test failures more visible and actionable.

Signed-off-by: bitliu <[email protected]>
@github-actions
Copy link

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 e2e

Owners: @Xunzhuo
Files changed:

  • e2e/testcases/domain_classify.go
  • e2e/testcases/jailbreak_detection.go
  • e2e/testcases/pii_detection.go

vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

@netlify
Copy link

netlify bot commented Nov 17, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit 9d97da1
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/691a88e016a32c00087fd048
😎 Deploy Preview https://deploy-preview-676--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@wangchen615 wangchen615 self-requested a review November 17, 2025 03:03
Copy link
Collaborator

@wangchen615 wangchen615 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangchen615 wangchen615 merged commit 84b6fd7 into main Nov 17, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants