Skip to content

Conversation

@yossiovadia
Copy link
Collaborator

Summary

This PR fixes the PII API endpoint to return actual confidence scores from the LoRA model instead of hardcoded 0.9 values.

Fixes #717

Changes

  1. Added ClassifyPIIWithDetails() method - New classifier method that returns full entity details including actual confidence scores
  2. Updated DetectPII() service - Modified to use ClassifyPIIWithDetails() instead of ClassifyPII() to access real confidence values
  3. Updated config.e2e.yaml - Configured to use LoRA PII model and added default catch-all decision for E2E testing

Before vs After

Before (Hardcoded)

{
  "entities": [
    {"type": "B-EMAIL_ADDRESS", "confidence": 0.9},
    {"type": "B-PERSON", "confidence": 0.9}
  ]
}

After (Actual Scores)

{
  "entities": [
    {"type": "B-EMAIL_ADDRESS", "confidence": 0.9869},
    {"type": "B-PERSON", "confidence": 0.9991}
  ]
}

Test Results

Local testing shows the API now correctly returns actual model confidence scores:

Test Case Before After
Email detection 0.9 (hardcoded) 0.9869 (actual)
SSN detection 0.9 (hardcoded) 0.9916, 0.8326 (actual)
Multiple PII 0.9 (hardcoded) 0.7734-0.9991 range (actual)

Technical Details

The underlying LoRA PII model was already producing accurate confidence scores (fixed in #709), but the API layer was discarding them. This change passes through the actual scores to API consumers.

The new ClassifyPIIWithDetails() method returns []PIIDetection with full entity information, while maintaining backward compatibility by keeping the existing ClassifyPII() method for callers that only need entity types.

Testing

  • Local testing with make run-router-e2e
  • Manual API testing with curl
  • Pre-commit checks passed
  • DCO sign-off added

Fixes vllm-project#717

Changes:
- Add ClassifyPIIWithDetails() method to classifier that returns full
  entity details including actual confidence scores
- Update DetectPII() service to use ClassifyPIIWithDetails() instead
  of ClassifyPII() to access real confidence values
- Update config.e2e.yaml to use LoRA PII model and add default
  catch-all decision for E2E testing

Before: API returned hardcoded confidence=0.9 for all PII entities
After: API returns actual model confidence scores (0.77-0.99 range)

Test results:
- Email detection: 0.9869 (was 0.9)
- SSN detection: 0.9916, 0.8326 (was 0.9)
- Multiple PII: 0.7734-0.9991 range (was all 0.9)

The underlying LoRA PII model was already producing accurate confidence
scores (fixed in vllm-project#709), but the API layer was discarding them. This
change passes through the actual scores to API consumers.

Signed-off-by: Yossi Ovadia <[email protected]>
@netlify
Copy link

netlify bot commented Nov 22, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit f63cf1e
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/692117dc83299f0008e4c278
😎 Deploy Preview https://deploy-preview-718--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions
Copy link

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 config

Owners: @rootfs, @Xunzhuo
Files changed:

  • config/testing/config.e2e.yaml

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

  • src/semantic-router/pkg/classification/classifier.go
  • src/semantic-router/pkg/services/classification.go

vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

@github-actions github-actions bot deleted a comment from codecov-commenter Nov 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PII API endpoint hardcodes confidence to 0.9 instead of using actual model scores

4 participants