Allow jailbreak detection and threshold to be configured at the category level

[Copilot]

Overview

This PR implements category-level jailbreak detection configuration, allowing administrators to enable/disable jailbreak protection and customize detection thresholds on a per-category basis. Previously, jailbreak detection was a global setting that applied uniformly to all requests. With this change, different categories can have different security policies based on their specific risk profiles and use cases.

Problem Statement

The global prompt_guard.enabled and prompt_guard.threshold settings applied jailbreak detection uniformly across all categories. However, different categories have different security requirements:

• Public-facing categories (e.g., customer support, business advice) need strict jailbreak detection with high thresholds
• Technical categories (e.g., code generation) may need relaxed thresholds to avoid false positives on code/technical jargon
• Internal tools may need it disabled entirely for trusted environments

Without category-level control, administrators had to choose between:

1. Enabling globally with one threshold (potentially blocking legitimate requests or missing threats)
2. Disabling globally (leaving endpoints vulnerable)

Solution

New Configuration Fields

Added jailbreak_enabled and jailbreak_threshold as optional fields in category configuration:

prompt_guard:
  enabled: true       # Global default
  threshold: 0.7      # Global default threshold

categories:
  - name: customer_support
    jailbreak_enabled: true      # Explicitly enable for public-facing
    jailbreak_threshold: 0.9     # High threshold for strict detection
    model_scores:
      - model: qwen3
        score: 0.8

  - name: code_generation
    jailbreak_enabled: true      # Keep enabled
    jailbreak_threshold: 0.5     # Lower threshold to reduce false positives
    model_scores:
      - model: qwen3
        score: 0.9

  - name: general
    # No jailbreak_enabled or jailbreak_threshold specified
    # Inherits global settings (enabled: true, threshold: 0.7)
    model_scores:
      - model: qwen3
        score: 0.5

Behavior

• When jailbreak_enabled is not specified: Category inherits from global prompt_guard.enabled
• When jailbreak_enabled: true/false: Jailbreak detection is explicitly enabled/disabled for this category
• When jailbreak_threshold is not specified: Category inherits from global prompt_guard.threshold
• When jailbreak_threshold: 0.X: Uses category-specific threshold (0.0-1.0)
• Category-specific settings always override global settings when explicitly configured

Threshold Tuning Guidelines

• High threshold (0.8-0.95): Stricter detection, fewer false positives, for high-security categories
• Medium threshold (0.6-0.8): Balanced detection, good for most use cases
• Low threshold (0.4-0.6): More sensitive, catches more attacks, for technical categories with higher false positive tolerance

Implementation Details

1. Configuration Structure (pkg/config/config.go):

   • Added JailbreakEnabled *bool field to Category struct
   • Added JailbreakThreshold *float32 field to Category struct
   • Implemented IsJailbreakEnabledForCategory(categoryName string) bool method
   • Implemented GetJailbreakThresholdForCategory(categoryName string) float32 method
   • Uses pointer types to distinguish between "not set" (nil) and explicitly set values

2. Request Processing (pkg/extproc/request_handler.go):

   • Moved category classification before security checks to enable category-aware detection
   • Updated performSecurityChecks() to accept category name and use category-specific settings
   • Retrieves both enabled status and threshold based on category
   • Maintains backward compatibility when category cannot be determined

3. Classifier (pkg/utils/classification/classifier.go):

   • Added CheckForJailbreakWithThreshold() method that accepts custom threshold
   • Added AnalyzeContentForJailbreakWithThreshold() for batch analysis with custom threshold
   • Original methods delegate to new threshold-aware methods for backward compatibility

4. Testing (pkg/config/config_test.go):

   • Added 7 test cases for jailbreak_enabled configuration
   • Added 5 test cases for jailbreak_threshold configuration
   • Tests cover inheritance behavior, explicit overrides, and edge cases
   • All 132 tests pass

Documentation

• Created config/examples/jailbreak_category_example.yaml with comprehensive examples and threshold tuning guidance
• Updated website/docs/tutorials/content-safety/jailbreak-protection.md with category-level configuration
• Updated website/docs/overview/categories/configuration.md with parameter documentation and examples
• Updated website/docs/installation/configuration.md with inline comments
• Updated config/config.yaml with inline comments explaining the feature
• Updated README.md to mention fine-grained security control

Security Considerations

• Default behavior unchanged: Existing configurations continue to work without modification
• No silent failures: CodeQL analysis shows 0 security alerts
• Best practice: Global jailbreak should be enabled by default, with category-specific overrides only where risk is managed differently
• Explicit control: Administrators must explicitly configure per-category settings
• Threshold monitoring: Documentation includes guidance on monitoring false positive/negative rates

Testing

• ✅ All 132 config tests pass (12 new tests added)
• ✅ New tests cover all inheritance and override scenarios for both enabled status and threshold
• ✅ CodeQL security analysis: 0 alerts
• ✅ Backward compatibility verified
• ✅ YAML syntax validated
• ✅ Go packages compile successfully

Migration Guide

No migration needed - this is a backward-compatible addition. Existing configurations will continue to work exactly as before. To use category-level jailbreak detection:

1. Set your global defaults: prompt_guard.enabled: true/false and prompt_guard.threshold: 0.0-1.0
2. Override per category as needed: jailbreak_enabled: true/false and/or jailbreak_threshold: 0.0-1.0
3. Omit category-level settings to inherit the global defaults

Related Issues

Fixes #507

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Allow jailbreak to be set at the category level</issue_title>
<issue_description>Is your feature request related to a problem? Please describe.

Currently, jailbreak prompt guard is based off a global setting for doing jailbreak detection.

Given the possible consequences and specifics for different categories, this should really be set at the category example.

Please allow setting jailbreak on/off at the category level.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #507

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	ea50e6c
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/68f88c6e7dee4400087dd50b
😎 Deploy Preview	https://deploy-preview-508--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 config

Owners: @rootfs
Files changed:

• config/examples/jailbreak_category_example.yaml
• config/config.yaml

📁 Root Directory

Owners: @rootfs, @Xunzhuo
Files changed:

• README.md

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

• src/semantic-router/pkg/config/config.go
• src/semantic-router/pkg/config/config_test.go
• src/semantic-router/pkg/extproc/request_handler.go
• src/semantic-router/pkg/utils/classification/classifier.go
• src/training/training_lora/classifier_model_fine_tuning_lora/go.mod
• src/training/training_lora/pii_model_fine_tuning_lora/go.mod
• src/training/training_lora/prompt_guard_fine_tuning_lora/go.mod

📁 website

Owners: @Xunzhuo, @rootfs, @yuluo-yx
Files changed:

• website/docs/installation/configuration.md
• website/docs/overview/categories/configuration.md
• website/docs/tutorials/content-safety/jailbreak-protection.md

---

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

[Xunzhuo]

@copilot good to go

[Xunzhuo]

@copilot please update relevant docs in site

[Copilot]

@copilot please update relevant docs in site

Updated documentation in commit f75a8c1:

• Added category-level jailbreak configuration section to jailbreak-protection.md
• Updated category configuration docs with jailbreak_enabled parameter explanation
• Added security-focused configuration example showing public-facing vs internal tool settings
• Updated global configuration docs with category override notes
• Updated README to mention fine-grained security control

[Xunzhuo]

@copilot threshold should be configurable in category level (override)

[Copilot]

@copilot threshold should be configurable in category level (override)

Implemented category-level threshold configuration in commit ea50e6c:

• Added jailbreak_threshold field to Category struct (0.0-1.0)
• Categories can now override global threshold per category
• Added helper method GetJailbreakThresholdForCategory
• Updated jailbreak detection to use category-specific thresholds
• Added 5 comprehensive tests for threshold configuration
• Updated documentation with threshold tuning guidelines:
  • High-security (0.8-0.95): Customer support, public-facing
  • Standard (0.6-0.8): General queries
  • Technical (0.4-0.6): Code generation, dev tools
• Example configs show threshold tuning for different risk profiles

[Xunzhuo]

@copilot it is ready for review and merge