Modern Dashboard for vLLM Semantic Router Management

[Xunzhuo]

Is your feature request related to a problem? Please describe.

Currently, vLLM Semantic Router lacks a user-friendly, modern dashboard for managing and operating the system. Users face several challenges:

1. Complex Configuration Management: Editing YAML configuration files (config/config.yaml, SemanticRoute CRDs) manually is error-prone and requires deep technical knowledge
2. Limited Visibility: While Grafana provides excellent metrics visualization, it doesn't offer interactive management capabilities or a unified control plane
3. Steep Learning Curve: New users struggle to understand routing rules, model configurations, filters, and intent-based routing without a visual interface
4. No Interactive Testing: There's no built-in playground to test routing decisions, classification results, or filter chains before deploying to production
5. Operational Complexity: Managing multiple semantic routes, monitoring model performance, and troubleshooting routing issues requires navigating multiple tools and log files

These barriers significantly increase the operational complexity and slow down adoption, especially for teams without deep expertise in semantic routing or LLM infrastructure.

Describe the solution you'd like

A comprehensive, modern web-based dashboard that serves as the central control plane for vLLM Semantic Router, inspired by LiteLLM's dashboard but tailored to semantic routing capabilities. The dashboard should include:

1. Configuration Management UI

• Visual Route Builder: Drag-and-drop interface for creating and editing SemanticRoute configurations
  • Intent definition with category, description, and threshold sliders
  • Model selection with weight and priority configuration
  • Filter chain builder with visual pipeline representation
• Model Registry: Manage vLLM endpoints, model configurations, and reasoning families
• Category Management: Edit categories, system prompts, and model scores with live preview
• Configuration Validation: Real-time validation with helpful error messages and suggestions
• Import/Export: Support for YAML import/export to maintain GitOps workflows

2. Interactive Playground

• Routing Simulator: Test prompts and see which route/model would be selected
  • Display classification scores, intent matching, and routing decision tree
  • Show filter chain execution (PII detection, prompt guard, tool selection)
  • Compare routing decisions across different configurations
• Model Comparison: Side-by-side testing of different models with the same prompt
• Reasoning Mode Testing: Toggle reasoning modes and see the impact on responses
• Cache Hit Visualization: Show semantic cache hits/misses with similarity scores

3. Real-time Monitoring & Observability

• Live Dashboard: Real-time metrics with modern, interactive charts
  • Request rate, latency (TTFT, completion), token usage per model
  • Category classification distribution
  • Routing modifications and model selection patterns
  • Cache hit rates and filter execution statistics
• Request Tracing: Detailed view of individual requests
  • Full routing decision path with timestamps
  • Classification results and confidence scores
  • Filter execution results (PII detected, prompt guard triggers)
  • Model selection reasoning and fallback chains
• Alert Configuration: Visual alert rule builder for Prometheus metrics
• Health Status: Real-time health checks for all vLLM endpoints and models

4. Analytics & Insights

• Cost Analysis: Token usage and estimated costs per model, category, and user
• Performance Benchmarks: Compare model performance across categories
• Routing Effectiveness: Analyze routing decisions and model utilization
• Trend Analysis: Historical data visualization for capacity planning
• A/B Testing Results: Compare different routing strategies with statistical significance

5. User Management & Access Control

• Multi-user Support: Role-based access control (Admin, Operator, Viewer)
• API Key Management: Create, rotate, and revoke API keys with usage tracking
• Team Workspaces: Isolated environments for different teams or projects
• Audit Logs: Complete audit trail of configuration changes and user actions

6. Modern UI/UX Design

• Responsive Design: Works seamlessly on desktop, tablet, and mobile
• Dark/Light Mode: User preference with system theme detection
• Interactive Visualizations: D3.js/Chart.js for beautiful, interactive charts
• Real-time Updates: WebSocket-based live updates without page refresh
• Intuitive Navigation: Clean, modern interface with contextual help and tooltips
• Search & Filters: Quick access to routes, models, and configurations

Technical Implementation Considerations

• Frontend: React/Vue.js with TypeScript, Tailwind CSS for modern styling
• Backend API: RESTful API with WebSocket support for real-time updates
• Authentication: JWT-based auth with SSO support (OAuth2, SAML)
• Database: PostgreSQL for configuration and metadata storage
• Integration: Seamless integration with existing Prometheus/Grafana stack
• Deployment: Docker container with Kubernetes manifests, minimal dependencies

Additional context

Reference Implementations:

• LiteLLM Dashboard: Excellent reference for LLM proxy management UI
• Current Grafana Dashboard (deploy/llm-router-dashboard.json): Strong foundation for metrics visualization
• OpenWebUI Integration (openwebui-filter/vllm_semantic_router_pipe.py): Shows demand for better UI/UX

Key Differentiators from Grafana:

• Interactive Management: Not just monitoring, but full CRUD operations on configurations
• Semantic Routing Specific: Purpose-built for intent-based routing, classification, and filter chains
• Developer Experience: Playground and testing tools to accelerate development
• Operational Simplicity: Reduce time-to-value from hours to minutes

Success Metrics:

• Reduce configuration time from 30+ minutes to <5 minutes
• Enable non-technical users to create and manage routes
• Decrease troubleshooting time with visual request tracing
• Increase adoption rate through improved onboarding experience

Phased Rollout:

1. Phase 1 (MVP): Configuration viewer, basic playground, read-only monitoring
2. Phase 2: Full configuration management, advanced playground, request tracing
3. Phase 3: Analytics, A/B testing, multi-user support, SSO integration

Related Features:

• Existing Grafana dashboard provides metrics foundation
• Prometheus metrics (src/semantic-router/cmd/main.go) already expose necessary data
• OpenAPI/Swagger documentation available at proxy root
• SemanticRoute CRD examples in examples/semanticroute/

This dashboard will dramatically lower the barrier to entry, improve operational efficiency, and make vLLM Semantic Router accessible to a much broader audience.

[Xunzhuo]

/area user-experience observability core docs
/milestone v0.1
/priority P1

[rootfs]

@yuluo-yx @JaredforReal @windsonsea @samzong do you like to give this a try?

[Aias00]

hi, anything i can do?

[rootfs]

@Aias00 welcome!

For starting, please share a plan, describing the admin workflows and create user stories,, and use them to identify what metrics to use, what operations to support, and what the user experience to satisfy.

Along that line, please also survey existing frameworks/products and come up with an analysis for the following:

• should we build a dashboard of our own or can we plug into the existing frameworks/products
• Who/how to maintain the dashboard in the long term
• What interactions can be delegated to models/agent, instead of coding.

[Aias00]

@Aias00 welcome!

For starting, please share a plan, describing the admin workflows and create user stories,, and use them to identify what metrics to use, what operations to support, and what the user experience to satisfy.

Along that line, please also survey existing frameworks/products and come up with an analysis for the following:

• should we build a dashboard of our own or can we plug into the existing frameworks/products
• Who/how to maintain the dashboard in the long term
• What interactions can be delegated to models/agent, instead of coding.

thx, i will think about it and reply later

[JaredforReal]

@rootfs @Xunzhuo I would like to work on this. Please assign this to me! It may need quite a few PRs to get it done. I will create issues to share plans and thoughts.