Skip to content

Commit 54b0ca1

Browse files
committed
feat(router): add LLM routing with cost optimization and pretrained configs
Extend SemanticRouter to support LLM model selection by adding optional model, confidence, cost optimization, and multi-match capabilities to the existing routing infrastructure. When a Route includes a `model` field, the router returns the LiteLLM- compatible model identifier alongside the match, with a confidence score derived from vector distance. Cost-optimized routing biases toward cheaper models when semantic distances are close, using a configurable cost_weight penalty. Key additions to SemanticRouter: - Route.model (optional) for LiteLLM model identifiers - RouteMatch.confidence, .alternatives, .metadata fields - RoutingConfig.cost_optimization and .cost_weight settings - RoutingConfig.default_route for fallback when no match found - from_pretrained() to load routers with pre-computed embeddings - export_with_embeddings() to serialize routers with vectors - AsyncSemanticRouter with full async parity A built-in "default" pretrained config ships with 3 tiers (simple, standard, expert) mapped to GPT-4.1 Nano, Claude Sonnet 4.5, and Claude Opus 4.5, using pre-computed sentence-transformers embeddings. Backward compatibility: - LLMRouter/AsyncLLMRouter provided as deprecated wrappers - ModelTier subclass enforces required model field - Legacy field names (tiers/default_tier) mapped bidirectionally - Existing SemanticRouter usage is fully unaffected Includes integration tests, unit tests for schema validation, a user guide notebook, and a pretrained config generation script.
1 parent 5601ef0 commit 54b0ca1

File tree

14 files changed

+44549
-48
lines changed

14 files changed

+44549
-48
lines changed

docs/user_guide/13_llm_router.ipynb

Lines changed: 706 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,25 @@
1-
from redisvl.extensions.router.schema import Route, RoutingConfig
2-
from redisvl.extensions.router.semantic import SemanticRouter
1+
from redisvl.extensions.router.schema import (
2+
DistanceAggregationMethod,
3+
PretrainedReference,
4+
PretrainedRoute,
5+
PretrainedRouterConfig,
6+
Route,
7+
RouteMatch,
8+
RoutingConfig,
9+
)
10+
from redisvl.extensions.router.semantic import AsyncSemanticRouter, SemanticRouter
311

4-
__all__ = ["SemanticRouter", "Route", "RoutingConfig"]
12+
__all__ = [
13+
# Main router classes
14+
"SemanticRouter",
15+
"AsyncSemanticRouter",
16+
# Schema classes
17+
"Route",
18+
"RouteMatch",
19+
"RoutingConfig",
20+
"DistanceAggregationMethod",
21+
# Pretrained classes
22+
"PretrainedReference",
23+
"PretrainedRoute",
24+
"PretrainedRouterConfig",
25+
]
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
"""Pretrained router configurations.
2+
3+
This module provides pre-built router configurations that can be loaded
4+
without needing to re-embed references.
5+
"""
6+
7+
from pathlib import Path
8+
9+
PRETRAINED_DIR = Path(__file__).parent
10+
11+
12+
def get_pretrained_path(name: str) -> Path:
13+
"""Get path to a pretrained configuration.
14+
15+
Args:
16+
name: Pretrained config name (e.g., "default", "cost_optimized")
17+
18+
Returns:
19+
Path to the pretrained JSON file.
20+
"""
21+
path = PRETRAINED_DIR / f"{name}.json"
22+
if not path.exists():
23+
available = [f.stem for f in PRETRAINED_DIR.glob("*.json")]
24+
raise ValueError(
25+
f"Pretrained config '{name}' not found. " f"Available: {available}"
26+
)
27+
return path
28+
29+
30+
__all__ = ["get_pretrained_path", "PRETRAINED_DIR"]

0 commit comments

Comments
 (0)