Merge main to candle refactoring (#523)

* Update test description from Math to General (#483)

Signed-off-by: carlory <[email protected]>

* feat: add HuggingChat support (#477)

* add chat ui to dashboard and docker compose & refactor dashboard/backend/

Signed-off-by: JaredforReal <[email protected]>

* try fix network error

Signed-off-by: JaredforReal <[email protected]>

* more

---------

Signed-off-by: JaredforReal <[email protected]>
Co-authored-by: bitliu <[email protected]>

* project: 2025 Q4 roadmap (#487)

* project: q4 roadmap

* project: q4 roadmap

* project: q4 roadmap

* more

* more

* more

* more

* feat: add shelleck precommit hook (#488)

* feat: add shelleck precommit hook

Signed-off-by: yuluo-yx <[email protected]>

* feat: add shelleck precommit hook

Signed-off-by: yuluo-yx <[email protected]>

* feat: add shelleck precommit hook

Signed-off-by: yuluo-yx <[email protected]>

---------

Signed-off-by: yuluo-yx <[email protected]>

* project: add q4 roadmap news (#495)

* fix missing shellcheck in pre-commit image (#497)

Signed-off-by: carlory <[email protected]>

* infra: update tools (#501)

Signed-off-by: yuluo-yx <[email protected]>

* feat(demo): enhance OpenShift demo scripts with improved UX (#478)

- Reduce model selection test to 4 categories (2×Model-A, 2×Model-B)
- Add new "Classification Examples" option calling curl-examples.sh
- Update reasoning examples to avoid cache hits from previous tests
- Remove benign examples from PII and Jailbreak tests (show only attacks)
- Enhance live-semantic-router-logs.sh with better color visibility:
  - Fix duplicate "WITH SCORE" text in classification output
  - Fix CACHE HIT background color extending over timestamp
  - Distinguish reasoning enabled vs disabled messages
  - Remove redundant "(standard routing)" text
  - Add background colors for Model-A/Model-B routing display

These improvements make the live demo clearer and more impactful for
presentations and demonstrations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Yossi Ovadia <[email protected]>
Co-authored-by: Claude <[email protected]>

* fix: fix precommit Argument list too long error (#502)

Signed-off-by: yuluo-yx <[email protected]>

* feat: enforce milvus dial timeout if set (#503)

Signed-off-by: cryo <[email protected]>

* Add IETF draft publication: Multi-Provider Extensions for Agentic AI Inference APIs (#506)

* Initial plan

* Add new IETF draft publication for Multi-Provider Extensions for Agentic AI Inference APIs

Co-authored-by: rootfs <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: rootfs <[email protected]>

* Allow semantic cache similarity threshold to be set at the category level (#493)

* Initial plan

* Add category-level cache settings: enabled and similarity_threshold

Co-authored-by: rootfs <[email protected]>

* Add comprehensive tests for category-level cache settings

Co-authored-by: rootfs <[email protected]>

* Update config files and documentation for category-level cache settings

- Updated 7 config YAML files (development, production, testing, e2e, and 3 recipes) with commented examples of category-level cache settings
- Added comprehensive documentation section explaining category-level cache configuration
- Updated semantic cache overview and in-memory cache docs with category-level examples
- Added best practices for threshold selection and privacy considerations

Co-authored-by: rootfs <[email protected]>

* Remove duplicate code in FindSimilar functions

Refactored FindSimilar() to delegate to FindSimilarWithThreshold() with default threshold instead of duplicating the entire implementation. This eliminates 226 lines of duplicate code across inmemory_cache.go and milvus_cache.go.

Co-authored-by: rootfs <[email protected]>

* Update src/semantic-router/pkg/extproc/request_handler.go

Co-authored-by: Copilot <[email protected]>

* Revert changes from unsigned commit ae39fe2

Restored the classificationText empty check that was removed in the previous commit.

Co-authored-by: rootfs <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: rootfs <[email protected]>
Co-authored-by: Huamin Chen <[email protected]>
Co-authored-by: Copilot <[email protected]>

* Allow jailbreak detection and threshold to be configured at the category level (#508)

* Initial plan

* Add category-level jailbreak detection configuration

Co-authored-by: Xunzhuo <[email protected]>

* Add documentation for category-level jailbreak settings

Co-authored-by: Xunzhuo <[email protected]>

* Update documentation for category-level jailbreak detection

- Add category-level jailbreak configuration to jailbreak-protection.md
- Update category configuration docs with jailbreak_enabled parameter
- Add security-focused configuration example
- Update global configuration docs with category override notes
- Update README to mention fine-grained security control

Co-authored-by: Xunzhuo <[email protected]>

* Add category-level jailbreak threshold configuration

- Add JailbreakThreshold field to Category struct
- Add GetJailbreakThresholdForCategory helper method
- Create CheckForJailbreakWithThreshold and AnalyzeContentForJailbreakWithThreshold methods
- Update performSecurityChecks to use category-specific threshold
- Add 5 comprehensive tests for threshold configuration
- Update example configs with threshold tuning examples
- Update documentation with threshold configuration and tuning guidelines
- Add threshold tuning guide with recommendations for different category types

Co-authored-by: Xunzhuo <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: Xunzhuo <[email protected]>

* Allow PII detection threshold to be set at the category level (#510)

* Initial plan

* Add category-level PII threshold support

Co-authored-by: Xunzhuo <[email protected]>

* Update documentation with API integration notes

Co-authored-by: Xunzhuo <[email protected]>

* Fix markdown linting issues

Co-authored-by: Xunzhuo <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: Xunzhuo <[email protected]>

* Fix: The caller information points to the wrapper function instead of the actual call location (#518)

Signed-off-by: carlory <[email protected]>

* feat: Implement hybrid cache that use in-memory index and milvus based doc store (#504)

* feat: add HNSW index to inmemory semantic cache and implement hybrid cache that use in-memory index and milvus based doc store

Signed-off-by: Huamin Chen <[email protected]>

* chore: run go mod tidy to clean up module dependencies

Signed-off-by: Huamin Chen <[email protected]>

* conditionally build candle cuda support

Signed-off-by: Huamin Chen <[email protected]>

* rebuild index upon restart

Signed-off-by: Huamin Chen <[email protected]>

* precommit fix

Signed-off-by: Huamin Chen <[email protected]>

* fix precommit

Signed-off-by: Huamin Chen <[email protected]>

* fix precommit

Signed-off-by: Huamin Chen <[email protected]>

* fix precommit

Signed-off-by: Huamin Chen <[email protected]>

* disable cuda build on ci

Signed-off-by: Huamin Chen <[email protected]>

* review feedback

Signed-off-by: Huamin Chen <[email protected]>

* review feedback

Signed-off-by: Huamin Chen <[email protected]>

* review feedback

Signed-off-by: Huamin Chen <[email protected]>

* review feedback

Signed-off-by: Huamin Chen <[email protected]>

---------

Signed-off-by: Huamin Chen <[email protected]>

---------

Signed-off-by: carlory <[email protected]>
Signed-off-by: JaredforReal <[email protected]>
Signed-off-by: yuluo-yx <[email protected]>
Signed-off-by: Yossi Ovadia <[email protected]>
Signed-off-by: cryo <[email protected]>
Signed-off-by: Huamin Chen <[email protected]>
Co-authored-by: 杨朱 · Kiki <[email protected]>
Co-authored-by: Jared <[email protected]>
Co-authored-by: bitliu <[email protected]>
Co-authored-by: shown <[email protected]>
Co-authored-by: Yossi Ovadia <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: cryo <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: rootfs <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Xunzhuo <[email protected]>

Author: 10 people

candle-binding/Cargo.lock

@@ -10,7 +10,7 @@ name = "candle_semantic_router"
 crate-type = ["staticlib", "cdylib"]
 
 [features]
-default = []
+default = ["cuda"]
 # CUDA support (enables GPU acceleration)
 cuda = ["candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]
 # Flash Attention 2 support (requires CUDA and compatible GPU)

candle-binding/Cargo.toml

@@ -24,6 +24,15 @@ semantic_cache:
   # Options: "bert" (fast, 384-dim), "qwen3" (high quality, 1024-dim, 32K context), "gemma" (balanced, 768-dim, 8K context)
   # Default: "bert" (fastest, lowest memory)
   embedding_model: "bert"
+  # HNSW index configuration (for memory backend only)
+  use_hnsw: true # Enable HNSW index for faster similarity search
+  hnsw_m: 16 # Number of bi-directional links (higher = better recall, more memory)
+  hnsw_ef_construction: 200 # Construction parameter (higher = better quality, slower build)
+  
+  # Hybrid cache configuration (when backend_type: "hybrid")
+  # Combines in-memory HNSW for fast search with Milvus for scalable storage
+  # max_memory_entries: 100000 # Max entries in HNSW index (default: 100,000)
+  # backend_config_path: "config/milvus.yaml" # Path to Milvus config
 
 tools:
   enabled: true

config/config.yaml

@@ -24,8 +24,8 @@ func NewCacheBackend(config CacheConfig) (CacheBackend, error) {
 	switch config.BackendType {
 	case InMemoryCacheType, "":
 		// Use in-memory cache as the default backend
-		observability.Debugf("Creating in-memory cache backend - MaxEntries: %d, TTL: %ds, Threshold: %.3f, UseHNSW: %t, EmbeddingModel: %s",
-			config.MaxEntries, config.TTLSeconds, config.SimilarityThreshold, config.UseHNSW, config.EmbeddingModel)
+		observability.Debugf("Creating in-memory cache backend - MaxEntries: %d, TTL: %ds, Threshold: %.3f, EmbeddingModel: %s, UseHNSW: %t",
+			config.MaxEntries, config.TTLSeconds, config.SimilarityThreshold, config.EmbeddingModel, config.UseHNSW)
 
 		options := InMemoryCacheOptions{
 			Enabled:             config.Enabled,
@@ -37,6 +37,9 @@ func NewCacheBackend(config CacheConfig) (CacheBackend, error) {
 			HNSWM:               config.HNSWM,
 			HNSWEfConstruction:  config.HNSWEfConstruction,
 			EmbeddingModel:      config.EmbeddingModel,
+			UseHNSW:             config.UseHNSW,
+			HNSWM:               config.HNSWM,
+			HNSWEfConstruction:  config.HNSWEfConstruction,
 		}
 		return NewInMemoryCache(options), nil
 

src/semantic-router/pkg/cache/cache_factory.go

@@ -120,4 +120,16 @@ type CacheConfig struct {
 	// EmbeddingModel specifies which embedding model to use
 	// Options: "bert" (default), "qwen3", "gemma"
 	EmbeddingModel string `yaml:"embedding_model,omitempty"`
+
+	// UseHNSW enables HNSW index for faster search in memory backend
+	UseHNSW bool `yaml:"use_hnsw,omitempty"`
+
+	// HNSWM is the number of bi-directional links per node (default: 16)
+	HNSWM int `yaml:"hnsw_m,omitempty"`
+
+	// HNSWEfConstruction is the size of dynamic candidate list during construction (default: 200)
+	HNSWEfConstruction int `yaml:"hnsw_ef_construction,omitempty"`
+
+	// Hybrid cache specific settings
+	MaxMemoryEntries int `yaml:"max_memory_entries,omitempty"` // Max entries in HNSW for hybrid cache
 }

src/semantic-router/pkg/cache/cache_interface.go

@@ -52,6 +52,9 @@ type InMemoryCache struct {
 	useHNSW             bool
 	hnswEfSearch        int // Search-time ef parameter
 	embeddingModel      string // "bert", "qwen3", or "gemma"
+	hnswIndex           *HNSWIndex
+	useHNSW             bool
+	hnswEfSearch        int // Search-time ef parameter
 }
 
 // InMemoryCacheOptions contains configuration parameters for the in-memory cache
@@ -66,6 +69,10 @@ type InMemoryCacheOptions struct {
 	HNSWEfConstruction  int  // Size of dynamic candidate list during construction (default: 200)
 	HNSWEfSearch        int  // Size of dynamic candidate list during search (default: 50)
 	EmbeddingModel      string // "bert", "qwen3", or "gemma"
+	UseHNSW             bool   // Enable HNSW index for faster search
+	HNSWM               int    // Number of bi-directional links (default: 16)
+	HNSWEfConstruction  int    // Size of dynamic candidate list during construction (default: 200)
+	HNSWEfSearch        int    // Size of dynamic candidate list during search (default: 50)
 }
 
 // NewInMemoryCache initializes a new in-memory semantic cache instance

src/semantic-router/pkg/cache/inmemory_cache.go

@@ -118,3 +118,20 @@ rust-flash-attn: ## Build Rust library with Flash Attention 2 (requires CUDA env
 		exit 1; \
 	fi
 	@cd candle-binding && cargo build --release --features flash-attn
+
+# Build the Rust library without CUDA (for CI/CD environments)
+rust-ci: ## Build the Rust library without CUDA support (for GitHub Actions/CI)
+	@$(LOG_TARGET)
+	@bash -c 'if ! command -v rustc >/dev/null 2>&1; then \
+		echo "rustc not found, installing..."; \
+		curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y; \
+	fi && \
+	if [ -f "$$HOME/.cargo/env" ]; then \
+		echo "Loading Rust environment from $$HOME/.cargo/env..." && \
+		. $$HOME/.cargo/env; \
+	fi && \
+	if ! command -v cargo >/dev/null 2>&1; then \
+		echo "Error: cargo not found in PATH" && exit 1; \
+	fi && \
+	echo "Building Rust library without CUDA (CPU-only)..." && \
+	cd candle-binding && cargo build --release --no-default-features'