fix lint

Signed-off-by: Ryan Cook <[email protected]>

Author: cooktheryan

deploy/kserve/QUICKSTART.md

@@ -41,13 +41,15 @@ cd deploy/kserve
 ```
 
 **Example:**
+
 ```bash
 ./deploy.sh --namespace semantic --inferenceservice granite32-8b --model granite32-8b
 ```
 
 ### Step 3: Wait for Ready
 
 The script will:
+
 - ✓ Validate your environment
 - ✓ Download classification models (~2-3 minutes)
 - ✓ Start the semantic router
@@ -97,6 +99,7 @@ Using a specific storage class, larger PVCs, and custom embedding model:
 ```
 
 **Available Embedding Models:**
+
 - `all-MiniLM-L12-v2` (default) - Balanced speed/quality (~90MB)
 - `all-mpnet-base-v2` - Higher quality, larger (~420MB)
 - `all-MiniLM-L6-v2` - Faster, smaller (~80MB)
@@ -241,6 +244,7 @@ Simply redeploy:
 ## Next Steps
 
 1. **Run validation tests**:
+
    ```bash
    # Set namespace and model name
    NAMESPACE=<namespace> MODEL_NAME=<model> ./test-semantic-routing.sh
@@ -274,6 +278,7 @@ Simply redeploy:
 ## Want More Control?
 
 This quick start uses the automated `deploy.sh` script for simplicity. If you need:
+
 - Manual step-by-step deployment
 - Deep understanding of configuration options
 - Advanced customization

deploy/kserve/README.md

@@ -103,6 +103,14 @@ INFERENCESERVICE_NAME=<your-inferenceservice-name>
 
 # KServe creates a headless service by default (no stable ClusterIP)
 # Create a stable ClusterIP service for consistent routing
+
+# Option 1: Using the template file (recommended)
+# Substitute variables and apply
+sed -e "s/{{INFERENCESERVICE_NAME}}/$INFERENCESERVICE_NAME/g" \
+    -e "s/{{NAMESPACE}}/$NAMESPACE/g" \
+    service-predictor-stable.yaml | oc apply -f - -n $NAMESPACE
+
+# Option 2: Using heredoc
 cat <<EOF | oc apply -f - -n $NAMESPACE
 apiVersion: v1
 kind: Service
@@ -239,6 +247,7 @@ Find the `kserve_dynamic_cluster` section and update:
 ```
 
 Replace:
+
 - `my-model` with your InferenceService name
 - `my-namespace` with your namespace
 
@@ -709,6 +718,7 @@ oc logs -l app=semantic-router -c semantic-router -n $NAMESPACE -f
 **Important Log Events:**
 
 - `routing_decision`: Which model was selected and why
+
   ```json
   {
     "msg": "routing_decision",
@@ -720,6 +730,7 @@ oc logs -l app=semantic-router -c semantic-router -n $NAMESPACE -f
   ```
 
 - `cache_hit`/`cache_miss`: Cache performance
+
   ```json
   {
     "msg": "cache_hit",
@@ -730,6 +741,7 @@ oc logs -l app=semantic-router -c semantic-router -n $NAMESPACE -f
   ```
 
 - `llm_usage`: Token usage and costs
+
   ```json
   {
     "msg": "llm_usage",
@@ -742,6 +754,7 @@ oc logs -l app=semantic-router -c semantic-router -n $NAMESPACE -f
   ```
 
 - `pii_detection`: PII found in requests
+
   ```json
   {
     "msg": "pii_detection",
@@ -815,9 +828,11 @@ oc describe pod -l app=semantic-router -n $NAMESPACE
    - Solution: Check network policies, proxy settings
 
 2. **PVC not bound**: Storage not provisioned
+
    ```bash
    oc get pvc -n $NAMESPACE
    ```
+
    - Solution: Check StorageClass, provision capacity
 
 3. **OOM during model download**: Insufficient memory
@@ -836,21 +851,27 @@ oc logs -l app=semantic-router -c semantic-router -n $NAMESPACE --previous
 **Common Causes**:
 
 1. **Configuration error**: Invalid YAML or missing fields
+
    ```
    Failed to load config: yaml: unmarshal errors
    ```
+
    - Solution: Validate YAML syntax, check required fields
 
 2. **Invalid IP address**: Router validation failed
+
    ```
    invalid IP address format, got: my-model.svc.cluster.local
    ```
+
    - Solution: Use service ClusterIP (not DNS) in `vllm_endpoints.address` - see Step 1 for creating stable service
 
 3. **Missing models**: Classification models not downloaded
+
    ```
    failed to read mapping file: no such file or directory
    ```
+
    - Solution: Check init container completed successfully
 
 ### Cannot Connect to InferenceService
@@ -870,24 +891,30 @@ oc exec $POD -c semantic-router -n $NAMESPACE -- \
 **Common Causes**:
 
 1. **InferenceService not ready**:
+
    ```bash
    oc get inferenceservice -n $NAMESPACE
    ```
+
    - Solution: Wait for READY=True, check predictor logs
 
 2. **Wrong DNS name**: Incorrect service name in Envoy config
    - Solution: Verify format: `<inferenceservice>-predictor.<namespace>.svc.cluster.local`
 
 3. **Network policy blocking**: Istio/NetworkPolicy restrictions
+
    ```bash
    oc get networkpolicies -n $NAMESPACE
    ```
+
    - Solution: Add policy to allow traffic from router to predictor
 
 4. **PeerAuthentication conflict**: mTLS mode mismatch
+
    ```bash
    oc get peerauthentication -n $NAMESPACE
    ```
+
    - Solution: Ensure PERMISSIVE mode or adjust Envoy TLS config
 
 ### Predictor Pod IP Changed (If Using Pod IP Instead of Service IP)
@@ -901,6 +928,7 @@ oc exec $POD -c semantic-router -n $NAMESPACE -- \
 **Solution**:
 
 1. Switch to stable service approach (recommended):
+
    ```bash
    # Create stable service
    cat <<EOF | oc apply -f - -n $NAMESPACE
@@ -944,9 +972,11 @@ oc logs -l app=semantic-router -c semantic-router -n $NAMESPACE \
 **Common Causes**:
 
 1. **Threshold too high**: Similarity threshold prevents matches
+
    ```yaml
    similarity_threshold: 0.99  # Too strict
    ```
+
    - Solution: Lower threshold to 0.8-0.85
 
 2. **Cache disabled**: Not enabled in config
@@ -1066,6 +1096,7 @@ For multi-replica deployments with shared cache:
 
 1. Deploy Milvus in your cluster
 2. Update `configmap-router-config.yaml`:
+
    ```yaml
    semantic_cache:
      enabled: true
@@ -1077,6 +1108,7 @@ For multi-replica deployments with shared cache:
    ```
 
 3. Apply and restart:
+
    ```bash
    oc apply -f configmap-router-config.yaml -n $NAMESPACE
    oc rollout restart deployment/semantic-router-kserve -n $NAMESPACE

deploy/kserve/deploy.sh

@@ -212,7 +212,14 @@ if [ "$SKIP_VALIDATION" = false ]; then
 
     # Create stable ClusterIP service for predictor (KServe creates headless service by default)
     echo "Creating stable ClusterIP service for predictor..."
-    cat <<EOF | oc apply -f - -n "$NAMESPACE" > /dev/null 2>&1
+
+    # Use template file for stable service
+    if [ -f "$SCRIPT_DIR/service-predictor-stable.yaml" ]; then
+        substitute_vars "$SCRIPT_DIR/service-predictor-stable.yaml" "$TEMP_DIR/service-predictor-stable.yaml.tmp"
+        oc apply -f "$TEMP_DIR/service-predictor-stable.yaml.tmp" -n "$NAMESPACE" > /dev/null 2>&1
+    else
+        # Fallback to inline creation if template not found
+        cat <<EOF | oc apply -f - -n "$NAMESPACE" > /dev/null 2>&1
 apiVersion: v1
 kind: Service
 metadata:
@@ -233,6 +240,7 @@ spec:
     targetPort: 8080
     protocol: TCP
 EOF
+    fi
 
     # Get the stable ClusterIP
     PREDICTOR_SERVICE_IP=$(oc get svc "${INFERENCESERVICE_NAME}-predictor-stable" -n "$NAMESPACE" -o jsonpath='{.spec.clusterIP}' 2>/dev/null || echo "")

deploy/kserve/service-predictor-stable.yaml

@@ -3,17 +3,17 @@
 apiVersion: v1
 kind: Service
 metadata:
-  name: {{INFERENCESERVICE_NAME}}-predictor-stable
-  namespace: {{NAMESPACE}}
+  name: "{{INFERENCESERVICE_NAME}}-predictor-stable"
+  namespace: "{{NAMESPACE}}"
   labels:
-    app: {{INFERENCESERVICE_NAME}}
+    app: "{{INFERENCESERVICE_NAME}}"
     component: predictor-stable
   annotations:
     description: "Stable ClusterIP service for semantic router to use (headless service doesn't provide ClusterIP)"
 spec:
   type: ClusterIP
   selector:
-    serving.kserve.io/inferenceservice: {{INFERENCESERVICE_NAME}}
+    serving.kserve.io/inferenceservice: "{{INFERENCESERVICE_NAME}}"
   ports:
   - name: http
     port: 8080