Merge branch 'main' into fix-score-payload

Author: keyuchen21

README.md

@@ -13,7 +13,7 @@
 
 We host **bi-weekly** community meetings at the following timeslot:
 
-- Every other Tuesdays at 5:30 PM PT – [Add to Calendar](https://drive.usercontent.google.com/u/0/uc?id=1I3WuivUVAq1vZ2XSW4rmqgD5c0bQcxE0&export=download)
+- Every other Tuesdays at 5:30 PM PT – [Add to Calendar](https://drive.google.com/uc?export=download&id=1D4SqQiqzdSx_xsEwS0QTd592zd3Xourh)
 
 All are welcome to join!
 

community/community-event.md

@@ -8,4 +8,4 @@ Info can be found in the [Google Doc](https://docs.google.com/document/d/1SCye2q
 
 Time: Bi-weekly
 
-**Every other Tuesday 5:30 - 6:00 PM PT** – [Add to Calendar](https://drive.usercontent.google.com/u/0/uc?id=1I3WuivUVAq1vZ2XSW4rmqgD5c0bQcxE0&export=download)
+**Every other Tuesday 5:30 - 6:00 PM PT** – [Add to Calendar](https://drive.google.com/uc?export=download&id=1D4SqQiqzdSx_xsEwS0QTd592zd3Xourh)

docs/source/community/meetings.rst

@@ -6,6 +6,6 @@ Community Events
 
 We host bi-weekly community meetings at the following timeslot:
 
-**Every other Tuesday at 5:30 PM PT** – `Add to Calendar <https://drive.usercontent.google.com/u/0/uc?id=1I3WuivUVAq1vZ2XSW4rmqgD5c0bQcxE0&export=download>`_
+**Every other Tuesday at 5:30 PM PT** – `Add to Calendar <https://drive.google.com/uc?export=download&id=1D4SqQiqzdSx_xsEwS0QTd592zd3Xourh>`_
 
 All are welcome to join!

docs/source/use_cases/semantic-router-integration.rst

@@ -74,21 +74,29 @@ Identify the ClusterIP and port of your router Service:
 Step 2: Deploy vLLM Semantic Router
 ------------------------------------
 
-Follow the official `Install in Kubernetes <https://vllm-semantic-router.com/docs/installation/kubernetes>`_ guide with the updated configuration.
+Follow the official `Install in Kubernetes <https://vllm-semantic-router.com/docs/installation/k8s/ai-gateway>`_ guide with the updated configuration.
+
+Deploy vLLM Semantic Router using Helm:
 
 .. code-block:: bash
 
-   # Deploy vLLM Semantic Router manifests
-   kubectl apply -k deploy/kubernetes/ai-gateway/semantic-router
+   # Deploy vLLM Semantic Router with custom values from GHCR OCI registry
+   # (Optional) If you use a registry mirror/proxy, append: --set global.imageRegistry=<your-registry>
+   helm install semantic-router oci://ghcr.io/vllm-project/charts/semantic-router \
+     --version v0.0.0-latest \
+     --namespace vllm-semantic-router-system \
+     --create-namespace \
+     -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/semantic-router-values/values.yaml
+
    kubectl wait --for=condition=Available deployment/semantic-router \
      -n vllm-semantic-router-system --timeout=600s
 
    # Install Envoy Gateway
-  helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \
-    --version v0.0.0-latest \
-    --namespace envoy-gateway-system \
-    --create-namespace \
-    -f https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/manifests/envoy-gateway-values.yaml
+   helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \
+     --version v0.0.0-latest \
+     --namespace envoy-gateway-system \
+     --create-namespace \
+     -f https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/manifests/envoy-gateway-values.yaml
 
    # Install Envoy AI Gateway
    helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \
@@ -97,20 +105,27 @@ Follow the official `Install in Kubernetes <https://vllm-semantic-router.com/doc
      --create-namespace
 
    # Install Envoy AI Gateway CRDs
-   helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm --version v0.0.0-latest --namespace envoy-ai-gateway-system
+   helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \
+     --version v0.0.0-latest \
+     --namespace envoy-ai-gateway-system
 
    # Wait for AI Gateway to be ready
    kubectl wait --timeout=300s -n envoy-ai-gateway-system \
      deployment/ai-gateway-controller --for=condition=Available
 
+.. note::
+
+   The values file contains the configuration for the semantic router including domain classification, LoRA routing, and plugin settings. You can download and customize it from the `semantic-router-values <https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/semantic-router-values/values.yaml>`_ to match your vLLM Production Stack setup.
+
 Create LLM Demo Backends and AI Gateway Routes:
 
 .. code-block:: bash
 
    # Apply LLM demo backends
-   kubectl apply -f deploy/kubernetes/ai-gateway/aigw-resources/base-model.yaml
+   kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/base-model.yaml
+
    # Apply AI Gateway routes
-   kubectl apply -f deploy/kubernetes/ai-gateway/aigw-resources/gwapi-resources.yaml
+   kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/gwapi-resources.yaml
 
 Step 3: Test the Deployment
 ----------------------------

helm/README.md

@@ -201,6 +201,14 @@ This table documents all available configuration values for the Production Stack
 | `routerSpec.readinessProbe.failureThreshold` | integer |`3`| Failure threshold for router's readiness probe |
 | `routerSpec.readinessProbe.httpGet.path` | string |`"/health"`| Endpoint that the router's readiness probe will be testing |
 
+#### Router OpenTelemetry Configuration
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `routerSpec.otel.endpoint` | string | `""` | OTLP endpoint for tracing (e.g., "otel-collector:4317"). Tracing is enabled when this is set. |
+| `routerSpec.otel.serviceName` | string | `"vllm-router"` | Service name for OpenTelemetry traces |
+| `routerSpec.otel.secure` | boolean | `false` | Use secure (TLS) connection for OTLP exporter |
+
 #### Router Ingress Configuration
 
 | Field | Type | Default | Description |

helm/templates/deployment-router.yaml

@@ -136,6 +136,15 @@ spec:
           - "--lmcache-controller-port"
           - "{{ .Values.routerSpec.lmcacheControllerPort }}"
           {{- end }}
+          {{- if .Values.routerSpec.otel.endpoint }}
+          - "--otel-endpoint"
+          - "{{ .Values.routerSpec.otel.endpoint }}"
+          - "--otel-service-name"
+          - "{{ .Values.routerSpec.otel.serviceName | default "vllm-router" }}"
+          {{- if .Values.routerSpec.otel.secure }}
+          - "--otel-secure"
+          {{- end }}
+          {{- end }}
         {{- if .Values.routerSpec.resources }}
         resources:
           {{- if .Values.routerSpec.resources.requests }}

helm/tests/routerOtel_test.yaml

@@ -0,0 +1,77 @@
+suite: test router OpenTelemetry configuration
+templates:
+  - deployment-router.yaml
+tests:
+  - it: should not include otel args when endpoint is not set
+    set:
+      routerSpec:
+        enableRouter: true
+        otel:
+          endpoint: ""
+    asserts:
+      - template: deployment-router.yaml
+        notContains:
+          path: spec.template.spec.containers[0].args
+          content: "--otel-endpoint"
+
+  - it: should include otel args when endpoint is set
+    set:
+      routerSpec:
+        enableRouter: true
+        otel:
+          endpoint: "otel-collector:4317"
+          serviceName: "vllm-router"
+          secure: false
+    asserts:
+      - template: deployment-router.yaml
+        contains:
+          path: spec.template.spec.containers[0].args
+          content: "--otel-endpoint"
+      - template: deployment-router.yaml
+        contains:
+          path: spec.template.spec.containers[0].args
+          content: "otel-collector:4317"
+      - template: deployment-router.yaml
+        contains:
+          path: spec.template.spec.containers[0].args
+          content: "--otel-service-name"
+      - template: deployment-router.yaml
+        contains:
+          path: spec.template.spec.containers[0].args
+          content: "vllm-router"
+      - template: deployment-router.yaml
+        notContains:
+          path: spec.template.spec.containers[0].args
+          content: "--otel-secure"
+
+  - it: should use custom service name when specified
+    set:
+      routerSpec:
+        enableRouter: true
+        otel:
+          endpoint: "jaeger:4317"
+          serviceName: "my-custom-router"
+          secure: false
+    asserts:
+      - template: deployment-router.yaml
+        contains:
+          path: spec.template.spec.containers[0].args
+          content: "my-custom-router"
+
+  - it: should include otel-secure flag when secure is true
+    set:
+      routerSpec:
+        enableRouter: true
+        otel:
+          endpoint: "otel-collector:4317"
+          serviceName: "vllm-router"
+          secure: true
+    asserts:
+      - template: deployment-router.yaml
+        contains:
+          path: spec.template.spec.containers[0].args
+          content: "--otel-endpoint"
+      - template: deployment-router.yaml
+        contains:
+          path: spec.template.spec.containers[0].args
+          content: "--otel-secure"

helm/values.schema.json

@@ -580,6 +580,26 @@
           "additionalProperties": {
             "type": "string"
           }
+        },
+        "otel": {
+          "type": "object",
+          "description": "OpenTelemetry tracing configuration for the router",
+          "properties": {
+            "endpoint": {
+              "type": "string",
+              "description": "OTLP endpoint for tracing (e.g., 'otel-collector:4317'). Tracing is enabled when this is set."
+            },
+            "serviceName": {
+              "type": "string",
+              "description": "Service name for OpenTelemetry traces",
+              "default": "vllm-router"
+            },
+            "secure": {
+              "type": "boolean",
+              "description": "Use secure (TLS) connection for OTLP exporter",
+              "default": false
+            }
+          }
         }
       }
     }

helm/values.yaml

@@ -377,6 +377,16 @@ routerSpec:
   # -- Window size in seconds to calculate the request statistics
   requestStatsWindow: 60
 
+  # -- OpenTelemetry tracing configuration
+  # When otelEndpoint is set, tracing is automatically enabled
+  otel:
+    # -- OTLP endpoint for tracing (e.g., "localhost:4317" or "otel-collector:4317")
+    endpoint: ""
+    # -- Service name for traces (default: "vllm-router")
+    serviceName: "vllm-router"
+    # -- Use secure (TLS) connection for OTLP exporter (default: false, i.e., insecure)
+    secure: false
+
   # -- deployment strategy
   strategy: {}
 

operator/api/v1alpha1/vllmruntime_types.go

@@ -30,6 +30,9 @@ type DeploymentConfig struct {
 	// +kubebuilder:default=1
 	Replicas int32 `json:"replicas,omitempty"`
 
+	// Node selector
+	NodeSelectorTerms []corev1.NodeSelectorTerm `json:"nodeSelectorTerms,omitempty"`
+
 	// Deploy strategy
 	// +kubebuilder:validation:Enum=RollingUpdate;Recreate
 	// +kubebuilder:default=RollingUpdate
@@ -122,6 +125,9 @@ type ModelSpec struct {
 
 	// Maximum number of sequences
 	MaxNumSeqs int32 `json:"maxNumSeqs,omitempty"`
+
+	// Chat template
+	ChatTemplate string `json:"chatTemplate,omitempty"`
 }
 
 // LMCacheConfig defines the LM Cache configuration