update documentations for streaming mode

Signed-off-by: Jintao Zhang <[email protected]>

Author: tao12345666333

website/docs/api/router.md

@@ -351,6 +351,12 @@ histogram_quantile(0.95, sum(rate(llm_model_tpot_seconds_bucket[5m])) by (le, mo
 
 These are included in the provided Grafana dashboard at deploy/llm-router-dashboard.json as “TTFT (p95) by Model” and “TPOT (p95) by Model (sec/token)”.
 
+#### Streaming (SSE) notes
+
+- For Server-Sent Events (SSE) responses, the router measures TTFT on the first streamed body chunk (i.e., the first token), not on response headers.
+- No manual change to your Envoy config is required: the ExtProc handler automatically sets a ModeOverride with `response_body_mode: STREAMED` for SSE responses so the first chunk reaches ExtProc immediately.
+- Prerequisite: Envoy’s ext_proc filter must have `allow_mode_override: true` (the default configs in `config/envoy.yaml` and `config/envoy-docker.yaml` already include this). Keeping `response_body_mode: BUFFERED` in the static processing mode is fine; the router will flip it to STREAMED at runtime for SSE.
+
 ### Pricing Configuration
 
 Provide per-1M pricing for your models so the router can compute request cost and emit metrics/logs.

website/docs/overview/architecture/envoy-extproc.md

@@ -410,7 +410,7 @@ static_resources:
                 request_header_mode: "SEND"
                 response_header_mode: "SEND"
                 request_body_mode: "BUFFERED"     # Required for content analysis
-                response_body_mode: "BUFFERED"    # Required for caching
+                response_body_mode: "BUFFERED"    # Default: router flips to STREAMED at runtime for SSE
                 request_trailer_mode: "SKIP"
                 response_trailer_mode: "SKIP"
               
@@ -419,6 +419,13 @@ static_resources:
               allow_mode_override: true           # Allow ExtProc to change modes
               message_timeout: 300s               # Timeout for ExtProc responses
               max_message_timeout: 600s           # Maximum allowed timeout
+
+> Note on SSE (streaming):
+>
+> When the upstream responds with `Content-Type: text/event-stream`, the router sets a per-message
+> `ModeOverride` with `response_body_mode: STREAMED` so the first chunk reaches ExtProc immediately.
+> This enables accurate TTFT measurement on the first token. No manual change to the static
+> `processing_mode` is required as long as `allow_mode_override: true` is set (it is in the default configs).
 
               # Advanced configuration
               mutation_rules: