Router fails for some prompts and works for others

[shira-g]

I configured SR using two models and run benchmark.
Some prompts (from MMLU-Pro) works ok, I can send a request and I am getting an answer, however for some prompts sending request fails and I am getting an error: "no healthy upstream". I see in the envoy log that response code is 503.

models:

1. vllm serve microsoft/Phi-3-mini-4k-instruct --port=8002 --host 127.0.0.1 --max-model-len 8192 --enforce-eager --served-model-name phi3
2. vllm serve openai/gpt-oss-20b --port=8003 --host 0.0.0.0 --served-model-name gpt-oss --max-model-len 8192 --enforce-eager

Failing request for example:
curl -X POST http://localhost:8801/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "MoM",
"messages": [
{"role": "user", "content": "who are you"}
]
}'

Working request for example:
Adding a question mark for the prompt above ("who are you?").

config.yaml
fail_envoy_log.txt
fail_router_log.txt

How can I debug this?
attaching the logs.

Thank you

[Xunzhuo]

the default model you are configuring is incorrect, you need to pick one of your models

[shira-g]

Shouldn't default model be any name I choose? I don't want to use a specific model, I want to call the router option (e.g. 'auto' or 'MoM'.)
Also, for some prompts the routing works fine, so I assume it's not the issue, am I wrong?

I changed default model in the yaml and the failure still occurs.

config.yaml

Thank you

[rootfs]

@shira-g the router log is incomplete, do you have a full log?

[Xunzhuo]

@shira-g default model is not the model you specify in model name in request, it is a fallback model when the entropy is lower than the threshold

[shira-g]

@rootfs is there a better way to extract router log rather than copy from terminal?

[rootfs]

@shira-g if you run the router inside of a container, you can get the container log, e.g. docker logs

[shira-g]

@rootfs I'm running using conda env

[shira-g]

@rootfs I attached full logs for both cases:

1. failing request:

curl -X POST http://localhost:8801/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "auto",
"messages": [
{"role": "user", "content": "who are you"}
]
}'

output:
"no healthy upstream"

2. success request:
curl -X POST http://localhost:8801/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "auto",
"messages": [
{"role": "user", "content": "who are you?"}
]
}'

output:
{"id":"chatcmpl-8c1252c9-1d4b-4212-8112-0c8e51fb4b1b","object":"chat.completion","created":1763408724,"model":"gpt-oss","choices":[{"index":0,"message":{"role":"assistant","content":"I’m ChatGPT, a large language‑model AI built by OpenAI. I’m powered by the GPT‑4 architecture and trained on a wide variety of text to help answer questions, explain concepts, offer suggestions, and chat about almost any topic you’re interested in. I don’t have personal experiences or feelings, and I don’t store or remember private details from our conversation. My goal is to provide clear, useful information and support you with your queries. How can I help you today?","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":"The user asks "who are you?" It's a straightforward question, so we should answer accordingly. They might expect a response describing being a large language model, ChatGPT, developed by OpenAI. Also mention limitations, privacy, that I don't have physical existence, etc.\n\nGiven instructions: "You are a helpful and knowledgeable assistant." Provide accurate, helpful responses across a wide range of topics. So answer accordingly: I am ChatGPT, an AI language model from OpenAI, trained up to 2023. I'm here to help, no consciousness, etc.\n\nAlso we need to tailor tone: professional but friendly.\n\nWe must consider whether there's any conflict with user request. The question is fairly innocuous.\n\nWe won't mention policies unless needed. There's no disallowed content.\n\nWe'll comply."},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":97,"total_tokens":364,"completion_tokens":267,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

fail_envoy.txt
fail_router.txt
success_envoy.txt
success_router.txt

[shira-g]

Hi @rootfs, any update on this?
From the logs it looks like the flow is failing when classification certainty < threshold so no category is returned:

classifier.go:699","msg":"Classification confidence (0.4178) below threshold (0.6000), but entropy analysis available"}
{"level":"info","ts":"2025-11-17T11:40:53.416964766-08:00","caller":"extproc/req_filter_classification.go:50","msg":"Classification Result: category=, confidence=0.418, reasoning=true"}

I would expect the request to be directed to the default model in that case, but "no healthy upstream" is returned.