Skip to content

Commit 77bbf40

Browse files
authored
Teach the router with one example per capability (#7771)
The top-level router picks one of six capabilities. On qwen3:8b it was getting that wrong roughly a quarter of the time. Two changes: six worked examples appended to the router prompt, and `temperature: 0` pinned on the router call. `build_model_settings` only ever set `max_tokens`, so the router ran at qwen3's own defaults (temperature 0.6, top_p 0.95) and the same question could route differently on repeat. The pin is on the router only, so generation elsewhere is unchanged. ## What the numbers say An ablation isolates each factor. 82 labelled turns against a live Ollama, two independent runs each: | variant | mean accuracy | |---|---| | ships today | 76.2% | | + temperature 0 | 80.5% | | + `max_tokens` 4096 | 81.7% | | + examples | **89.6%** | The prompt does the work. `max_tokens` was worth about 1.2 points - one case at this sample size - so **it is no longer part of this PR**. Its real justification was avoiding empty responses when thinking exhausts the budget, and no run recorded a single hard failure, so it was not earning its place. Raising it remains reasonable as a separate safety change. ## Why six examples and not more The first version of this PR carried thirteen, mostly contrasting pairs on the question-versus-edit boundary. Measured against a set with **one example per capability**: | prompt | mean accuracy | prompt tokens | |---|---|---| | 13 examples, boundary-heavy | 87.8% | 648 | | 4 examples, boundary only | 78.0% | 458 | | **6 examples, one per capability** | **89.6%** | **517** | Six is both better and smaller. Cutting to four - still boundary-only - threw the entire gain away, which is the tell: the router's dominant failure was never question-versus-edit, it was bailing to `unsupported` (15 of 19 errors in the original run). A block with no `unsupported`, `pdf_review`, `pdf_create` or `user_spec` example cannot fix the errors that actually dominate. Coverage matters more than volume. ## Also worth knowing Thinking stays on. With the same prompt, thinking on scored 91.5% against 89.0% off in the original run, and took destructive misroutes - read-only turns sent to a file-mutating capability - from three to zero. Reproduce with the harness in #7778: `uv run --group engine python evals/routing/runner.py`
1 parent 64edee2 commit 77bbf40

1 file changed

Lines changed: 16 additions & 2 deletions

File tree

engine/src/stirling/agents/orchestrator.py

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from pydantic import ConfigDict, Field
88
from pydantic_ai import Agent
99
from pydantic_ai.output import NativeOutput, ToolOutput
10+
from pydantic_ai.settings import ModelSettings
1011
from pydantic_ai.tools import RunContext
1112

1213
from stirling.agents.output_mode import output_retries, uses_tool_output
@@ -65,10 +66,23 @@ class _RouteDecision(ApiModel):
6566
"- pdf_create: generate a NEW document from scratch (invoice, report, letter) - no input file.\n"
6667
"- unsupported: none of the above fit, or the user asks about the assistant itself; put a "
6768
"helpful message in 'message'.\n"
68-
"Respond with the capability and (only for unsupported) a message."
69+
"Respond with the capability and (only for unsupported) a message.\n"
70+
"\n"
71+
"Decide by what the user wants BACK. Information read out of the document is pdf_question; "
72+
"a changed file handed back is pdf_edit. Mentioning a document is not asking to change it.\n"
73+
' "Is there a table of contents?" -> pdf_question\n'
74+
' "Add a table of contents" -> pdf_edit\n'
75+
' "Put a note on every paragraph that needs work" -> pdf_review\n'
76+
' "Build me a purchase order from scratch" -> pdf_create\n'
77+
' "Make a rule that stamps every upload" -> user_spec\n'
78+
' "Which version of the app is this?" -> unsupported'
6979
)
7080

7181

82+
def _router_model_settings(runtime: AppRuntime) -> ModelSettings:
83+
return {**runtime.fast_model_settings, "temperature": 0.0}
84+
85+
7286
class OrchestratorAgent:
7387
def __init__(self, runtime: AppRuntime) -> None:
7488
self.runtime = runtime
@@ -146,7 +160,7 @@ def __init__(self, runtime: AppRuntime) -> None:
146160
output_type=NativeOutput([_RouteDecision]),
147161
retries=output_retries(runtime.settings.chat_provider),
148162
system_prompt=_ROUTER_SYSTEM_PROMPT,
149-
model_settings=runtime.fast_model_settings,
163+
model_settings=_router_model_settings(runtime),
150164
)
151165
if self._route_via_enum
152166
else None

0 commit comments

Comments
 (0)