Commit 77bbf40
authored
Teach the router with one example per capability (#7771)
The top-level router picks one of six capabilities. On qwen3:8b it was
getting that wrong roughly a quarter of the time.
Two changes: six worked examples appended to the router prompt, and
`temperature: 0` pinned on the router call. `build_model_settings` only
ever set `max_tokens`, so the router ran at qwen3's own defaults
(temperature 0.6, top_p 0.95) and the same question could route
differently on repeat. The pin is on the router only, so generation
elsewhere is unchanged.
## What the numbers say
An ablation isolates each factor. 82 labelled turns against a live
Ollama, two independent runs each:
| variant | mean accuracy |
|---|---|
| ships today | 76.2% |
| + temperature 0 | 80.5% |
| + `max_tokens` 4096 | 81.7% |
| + examples | **89.6%** |
The prompt does the work. `max_tokens` was worth about 1.2 points - one
case at this sample size - so **it is no longer part of this PR**. Its
real justification was avoiding empty responses when thinking exhausts
the budget, and no run recorded a single hard failure, so it was not
earning its place. Raising it remains reasonable as a separate safety
change.
## Why six examples and not more
The first version of this PR carried thirteen, mostly contrasting pairs
on the question-versus-edit boundary. Measured against a set with **one
example per capability**:
| prompt | mean accuracy | prompt tokens |
|---|---|---|
| 13 examples, boundary-heavy | 87.8% | 648 |
| 4 examples, boundary only | 78.0% | 458 |
| **6 examples, one per capability** | **89.6%** | **517** |
Six is both better and smaller. Cutting to four - still boundary-only -
threw the entire gain away, which is the tell: the router's dominant
failure was never question-versus-edit, it was bailing to `unsupported`
(15 of 19 errors in the original run). A block with no `unsupported`,
`pdf_review`, `pdf_create` or `user_spec` example cannot fix the errors
that actually dominate. Coverage matters more than volume.
## Also worth knowing
Thinking stays on. With the same prompt, thinking on scored 91.5%
against 89.0% off in the original run, and took destructive misroutes -
read-only turns sent to a file-mutating capability - from three to zero.
Reproduce with the harness in #7778: `uv run --group engine python
evals/routing/runner.py`1 parent 64edee2 commit 77bbf40
1 file changed
Lines changed: 16 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
65 | 66 | | |
66 | 67 | | |
67 | 68 | | |
68 | | - | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
69 | 79 | | |
70 | 80 | | |
71 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
72 | 86 | | |
73 | 87 | | |
74 | 88 | | |
| |||
146 | 160 | | |
147 | 161 | | |
148 | 162 | | |
149 | | - | |
| 163 | + | |
150 | 164 | | |
151 | 165 | | |
152 | 166 | | |
| |||
0 commit comments