Document DeepSeek agent loop 50 query eval (#131)

Author: SonAIengine

examples/ablation/diagnostics/agent_loop_20260702_201106.md

@@ -0,0 +1,91 @@
+# Agent Loop Retrieval Benchmark — Synaptic
+
+- Run at: 2026-07-02 20:11:06 KST
+- Dataset path: tests/benchmark/data/msmarco_passage_full.json
+- SQLite DB path: tests/benchmark/data/msmarco_full.db
+- Subset: 50
+- Corpus limit: 8841823
+- LLM base URL: https://api.deepseek.com/v1
+- Model: deepseek-v4-flash
+- Max turns: 5
+- Sufficiency gate: yes
+- Force first tool: yes
+- Incremental JSONL: examples/ablation/diagnostics/agent_loop_deepseek_v4_flash_50.jsonl
+- SQLite FTS AND-first threshold: 20
+- SQLite FTS lexical rerank pool: 500
+
+This measures LLM-planned exploration. The agent can change follow-up queries and tool choices based on evidence from earlier turns. The main metric is document reach, not ranked MRR, because the agent loop returns a cumulative evidence set.
+
+## Summary
+
+- Reach: 23/50 (0.460)
+- Mean turns: 4.14
+- Mean tool calls: 5.78
+- Mean first relevant turn: 1.70
+- Mean first relevant tool calls: 2.22
+- Mean elapsed: 50.8s
+- P50/P90 elapsed: 48.7s / 67.0s
+- Mean prompt tokens: 19616
+- Mean completion tokens: 960
+- Mean unique tools: 2.38
+- Mean unique search targets: 5.40
+- Mean query rewrites: 4.20
+- Queries with >1 tool type: 48/50
+- Queries with query rewrites: 48/50
+- Duplicate tool calls: 0
+- Empty tool calls: 11
+
+## Per Query
+
+| QID | Reach | Turns | Calls | Tools | Targets | Rewrites | First Rel Turn | First Rel Calls | Found Relevant | Elapsed | Query |
+|-----|:-----:|------:|------:|------:|--------:|---------:|---------------:|----------------:|----------------|--------:|-------|
+| 300674 | yes | 4 | 4 | 2 | 3 | 3 | 1 | 1 | 7067032 | 36.2s | how many years did william bradford serve as governor of plymouth colony? |
+| 125705 | no | 3 | 3 | 2 | 2 | 2 | - | - | - | 38.5s | define preventive |
+| 94798 | yes | 4 | 5 | 2 | 5 | 4 | 1 | 1 | 7067181 | 46.4s | color overlay photoshop |
+| 9083 | yes | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 7067274 | 28.3s | ____________________ is considered the father of modern medicine. |
+| 174249 | no | 4 | 3 | 2 | 3 | 3 | - | - | - | 39.9s | does xpress bet charge to deposit money in your account |
+| 320792 | no | 5 | 7 | 3 | 7 | 7 | - | - | - | 61.3s | how much is a cost to run disneyland |
+| 1090270 | yes | 4 | 3 | 2 | 1 | 0 | 1 | 1 | 7067796 | 40.0s | botulinum definition |
+| 1101279 | no | 5 | 6 | 2 | 6 | 4 | - | - | - | 53.8s | do physicians pay for insurance from their salaries? |
+| 201376 | yes | 5 | 8 | 3 | 8 | 2 | 1 | 1 | 7068066 | 60.4s | here there be dragons comic |
+| 54544 | no | 5 | 10 | 3 | 10 | 10 | - | - | - | 67.0s | blood diseases that are sexually transmitted |
+| 118457 | no | 3 | 3 | 2 | 2 | 1 | - | - | - | 38.9s | define bona fides |
+| 178627 | yes | 5 | 13 | 3 | 13 | 12 | 3 | 6 | 7068519 | 75.8s | effects of detox juice cleanse |
+| 1101278 | no | 3 | 4 | 2 | 4 | 1 | - | - | - | 41.6s | do prince harry and william have last names |
+| 68095 | yes | 4 | 8 | 2 | 8 | 4 | 1 | 1 | 7069266 | 59.2s | can hives be a sign of pregnancy |
+| 87892 | yes | 5 | 10 | 3 | 8 | 8 | 3 | 5 | 7069601 | 64.6s | causes of petechial hemorrhage |
+| 257309 | no | 4 | 5 | 3 | 5 | 5 | - | - | - | 54.1s | how long does it take to get your bsrn if you already have a bachelors degree |
+| 1090242 | yes | 5 | 14 | 3 | 14 | 13 | 3 | 7 | 7070556 | 70.1s | symptoms of ptsd in vietnam veterans |
+| 211691 | no | 5 | 5 | 3 | 4 | 3 | - | - | - | 45.3s | how coffee works quote |
+| 165002 | yes | 4 | 4 | 2 | 4 | 1 | 1 | 1 | 7070877 | 44.6s | does contraction of the ciliary muscles shorten the lens |
+| 1101276 | yes | 3 | 2 | 2 | 2 | 1 | 1 | 1 | 7070950 | 34.6s | do spiders eat other animals |
+| 264827 | yes | 3 | 2 | 2 | 2 | 2 | 1 | 1 | 7071066 | 32.8s | how long is the flight from chicago to cairo |
+| 342285 | no | 4 | 6 | 2 | 6 | 6 | - | - | - | 48.3s | how titanic facts |
+| 372586 | no | 4 | 6 | 2 | 6 | 5 | - | - | - | 49.3s | how to play blu ray discs |
+| 89786 | yes | 3 | 2 | 2 | 2 | 1 | 1 | 1 | 7071501 | 33.0s | central city definition |
+| 118448 | no | 4 | 4 | 2 | 4 | 1 | - | - | - | 42.5s | define body muscular endurance |
+| 92542 | yes | 4 | 5 | 2 | 5 | 4 | 1 | 1 | 7072003 | 45.6s | circulation money definition |
+| 206117 | yes | 5 | 8 | 3 | 8 | 3 | 1 | 1 | 7072155, 7072160 | 56.9s | hotels in thornton co |
+| 141472 | yes | 3 | 3 | 2 | 2 | 1 | 1 | 1 | 7072290 | 41.1s | derriere definition |
+| 293992 | no | 5 | 9 | 3 | 9 | 9 | - | - | - | 79.4s | how many product lines does coca cola have |
+| 196232 | no | 2 | 1 | 1 | 1 | 0 | - | - | - | 32.2s | government does do |
+| 352818 | no | 5 | 7 | 2 | 6 | 5 | - | - | - | 62.2s | how to cook string beans |
+| 45924 | yes | 5 | 7 | 3 | 7 | 6 | 5 | 7 | 5167800 | 62.5s | average temperatures las vegas by month |
+| 208145 | no | 5 | 9 | 3 | 9 | 9 | - | - | - | 71.0s | how bicycle tire tubes are sized |
+| 79891 | no | 5 | 7 | 2 | 7 | 3 | - | - | - | 60.9s | can you substitute chocolate chips for semi-sweet |
+| 208494 | yes | 4 | 5 | 3 | 5 | 5 | 2 | 2 | 7073272 | 51.1s | how big do newfypoo's get |
+| 319564 | no | 3 | 3 | 2 | 3 | 3 | - | - | - | 37.7s | how much fiber is in carrots |
+| 155234 | no | 3 | 3 | 2 | 3 | 2 | - | - | - | 40.0s | do bigger tires affect gas mileage |
+| 14151 | no | 5 | 13 | 3 | 13 | 12 | - | - | - | 83.9s | age requirements for name change |
+| 67802 | yes | 3 | 4 | 2 | 4 | 3 | 1 | 1 | 7074071 | 45.9s | can green tea cause stomach problems |
+| 1090184 | yes | 5 | 7 | 3 | 7 | 1 | 1 | 1 | 7074235 | 52.4s | synonym of subordinate |
+| 323382 | no | 3 | 6 | 2 | 6 | 1 | - | - | - | 45.1s | how much is the stamp to send a card |
+| 323998 | yes | 5 | 5 | 3 | 5 | 5 | 3 | 4 | 7074377 | 47.8s | how much magnesium in kidney beans |
+| 91711 | no | 5 | 8 | 3 | 3 | 2 | - | - | - | 60.6s | child psychiatrist salary 2016 |
+| 125898 | no | 5 | 6 | 2 | 5 | 5 | - | - | - | 57.9s | define prosthetic device |
+| 289812 | no | 3 | 3 | 2 | 3 | 3 | - | - | - | 32.6s | how many mm is a nickel coin |
+| 333486 | yes | 5 | 4 | 3 | 4 | 4 | 4 | 4 | 7075218 | 56.1s | how old do you have to be to get a job in idaho |
+| 1090171 | yes | 4 | 7 | 2 | 4 | 3 | 1 | 1 | 7075317 | 50.9s | synonyms for the word discipline |
+| 73257 | no | 5 | 6 | 2 | 6 | 6 | - | - | - | 48.7s | can seizure meds cause low sodium? |
+| 1090170 | no | 5 | 6 | 4 | 6 | 6 | - | - | - | 48.7s | synonyms for the word, decline |
+| 237373 | no | 5 | 9 | 3 | 9 | 9 | - | - | - | 64.1s | how is soil created from rocks |