DeepScholar-Bench as a first-class task on the shared paper-search agent scaffold - #318
Draft
yilunzhao wants to merge 38 commits into
Draft
DeepScholar-Bench as a first-class task on the shared paper-search agent scaffold#318yilunzhao wants to merge 38 commits into
yilunzhao wants to merge 38 commits into
Conversation
Auditing a real 200-generation run surfaced scaffold leaking into the text handed to both judges: every gemma-4 report carried channel markers (three were nothing but markers yet were judged as reports), and one Qwen report ended in dangling tool-call tags. Extraction now removes tool-call, function, parameter, channel and role markers deterministically, keeps only final-channel content while preserving text written before it, and never alters code spans. Also stop the generation instruction from teaching a citation form models copy verbatim: the example placeholders can no longer be echoed as a usable citation, and clean_citation_url strips angle brackets so CommonMark autolinks validate instead of silently dropping out of the FACT denominator.
…ilures A live run produced a validation list of the right length whose items lacked the 'result' key; the KeyError escaped and the harness zeroed the whole instance, discarding its already-computed RACE scores (the official pipeline crashes outright on the same input). Malformed items now retry and fall back to all-unknown like other parse failures, and an unexpected FACT error zeroes only the FACT channels for that instance.
(cherry picked from commit ceb8aa3)
(cherry picked from commit 91b861c)
(cherry picked from commit b563f80)
(cherry picked from commit 2d9b7d6)
(cherry picked from commit 48c1c7b)
(cherry picked from commit 197b09e)
(cherry picked from commit 28b33f2)
(cherry picked from commit 7140aa6)
(cherry picked from commit 65979c4)
(cherry picked from commit f9b2403)
(cherry picked from commit 8820a47)
FACT extracts every (statement, reference, url) triplet from a report, deduplicates them, crawls each cited URL and judges support one statement at a time. RACE needs one judge call per instance. The cost difference is orders of magnitude, and there was no way to ask for RACE alone. Right now most of what FACT would crawl is already known to be fabricated: one run cites www.example.com 1,023 times and collapses 638 arXiv citations onto 32 distinct IDs, so scoring it would spend heavily to re-confirm a defect that was established for free by reading the generations. Gate it on DEEPRESEARCH_SKIP_FACT rather than a TaskConfig field, because TaskConfig is the shared base for every task and a knob only this one understands does not belong there. Default is unchanged, so an unset environment still runs both. A skipped run records fact_details as {"fact_scoring": "skipped"} and logs a warning naming the count, because zeroed metrics that silently mean "not run" are exactly what made an earlier all-zero column unreadable.
researchqa.py already reads OLMO_EVAL_JUDGE and falls back to its default spec. This task hardcoded gpt-5.5 for RACE and gpt-5.4-mini for FACT, so scoring the two benchmarks in one comparison with the same judge meant editing a constant. Read the same variable here, keeping both defaults and both effort levels, so an unset environment behaves exactly as before.
Generation and scoring are coupled in `run`, so obtaining judge scores for work already finished meant regenerating it: burning GPU to produce different answers and then scoring those instead of the ones that were analysed. One DeepResearch Bench cell takes about 17 hours to regenerate for the heaviest system. Match on a sha256 of the canonical request payload, tagged chat or text, which is exactly what the runner writes to requests JSONL after applying harness config. Tools, sampling params and system prompt are excluded from the key because they describe how to generate rather than which instance; any collision they would have disambiguated is caught at load, where two entries sharing a key with different stored text is an error. The request-to-prediction join runs on doc_id with native_id as a witness, so a one-off shift between the two files raises instead of answering every instance with its neighbour's report. A missing prediction aborts the batch and names whether the instance was unknown or merely absent. There is no flag to continue past it, because there is no honest value to return. A prediction that is genuinely an empty string replays as empty and is counted separately; a null or absent one counts as missing. logprobs raise rather than fabricate. Trajectories are re-attached to output metadata: DeepResearch Bench does not read them, but litsearch and sage score against response.trajectory, which the runner rebuilds from that field. Round-trips 100/100 byte-identical against both real DRB prediction files, with negative controls for a dropped row and a one-character prompt change. The requests side of that check was reconstructed through the task's own format_request and the runner's build_requests, since no saved requests file was available on the host; a mismatch against genuine artifacts would surface as a loud coverage error rather than a wrong score.
Our runs drew roughly 5,000 HTTP 429s from Semantic Scholar. Answers survived, because the retry ladder absorbs them, but each exhausted retry sleeps through a 1/2/4/8/16 second backoff and one system's batches ran 35-170x slower than the single-agent pipelines. Read S2_API_KEY plus any numbered S2_API_KEY_<n>, each of which may itself hold a comma-separated list, because Beaker mounts secrets one value per variable while a local .env is easier as one string. Blanks are dropped and duplicates collapse, so the same key configured twice cannot inflate the request rate. Selection cycles rather than drawing at random: the limit is metered per key, and independent draws land on the same key twice in a row one time in N, re-firing the 429 the second key was added to avoid. The cursor starts at a random offset so separate processes do not begin on the same key. Rotation alone would have changed nothing. The rate gate enforced a process-global 1.1 second floor between all requests, so two keys would have split one key's budget; the interval is now divided by the number of configured keys. Four requests take 3.30s on one key and 1.65s on two, while each key still sees at most one request per 1.1 seconds. Behaviour with zero or one key is unchanged, including the keyless public-API path. Serper is deliberately untouched: no rate-limit failures have been observed there, it bills per account rather than per key, and its call sites error on a missing key rather than falling back silently. This assumes the keys belong to separate accounts. If they share a quota the gate now pushes twice as hard into it, which the retry ladder would absorb but which is worth knowing.
When vLLM runs from /opt/vllm-venv, only /opt/venv/bin is on PATH, so console scripts installed next to vLLM are invisible to the server process. flashinfer JIT-compiles kernels on the first forward pass by shelling out to a bare "ninja", which execvp resolves through PATH. Models with JIT-compiled kernels (Mamba/GDN linear attention) therefore killed the engine with "FileNotFoundError: ninja" right after the server had passed its readiness probe, failing every instance while the job still reported success. Prepend the interpreter's bin directory to the child PATH, never replacing it so the main venv stays reachable, warn at startup when ninja is still not resolvable, and install ninja into the isolated venv explicitly so the fix does not depend on a transitive dependency of whichever vLLM build lands there.
Gantry derives a job's code version from the git repository rooted at the
launching process's working directory: launch_experiment falls through to
GitRepoState.from_env, which does Repo(".") and takes str(repo.commit()).
olmo-eval never passed gantry's own ref parameter, so the same command run from
a second checkout of this repo launched that checkout's HEAD instead, with
nothing in olmo-eval's output to show it.
Add BeakerJobConfig.git_ref, forwarded to launch_experiment as ref, and log the
commit that will actually be cloned, including the working directory it was
derived from, before every launch. Note that passing a ref also disables
gantry's dirty-repo guard, which only runs when ref is None.
An inference server that dies mid-run leaves every instance failing inside the per-instance error handler, so the process exited 0 with an empty metrics file and Beaker reported the job SUCCEEDED. Results are already written and uploaded by the time run() returns, so failing here costs no artifacts.
…e time The base image is a CUDA runtime image and ships no nvcc, so FlashInfer's JIT path cannot succeed on it at all. Making ninja reachable only moved the failure from "ninja: not found" to "/usr/local/cuda/bin/nvcc: not found" inside the ninja build, still mid-run and still after the readiness probe passed. flashinfer-jit-cache ships those kernels prebuilt. When it is installed, JitSpec.is_aot is true and build_and_load() loads the .so directly, so ninja and nvcc are never invoked. Install it into the isolated vLLM venv, deriving the version from the flashinfer and CUDA versions actually present rather than pinning one, because flashinfer refuses to import when the two disagree. The step runs after provider overrides and never fails the job. Verified against flashinfer-jit-cache 0.6.6+cu128: it contains gdn_prefill_sm90/gdn_prefill_sm90.so, the .so needs only libcudart/libcuda/libc and no toolchain, and its cubins are sm_90a.
The prompt hardcoded an "Available tools" block naming serper_google_webpage_search and serper_fetch_webpage_content and told the model to search and fetch with them. Under paper_search_agent, which exposes only semantic_scholar_snippet_search, the model obeys the prompt over the tool schema and calls a tool that does not exist; the openai_agents scaffold then ends the episode with "[Tool error: Tool serper_google_webpage_search not found in agent openai_agents]" and an empty trajectory, so every instance scores near zero without measuring anything. Carry the harness tool names into TaskConfig and build the prompt's tool list and workflow steps from them, falling back to the previous hardcoded list when the harness exposes no tools. dr_tulu exposes exactly those three tools, so its prompt and every non-agentic prompt stay byte-identical; a test pins that. Verified locally against Qwen3.5-35B-A3B with paper_search_agent on 10 instances: before, 10/10 answers were the tool-not-found error with turns=[]; after, 10/10 are ~2k-character cited answers whose trajectories contain only semantic_scholar_snippet_search calls.
ResearchQA hardcoded an Available tools block naming the two serper tools, so under paper_search_agent the model called a tool that does not exist and every instance ended before its first turn. The merged branch builds that list from the harness tool names instead, leaving dr_tulu and no-tool runs byte-identical.
…on test DeepScholar-Bench credits a source only if it carries arXiv provenance and was published strictly before the query's cutoff, so a search tool that shows the model anything else is inviting citations the scorer will throw away. arxiv_paper_search reuses the Semantic Scholar client, keys, rate gate and backoff already in this module and changes only what it admits: arXiv provenance decided client-side on externalIds.ArXiv, never on the venue field, which an arXiv preprint loses the moment it appears at a conference. The date bound is the half S2 can enforce, pushed one day short of the cutoff because its ranges are inclusive and the benchmark's test is strictly-before; the client-side pass repeats it and falls back to the month the arXiv ID encodes when S2 reports no date. Because provenance has no server-side form, the requested page is widened fivefold and trimmed after filtering, which preserves S2's relevance order rather than re-ranking it. Results carry the arxiv.org/abs URL, the only form the benchmark's citation parser credits. Hits without an arXiv ID are not silently dropped -- a couple are shown marked as context the answer cannot cite, since a search that returns nothing is indistinguishable from a broken one. The new preset stays separate from paper_search_agent rather than joining it: which search tools an agent holds changes what it retrieves, so sharing the preset would move litsearch and SAGE numbers measured against S2 search.
63 arXiv papers, each supplying its abstract and its own publication date; the
model writes that paper's Related Works section citing only arXiv work
published before that date. The per-instance cutoff reaches the search tool
through retrieval_date_cutoff, so retrieval is constrained the same way the
scorer is.
The prompt is byte-identical to the one lit-agents runs, which is the only
reason the two systems' numbers can be compared at all; a golden-string test
pins it so a reworded prompt fails loudly instead of silently producing
incomparable results.
Scoring is deliberately a placeholder. The published metrics are a second pass
outside this repo, and deepscholar_export writes the per-query
{intro.md, final_report.md, paper.csv} folders that pass reads. What the task
does compute is whether a response is exportable at all, so a run that
generated nothing cannot be mistaken for one that scored zero.
paper.csv holds every arXiv source the tool showed, not only the cited ones,
because the contract the scorer checks is that cited IDs are a subset of the
file. Dates come from the arXiv ID and so carry month precision -- the same
fallback lit-agents uses, and all the rendered tool output can support.
paper.csv carried month precision for every source, because the month was all the arXiv ID could encode. The benchmark contract rejects a month-precise source dated inside the cutoff's own month, so any paper published in that month was discarded -- not because it failed the policy, but because the export had thrown away the day Semantic Scholar had already returned. arxiv_paper_search now renders a Published line and says which kind of date it is: "Published: 2025-04-03" when S2 dated the paper, "Published: 2025-04 (month precision)" when only the arXiv ID could, and no line at all when neither can. The export parses it straight into date_precision, falling back to the ID's month for predictions saved before the line existed. The preset still declares no S2 secret. required_secrets is a hard launch gate with no optional form, and declaring one would make a keyless run impossible, so a key stays a per-run --secret-env mapping; the docstring now says so instead of leaving it to be discovered.
…e it The export scored zero and the tests agreed with it. DeepScholarBaseParser credits a citation only as a markdown link whose URL matches arxiv.org/abs, and returns no documents at all for a query without one. The prompt -- correctly, verbatim -- mandates "[3]"-style numbered citations. Every folder this exporter wrote was therefore unscoreable, and exportable_rate reported otherwise because it looked for an arxiv.org URL anywhere in the text rather than for a citation the parser would see. deepscholar_citations ports render_intro's semantics: strip the reference list, resolve each inline citation, rewrite the resolved ones as [Title](https://arxiv.org/abs/<id>), delete the rest. What it cannot port is where the numbering comes from -- lit-agents' graphs publish a citation_order, and nothing here does -- so the answer's own reference list is parsed back out and matched to retrieved sources by arXiv ID, then by title. intro.md and final_report.md are the rewritten text; exportable_rate runs the same core and counts the answers it succeeds on. Version suffixes are normalised in generated URLs and when matching, because the parser keys its reference map on the raw URL text and compares it against paper.csv's normalised id: a v2 would silently resolve to an empty title. The admission test now mirrors _classify_source properly. A hit missing a title or an abstract is rejected rather than shown, since it can only become a paper.csv row that fails the contract. A publicationDate that is present but unreadable is rejected too, instead of falling back to the month its arXiv ID encodes -- that fallback is for papers Semantic Scholar cannot date at all, and reaching for it here admits post-cutoff papers on the strength of a date nothing read. paper.csv carries only cited sources, its snippet is the full abstract re-fetched from Semantic Scholar rather than the truncated preview the agent saw, and snippet_source records the rows where that lookup failed. export_manifest.json, summary.json and generation_manifest.json mirror lit-agents' own so the output can go to its preflight and not only to the parser. Sources are read only from the search tool's own results; another tool naming an arXiv ID never showed the agent a citable paper. IDs are read from their own line, so an abstract containing the block separator costs a fallback snippet rather than the source. A prediction without a positional native_id now raises: doc_id is a within-run counter, and falling back to it scores one paper's answer against another paper's ground truth. The pinned parser is vendored into tests/evals/tasks/fixtures and the end-to-end test runs it, because asserting this against a hand-written imitation would only prove the imitation self-consistent.
Two findings from 200 real recorded queries, both about the gap between what this tool is and what it looked like. Only 19.4% of the rows S2 returns carry an arXiv external ID. One widened page was sized for a density that does not exist: 77 of the 200 queries yielded fewer than ten arXiv candidates, and 11 came back empty from the filter alone, having had rows to filter. The tool now pages -- offset 0, 100, 200 -- stopping as soon as it has what the caller asked for, when S2 returns a short page, or when the budget runs out. Three pages of 100 expects around 58 arXiv rows at the measured density, and offset 200 plus limit 100 stays inside S2's cap of 1000. Pages are consumed in order and appended, so relevance order survives. A page that fails after the first is logged and the earlier pages are still returned; only an immediate failure has nothing to report. All 12 operator-bearing queries in that sample came from the sol single agent, and 8 of them returned zero rows: site: filters, quoted phrases, trailing "arXiv" hints. The model thought it was talking to a web search engine, and nothing in the tool description told it otherwise. Now it does. The query still goes to S2 exactly as written. Stripping operators in code would turn a measured, attributable failure into a silent rewrite, and the next person reading a trajectory would not be able to see what the model actually asked.
Twelve findings from a second review, most of them variations on one theme: the export trusted things it had not checked. The strictness was in the wrong layer. The tool rejected a publicationDate it could not read, which contradicted the admits() mirror it claimed -- the reference parser returns None for an unreadable date exactly as it does for an absent one, and falls back to the month the arXiv ID encodes in both cases. Retrieval is now lenient again and the strict test moved to where it belongs: the exporter revalidates every cited row against the query's cutoff, so a paper admitted on its ID month and then dated by the batch re-fetch to after the cutoff no longer exports as a success. Dropping a row used to leave its citation standing in intro.md, which is a folder that contradicts itself -- the contract checks that every cited ID has a row, and the parser would render a citation with an empty title and snippet. The rewrite now re-runs with the failed source withheld, so the citation leaves with the row. Identity was inferred from any line shaped `arXiv: ID`, including lines inside an abstract, so a paper could rename itself and a context-only result could smuggle itself back in as citable. Identity now comes from a block's header alone, and the renderer puts every identifying field before the abstract so the one field a paper controls the text of is terminal. Only the exact heading `References` was stripped, so a `## Bibliography` tail stayed in the prose and every `[1]` in it became a fabricated citation. Broadened to the headings models actually write, with a warning when the stripped heading is not the one lit-agents would have stripped. Also: rendered links are no longer reprocessed into `[Title](url)(url)`; `[1-3]` expands and `[^1]` resolves, while superscripts and author-year brackets are counted into a metric rather than silently scoring zero; pagination dedupes repeated papers and rotates keys per page instead of hammering one at the interval meant for the whole set; a late page of unreadable JSON no longer discards the pages that worked; an unreadable prediction row stops the export instead of quietly shrinking it; and a non-empty output directory is refused unless --force clears it, because a stale folder fails preflight in the name of a run that is not at fault. The reference validators are vendored into the fixtures and run against real exports. The suite had been exercising our own reimplementation of the contract, which could only ever prove it self-consistent.
BeakerStatusReporter guarded on whether a Beaker client could be configured, but a workstation with beaker set up passes that test and then crashes the worker on the missing BEAKER_WORKLOAD_ID -- which is the condition that actually decides. Check the workload id first; the reporter is a documented no-op everywhere else.
… error
gpt-5.6-sol refuses function tools and a reasoning effort together on
/v1/chat/completions, and nothing on this branch could say otherwise. The runs
that looked like they had said otherwise were carrying
OLMO_EVAL_REASONING_EFFORT, which appears nowhere in src/ and read nothing; they
worked because the server default happened to agree at the time.
scaffold_kwargs already reaches the scaffold, so model_settings goes there and
build_model_settings turns it into the SDK's ModelSettings. reasoning_effort is
accepted as the flat key an operator types and becomes
ModelSettings.reasoning.effort, which OpenAIChatCompletionsModel sends as a
top-level reasoning_effort argument. It is deliberately not routed through
extra_args: that call site splats extra_args in next to its own explicit
reasoning_effort= keyword, so the two would collide on this exact key. An
unknown key raises rather than being dropped, because a knob that silently does
nothing is the failure this replaces.
Nothing is pinned in the preset -- backbones that reason well should keep doing
so -- and the docstring names the flag instead:
-o scaffold_kwargs.model_settings.reasoning_effort=none
Separately, the response-shape check added last round was wrong about what
Semantic Scholar sends. A query matching nothing comes back as
{"total": 0, "offset": 0} with no data array, which the check read as a
malformed response and reported to the model as a search error. A search that
found nothing and a search that broke are different things to an agent choosing
what to do next, so the missing array is now an empty page. A data field that is
present and not a list is still an error.
Both were caught by running the thing rather than testing it: the first live
run died because this commit's own helper had been inserted between
@register_scaffold and the class it decorates, so the registry handed every
worker the helper. Direct imports of both names kept working, which is why the
unit tests were happy.
openai-agents 0.7.0 rejects openai>=2.54 (InputTokensDetails gained a required cache_write_tokens field). The ~=2.21 pin sat only in the clients extra, which Beaker launches never install, so containers resolved 2.54.0 and every agent call failed with a ValidationError while local venvs kept the working combination. Same lesson as the lit-agents agents extra, applied to the extra the launcher uses.
This reverts commit 8426736.
openai-agents 0.7.0 rejects openai>=2.54 (InputTokensDetails gained a required cache_write_tokens field). The only pin lived in the clients extra, which Beaker launches never install, so containers resolved 2.54.0 and every agent call failed with a ValidationError while local venvs kept the working combination. The litellm extra now carries the pin and is declared in conflict with openhands, whose openai==2.8 cannot coexist; the lock re-resolves cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
DeepScholar-Bench as a first-class olmo-eval task, so the same single-agent scaffold that runs litsearch / SAGE / the other paper-search benchmarks also runs DeepScholar — one arm, comparable across all six benchmarks — while every published number still comes from the benchmark's own evaluator.
Three pieces:
arxiv_paper_searchtool — searches Semantic Scholar and keeps only arXiv-provenanced papers (client-sideexternalIds.ArXivfilter), with a per-instance date cutoff so the agent cannot cite work newer than the query paper. Overfetches and pages (offsets 0/100/200) until it has arXiv papers to show. Two-tier output: citable results carryarxiv.org/abs/<id>links; context-only results don't.arxiv_paper_search_agentharness preset — the shared openai_agents scaffold with only this tool. Kept separate frompaper_search_agentso litsearch/SAGE numbers measured against Semantic Scholar search don't move.deepscholar_benchtask +deepscholar_exportadapter — the task prompts the upstream generation prompt verbatim over the 63 benchmark queries. The adapter turns each prediction into the exact folder layout upstream'sDeepScholarBaseParserreads (final_report.md/intro.md/paper.csv), resolving the model's citations ([n], footnotes, bare URLs) against the trajectory's retrieved sources and rewriting them as[Title](arxiv.org/abs/<id>)links — the only citation form the official scorer credits. Queries whose answer cites nothing resolvable are excluded and counted inexportable_rate; task-level metrics are bookkeeping only, never benchmark scores.Scoring stays out of band: the export is fed to the pinned official evaluator (
guestrin-lab/deepscholar-bench@c95413b3,python -m eval.main), so reported metrics come from the benchmark's own instrument.Why Semantic Scholar as the arXiv backend
Paired replay of 200 real agent queries against both backends: S2 gold-hit@10 24.2% vs 19.7% for the arXiv API (n=198, every backbone stratum favors S2). The deciding issues are behavioral: the arXiv API ANDs every query word over metadata and returns confident junk instead of an empty list when nothing matches; its
submittedDatefilter is silently dropped when combined with multi-word queries (a one-day window returns 646k results with a five-word query, 1 result with a single word); and its rate limiting 429s well-spaced requests after a burst. The approach follows the S2-with-arXiv-filter shim inroryd/deepscholar-bench-s2patch, reimplemented as a harness tool for the agent side.Status
Draft on purpose: the branch predates current main (known conflict surface:
pyproject.toml,uv.lock,cli/beaker/launch.py,evals/tasks/common/base.py,runners/asynq/preparation.py,tests/launch/test_beaker.py); a sync onto main incl. #300 is queued behind an in-flight full-63 run and will update this PR. Opening now to anchor the design discussion — in particular how this unified-agent arm and the externaldeepscholar_bencheval (which drives the upstreamdeepscholar_basereference pipeline) should coexist and be labeled.Validated end-to-end on Beaker: smoke and full-63 generation + official scoring for Qwen3.5-9B (task tests green; suite green at branch tip).