fix: honor required retrieval for evidence requests

[hedhoud]

Problem

OpenRAG's chat pipeline asks the query contextualizer whether a message needs document retrieval before it searches the selected partition. That decision is appropriate for greetings and casual conversation, but it is unsafe for evidence evaluation: a scientific claim can be sent as a statement with no question mark, and the classifier can label it conversational.

In the SciFact evaluation on September 10, 2026, this caused 71 of 300 claim requests to return no retrieved sources. The requests completed without HTTP or chat errors, so the failure was silent. Direct vector search had candidates for all 71; for 55 claims a gold document was in the top 50, including the expected document at rank 1 for 31 claims.

The failure occurs before vector search and reranking. When the classifier returns requires_retrieval: false and query_list: [], QueryService builds a direct answer with an empty context. The original evaluation did not record the classifier decision, but replaying the affected claims on develop reproduced this behavior for 70 of 71 claims. Calling retrieval directly with the same original claims returned 10 chunks each time.

For example, the claim Albendazole is used to treat lymphatic filariasis. had gold document 1215116. Direct search ranked 1215116 first, while the chat request returned all_retrieved_sources: [] and no error because retrieval was skipped.

Solution

This change adds the opt-in request metadata flag require_retrieval. When it is exactly JSON true, partition-backed chat, streaming chat, and text completions must attempt retrieval even if the contextualizer says to skip. If the classifier returns no query, the service searches the original user input. Existing partition, workspace, attachment, and temporal-filter restrictions remain in effect. The default remains false, so normal casual conversation keeps its current behavior.

The contextualizer prompt also explicitly identifies factual and scientific claims as retrieval requests. The prompt improvement helps ordinary clients, while require_retrieval provides a deterministic contract for evaluations and evidence-verification workflows.

The option guarantees an attempted search, not a relevant result or a correct answer. The response can still contain no matching evidence, and retrieval relevance and citation generation remain separate metrics.

Example

Request:

{
  "model": "openrag-scifact",
  "messages": [
    {"role": "user", "content": "Albendazole is used to treat lymphatic filariasis."}
  ],
  "metadata": {
    "require_retrieval": true
  }
}

With the option enabled, a requires_retrieval: false classifier response no longer ends the request. OpenRAG falls back to the claim itself, searches the scifact partition, and exposes the retrieved sources to the answer pipeline.

Validation

• 722 relevant unit and API tests pass.
• Independent review found no correctness, security, or regression issues.
• A 300-claim replay with require_retrieval: true attempted retrieval for 300/300 requests, returned sources for 300/300, and recovered sources for all 71 previously empty cases.
• In that replay, 258 claims had a gold document in the top 10 retrieved results and 211 answers cited a gold document. These figures are descriptive because the original sampling settings were not recorded; no answer-correctness judge was run.

Fixes #949.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}