Proposal: Implement "Local Hybrid Search" (BM25) to enable offline agent reasoning tests

[AviJxn]

Context

I've been following the recent discussions on enabling local development without GCP credentials (referencing the need for a MockRetriever). While setting up a mock is essential to prevent crashes, a static mock that returns synthetic/dummy data has a major limitation: it makes it impossible to test the Agent's reasoning logic or hallucination checks offline.

If the retriever returns random text ("Lorem ipsum"), the LLM cannot perform meaningful multi-hop reasoning, rendering local testing ineffective for logic improvements.

Proposed Solution

I propose implementing a LocalRetriever fallback in backend/retrieval.py that activates automatically when self.is_enabled is False (i.e., when GCP keys are missing).

Instead of returning dummy data, this system would:

1. Load a local local_knowledge.json file (which mirrors the production BigQuery schema).
2. Perform a lightweight BM25 / Keyword Search on the title and content chunks.
3. Return contextually relevant results to the LLM based on the user's query.

Technical Approach

I have already prototyped a working script (local_search.py) that implements this logic using standard Python libraries (no heavy vector DB dependencies required).

The flow would be:

• Check Env: If GCP is missing -> Initialize LocalRetriever.
• Search: Tokenize query -> Score documents based on term frequency (Title + Content) -> Sort by relevance.
• Output: Return List[RetrievedItem] identical to the Vertex AI response.

Benefits

• True Logic Testing: Developers can verify if the agent correctly summarizes/synthesizes data without spending cloud credits.
• Hallucination Debugging: We can test if the agent correctly identifies "Undisclosed" information using a controlled local dataset.
• Zero Cost: Fully offline and free.

I have the prototype ready and can submit a PR to integrate this into backend/retrieval.py.

[DhanashreePetare]

Greetings @AviJxn. I am interested to work on the issue. Can you please assign me the same.ThankYou

[AviJxn]

Hi @DhanashreePetare, thanks for the interest!

I actually opened this proposal to track the implementation of the Offline Search architecture I designed. The code is already complete and submitted in PR #6, which is currently awaiting mentor review.
Since this task is effectively done, I’d suggest checking the good first issue or the main knowledge-space repository for other open tasks.
Feel free to check out PR #6 if you're curious about how the BM25 fallback works!

[DhanashreePetare]

okay @AviJxn, ThankYou

[QuantumByte-01]

Hi,
This is not required for online usage, where datasets are stored remotely and accessed via GCP. The offline BM25 fallback is primarily useful for local development and testing.

You could keep this in a separate testing folder by importing the required functions from the main codebase. The application is fully developed and tested using GCP APIs.

[AviJxn]

Understood @QuantumByte-01 . I see your point about keeping the production agent code strictly focused on the Cloud/Vertex AI flow.
I will refactor this to live entirely within a tests/ or local_dev/ directory as a standalone utility. This way, it can still help new contributors test locally (by importing the necessary graph classes) but it won't touch or clutter the main application logic at all.
I'll push the refactored changes to this branch shortly.