[Bug]: CondensePlusContextChatEngine.astream_chat silently aborts generation and yields 'Empty Response' when Retriever returns 0 nodes

[jefersonlop3s]

Bug Description

When using CondensePlusContextChatEngine with an asynchronous stream (astream_chat), if the provided Retriever (e.g., QueryFusionRetriever or VectorIndexRetriever) returns 0 nodes (which can happen frequently with valid metadata filters like tenant IDs), the Chat Engine silently aborts the LLM generation process.

Instead of passing the system prompt and the user query to the LLM with an empty context, the synthesizer (called via _arun_c3 -> asynthesize) evaluates to an empty node list and completely skips the LLM API call. It instantly returns a hardcoded "Empty Response" string wrapped in an AsyncStreamingResponse.

This is highly problematic for production RAG systems (like multi-tenant architectures), where a user might ask a general question or have an empty vector space on their first day. Instead of the LLM answering naturally leveraging its baseline knowledge and the System Prompt, the application receives a silent "Empty Response" in less than 1 second, with no exceptions raised, masking the behavior.

llamaindex_bug_report.md

Version

Python 3.12.x LlamaIndex v0.10.x LLM Provider: Agnostic (Tested with Ollama, but reproducible via OpenAI due to synthesizer logic)

Steps to Reproduce

import asyncio
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.chat_engine import CondensePlusContextChatEngine
from llama_index.llms.ollama import Ollama # or OpenAI

async def main():
# 1. Create an empty index (or one where filters will yield 0 nodes)
index = VectorStoreIndex.from_documents([])
retriever = index.as_retriever(similarity_top_k=2)

# 2. Setup LLM
llm = Ollama(model="qwen2.5:0.5b") # Ensure model is running locally

# 3. Build CondensePlusContextChatEngine
engine = CondensePlusContextChatEngine.from_defaults(
    retriever=retriever,
    llm=llm,
    system_prompt="You are a helpful AI."
)

# 4. Attempt to trigger Async Stream
print("Sending query...")
chat_stream = await engine.astream_chat("Hello, can you help me?")

full_text = ""
async for token in chat_stream.async_response_gen():
    full_text += token
    
print(f"Final Response: '{full_text}'") 
# Expected output: "Hello! Yes, I can help you..."
# Actual output: "Empty Response" (Instantly, no LLM call dispatched)

if name == "main":
asyncio.run(main())

Relevant Logs/Tracbacks

INFO:src.agent:📂 Initializing local search tool (RAG)...
INFO:src.agent:   ✓ RAG tool initialized successfully
INFO:src.engine_builder:⚙️  Configuring Hybrid Search (Vector + BM25)...
INFO:src.engine_builder:   🔍 Tenant Filter applied: Jeferson
WARNING:src.engine_builder:   ⚠️  No documents found in ChromaDB for BM25.
INFO:src.engine_builder:   🧠 Engine initialized via OLLAMA with model qwen2.5:0.5b

[STARTING ASTREAM_CHAT TRACE]

INFO:llama_index.core.chat_engine.condense_plus_context:Condensed question: Hello Sovereign! This is a test being sent directly from N8N via Webhook in the Cybrid network.

# Execution HALTS in less than 1.2s without dispatching any POST request to /api/chat

==== DEBUG RESPONSE ====
TYPE: <class 'llama_index.core.chat_engine.types.StreamingAgentChatResponse'>
FULL TEXT FINAL: 'Empty Response'
chat_stream.response: 'Empty Response'

[dosubot]

Hi @jefersonlop3s! I'm Dosu and I’m helping the llama_index team.

This behavior is actually intentional by design, not a bug. The BaseSynthesizer checks if len(nodes) == 0 and returns a hardcoded "Empty Response" without calling the LLM—this was designed to avoid costly LLM calls when no relevant context is found. A maintainer acknowledged this as expected behavior.

That said, I understand this is problematic for multi-tenant RAG systems. Here are your options:

Simplest Workaround — Post-process the stream:

async for token in response.async_response_gen():
    if token == "Empty Response":
        yield "I couldn't find relevant information in your documents."
    else:
        yield token

PR #20503 — A configurable empty_response parameter was merged in January 2026. If you're on a version that includes this, you may be able to customize the empty response message. However, CondensePlusContextChatEngine hardcodes the CompactAndRefine synthesizer, so you'd need to subclass it to take full advantage.

Alternative: The Generation synthesizer removes the empty nodes check entirely and always calls the LLM, but using it requires building a custom chat engine since CondensePlusContextChatEngine doesn't accept custom synthesizers.

Given your use case (first-day users with empty vector spaces), you may want to check what version you're running and whether PR #20503 is available, or implement the post-processing workaround in the meantime.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[jefersonlop3s]

Hi @dosu, thank you so much for the detailed explanation! Upstaged by a feature, not a bug! It`s ok.. believe!

While I completely understand the design choice to save LLM/Compute costs for standard QA systems, this aggressive short-circuiting is extremely problematic for conversational, Multi-Tenant architectures like ours (where a "Day 1" user expects the AI to interact conversationally even if their dedicated vector space is currently empty).

Regarding the suggested workarounds:

1. Post-processing / PR feat: add configurable empty response message to synthesizers #20503 (empty_response): Yielding a hardcoded string completely kills the conversational vibe and the System Persona. The user just wants to chat with the AI, but gets a robotic static string instead.

Our Custom Workaround (The Bypass): Instead of subclassing the engine or yielding a static string, we implemented a custom interception at our FastAPI/Orchestrator layer. When the achat / astream_chat yields exactly "Empty Response", we instantly catch it, extract the engine._memory, prepend the engine._system_prompt, and dispatch a raw payload directly to the foundation LLM class (await engine._llm.astream_chat(messages)).

This approach completely bypasses the CompactAndRefine synthesizer constraint, restoring the LLM's full capability to act as an ungrounded conversational agent when RAG context is 0, preserving the UX without touching the core library.

Thanks again for the lightning-fast clarification! Feel free to close the issue or use this conversational-fallback mindset as inspiration for future synthesizer updates!

[dosubot]

Glad the investigation helped clarify things! Your custom workaround is really clever—intercepting the "Empty Response" at the orchestrator layer and falling back to a raw await engine._llm.astream_chat(messages) call with the memory and system prompt is an elegant solution that preserves the conversational experience without modifying the library.

This pattern (graceful degradation from grounded RAG to ungrounded conversational mode) is definitely a valuable approach for multi-tenant systems. Thanks for sharing it—it could help others facing the same Day 1 user scenario! 🙌

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[logan-markewich]

The intention here is thay a query engine SHOULDNT be responding with knowledge outside of its knowledge base. Silently responding with the LLM directly is not great, and empty responses are easy to detect if a dev wants to implement a general response