Follow-up to the RAG Failure Mode Checklist docs update: a visual debug companion

[onestardao]

Hi folks, quick follow-up for LlamaIndex builders who are already running RAG in real projects.

I keep seeing the same pattern:

Your LlamaIndex app runs.
Indexing succeeds.
Retrieval returns something.
The query engine completes.
But the final answer is still off-topic, unstable across runs, or clearly wrong in production.

Recently I submitted a docs PR that extends the existing RAG Failure Mode Checklist with several production-focused failure families, without changing the existing sections.
PR: #20760

The added sections are specifically aimed at the “everything looks healthy, but answers are still wrong” stage, including:

• embedding / metric mismatch issues
• session and cache related memory drift
• observability gaps (hard to tell what actually happened during a run)
• index lifecycle / deployment ordering pitfalls

That checklist format works well, but in practice many people want an even faster entry point when something breaks.
So I built a lightweight companion: a single visual card you can use as a debug prompt.

RAG 16 Problem Map · Global Debug Card

This is not a replacement for LlamaIndex.
It is meant to be a simple debugging layer you can use after your LlamaIndex app is already running, when you have a real failing run and you need a clearer path to “what likely went wrong and what to try next”.

How to use (super simple)

1. Save the card image .

2. Take one failing run from your app and summarize it briefly:

   • what the user asked
   • what your app retrieved or attempted to use
   • what the final answer was
   • why you believe it is wrong

3. Upload the card image + paste that failing run summary into any strong LLM, then ask:
   “Please follow this debug card to identify the likely RAG failure modes and suggest concrete fixes + quick verification checks.”

I have tested this workflow with ChatGPT, Claude, Gemini, Perplexity, and Grok.
They can all read the card and use it to classify common RAG failures and propose reasonable next-step fixes.

If you are a LlamaIndex user and you have ever hit problems like:

• retrieval looks fine but the answer still hallucinates
• the same question gives inconsistent quality across runs
• production Q&A drifts even though the pipeline “works”
• you need a shared failure vocabulary for debugging with your team

…then this might be useful.

global debug card + short README are here:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/wfgy-rag-16-problem-map-global-debug-card.md

If you try it on a real broken LlamaIndex run, I’d love to hear what failure modes it flagged and whether the suggested fixes helped.