[Docs] Add RAG / reasoning failure checklist for AI Hedge Fund research flows #491
Description
Hi,
I am the author of WFGY ProblemMap, a 16-problem failure map for RAG and LLM agent pipelines.
It has already been adopted or cited by RAGFlow, LlamaIndex, ToolUniverse (Harvard MIMS Lab), Rankify (Univ. of Innsbruck), the Multimodal RAG Survey (QCRI LLM Lab), Awesome LLM Apps, and appears in Awesome AI in Finance, so it has seen use across different parts of the RAG and agent ecosystem.
In an AI-driven hedge fund setting, people increasingly use LLMs to support research, strategy explanation, risk notes, and documentation. Typical failures include:
- narratives that sound plausible but ignore part of the data
- confusion between similar tickers or different time windows
- long explanations that drift away from the original hypothesis
- difficulty telling whether a bad suggestion is caused by retrieval, reasoning, or tooling
These align with several WFGY problems such as No.1 hallucination and chunk drift, No.2 interpretation collapse, No.3 long reasoning chains, No.4 overconfident bluffing, No.8 black-box debugging.
I would like to add a docs-only checklist for these cases. For instance:
docs/research/rag-failure-checklist.md
The document would:
- summarise the 16 failure modes in language that fits quant research and strategy development
- give short examples drawn from research notebooks or pipeline runs
- suggest how to tag incidents or strategy post-mortems using the checklist
- reference the full WFGY map at:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md
This would stay purely in documentation, without changing any code.
If this feels aligned with the goals of the project, I can open a small docs PR for review.