Cookbook: Haystack 2.x RAG — instrument → Observe traces → evaluate

[vedjaw]

Summary

Propose a new cookbook that walks through an end-to-end Haystack 2.x RAG path on Future AGI:

Haystack RAG pipeline → traceAI-haystack instrumentation → traces in Observe → at least one eval on answer / grounding quality

This was originally opened on the platform repo as future-agi/future-agi#1648. @hadarishav asked to move it here since cookbooks live in this docs repo. Happy to own the PR once scope looks good.

Why this cookbook

• Haystack already has auto-instrumentation docs (/docs/integrations/traceai/haystack, /docs/tracing/auto/haystack) and a traceAI-haystack package, but there is no cookbook parallel to LlamaIndex PDF RAG or LangChain RAG.
• Framework users typically land on cookbooks first; the integration page covers how to instrument, not how to go from first pipeline run to first useful eval.
• Closes the loop Future AGI advertises (observe → evaluate → improve) for the Haystack ecosystem specifically.

Proposed page

Field	Proposal
Path	src/pages/docs/cookbook/haystack-rag.mdx
URL	/docs/cookbook/haystack-rag
Sibling analogues	llamaindex-pdf-rag.mdx, rag-langchain.mdx, evaluate-rag.mdx
Related existing docs	/docs/integrations/traceai/haystack, /docs/tracing/auto/haystack

Frontmatter (draft)

title: "Haystack RAG with Future AGI Observability and Evaluation"
description: "Build a Haystack 2.x RAG pipeline, instrument it with traceAI-haystack, view traces in Observe, and evaluate answer grounding and relevance."

Scope (v1)

In scope

1. Minimal Haystack 2.x RAG
   • Retriever + generator (in-memory document store preferred so readers can run without external infra).
   • Small static corpus (a few short docs) checked into the cookbook or linked from a companion notebook/repo.
2. Instrumentation
   • fi-instrumentation + traceAI-haystack (HaystackInstrumentor) wired to Future AGI (cloud and/or self-host notes).
   • Env vars: FI_API_KEY, FI_SECRET_KEY, LLM provider key.
3. Observe
   • Steps to find the project, open a trace, and inspect retriever / LLM spans.
4. Evaluation
   • At least one built-in RAG-relevant eval (e.g. context adherence / answer relevance / groundedness-style metric), either inline eval tags on the tracer or a small post-run Evaluator path, matching patterns used in existing cookbooks.
5. Docs quality
   • Prerequisites, install, copy-pasteable Python, expected UI outcomes, troubleshooting table, next-steps links to Haystack integration + RAG eval cookbooks.
   • Register the page in navigation / cookbook index.

Out of scope for v1 (unless you prefer otherwise)

• Dataset export from traces for offline batch evals
• Citation / reference-range deep dive (related Haystack work: deepset-ai/haystack#11623)
• Multi-agent Haystack pipelines, production vector DBs, Gradio/Streamlit UI

Acceptance criteria

• Reader can run the example end-to-end with documented deps and env vars
• Traces appear in Observe for a sample query
• At least one eval result is visible (inline or experiment) for that run
• Page follows current cookbook conventions (frontmatter, no em-dashes, runnable code blocks, links to related docs)
• Linked from cookbook index / nav as appropriate

References

• Prior issue (closed, redirected here): Integration: Haystack 2.x end-to-end cookbook (instrument → trace → eval) future-agi#1648
• Integration page: https://docs.futureagi.com/docs/integrations/traceai/haystack
• Auto-instrumentation: https://docs.futureagi.com/docs/tracing/auto/haystack
• Package: https://github.com/future-agi/traceAI/tree/main/python/frameworks/haystack
• Closest existing cookbooks: LlamaIndex PDF RAG, LangChain RAG, Evaluating RAG

Questions for maintainers

1. Preferred path / slug: haystack-rag vs haystack-pdf-rag vs under cookbook/quickstart/?
2. Target surface: Future AGI cloud only for v1, or also call out self-host OTLP endpoint?
3. Eval style to match: inline eval_tags on register() (LangChain RAG style) vs separate Evaluator / experiment flow?
4. Any cookbook playbook / frontmatter schema (products, frameworks, difficulty, last-tested-with, etc.) I should follow for new pages right now?
5. Should companion code live only in the MDX, or also under future-agi/cookbooks (as with some LlamaIndex pages)?

Offer

I can open a PR against dev with a first draft once you confirm path + eval style. Starting point will be a minimal in-memory Haystack RAG + HaystackInstrumentor + one grounding/relevance eval, then iterate on review feedback.

[khushalsonawat]

Hi @vedjaw,
Thank you for opening this issue. Really appreciate your help here. Docs are one areas we're actively accepting contributions in. I'm leading the work on docs and will be reviewing any work that goes into the docs for the time being. However, there are certain guidelines that we follow while writing docs and cookbooks. Please allow me a couple of days to formally update those guidelines in the repo so that you can refer to it and write your cookbook in the expected format. I'll also review the scope of the issue and suggest changes if any.

Thank you