Tracking LLM usage

[sky-2002]

Hi haystack team,
I have been using haystack for my workflows and I have components that make LLM API calls.
I wanted to ask how do I add tracking for these components so that I can see usage for every run.
Please point me to any resources or docs if any.

[sky-2002]

Also is there a way to cache component outputs, for use cases like starting from earlier state if some component fails etc, would love to know.

@julian-risch

[sjrl]

Hey @sky-2002 glad to hear you've been using Haystack! I'd recommend checking out our different tracing options like Langfuse. Depending on the LLM components that you are using you should see token tracking and cost tracking within the langfuse dashboard.

[sjrl]

For starting from an earlier state we do have a pipeline breakpoints feature coming soon to Haystack. You can find an experimental implementation of it in our haystack-experimental repo. Here is the discussion on it and here is a google colab for it.

[sky-2002]

Hey @SRJL
Thanks for the info, will check the options and breakpoints feature, thanks for the links and resources

[sjrl]

@sky-2002 moved this to be a discussion since that feels more relevant than an issue currenty

[onestardao]

Awesome to see this thread — I've run into this exact pain point while building multi-hop RAG pipelines.

Tracking usage is one thing... but making sense of which step derailed the semantic flow is a whole different beast, especially once dynamic retrieval + fallback reranking enters the picture.

We’ve been testing a “semantic traceback” system (kind of a WFGY-style symbolic path debugger) — it lets you not only track LLM/token costs, but reconstruct where the logic flow diverged, even when caching kicks in or prompts vary subtly.

If helpful, here’s how we do it:
https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md

(esp. Problem #17, which covers caching misalignment + shadow trace fallback)

Still early stage, but might be useful if you’re trying to debug RAG workflows with more than one failpoint. Happy to trade notes!