Visualize prompt caching performance in the trace view

[CNSeniorious000]

Describe the feature or potential improvement

We need to visually debug prompt caching performance. Specifically, we want the content prefix to remain consistent across requests. However, due to the complex templating process used in rendering the final prompt, it’s hard to identify differences between requests. I propose adding a "show cached prefix" toggle in the trace and playground views. When activated, the cached segments would be dimmed, and a counter would indicate the number of tokens in the cached part. The cached prefix can be calculated by comparing across all "generation" observations from the last 5 minutes.

Additional information

The count result don't need to be as accurate as the value counted from the LLM provider. I just need a brief view of which part of the prompt began changed in the new generation request.

[marcklingen]

This is a great suggestion.

Why would you go for computing the cached context based on the observations of the last 5 minutes instead of using the "cached_tokens" value included in the LLM response?

[CNSeniorious000]

Because I don't just want to see a number; I want to see exactly which part of my entire prompt has changed, leading to a cache miss starting from this token.

This idea struck me because, in one of my projects — where the LLM autonomously executes code, with the prompt beginning with a series of few-shot examples containing both the code and its execution results — I noticed that the cached_token count was unexpectedly low. It wasn’t until later that I realized the few-shot section was being re-executed with every request, causing some hashes to change (since Python’s default repr for objects includes their memory address id).

In the end, I had to manually log the full prompt and use the diff view in my IDE to identify the issue. If I could intuitively spot this on the observation platform, that would be fantastic.