feat: Customizable Annotation View for Complex Traces

[hprzegendza]

Describe the feature or potential improvement

Problem:

When using frameworks like LangGraph, the resulting traces in Langfuse can be highly complex, deeply nested JSON objects. This is ok for debugging but makes annotation inefficient. Annotators have to manually dig through the JSON to find the relevant fields to score, which doesn't scale.

Here's a screen showing the typical trace complexity and how it looks in the annotation view:

Proposed Solution:

We propose a configurable "Annotation View" where we can define which parts of the trace object to display.

The UI for this could work just like the existing "Custom Evaluators" and "Variable mapping" when setting them up. The ability to define variables by selecting a trace object (Input, Output, Span) and providing an optional JSONPath seems like a good fit here.

By applying this to the annotation queue setup, an administrator could define all the necessary fields an annotator needs to see for a specific task. For example, we could configure a view with several custom-named variables:

Display Label	Mapped To
Question	(Object: Trace, Variable: Input, JsonPath: $.messages[0].content)
Parsed Question	(Object: Span, Name: problem_normalizer, Variable: Output, JsonPath: $.update.problem_description)
Reasoning	(Object: Span, Name: execute_query, Variable: Input, JsonPath: $.reason)
Response	(Object: Trace, Variable: Output, JsonPath: $.messages[-1:].content)

Example:

This way, annotators see only the fields they need for the task, instead of having to parse the full JSON trace by hand. It would make annotation faster and less error-prone.

Additional information

No response

[marcklingen]

Thanks for sharing! We are considering adding the option of custom rendering frames that get the full trace object and can map it to html. let's see how many people are interested in this here.

[Xuanaaa]

Looking forward to it, we have many scenarios that are limited by this scenario and cannot access Langfuse.

[Xuanaaa]

If we have HTML rendering capabilities, we can render JSON data using HTML to preview annotations. Additionally, we can evaluate multimodal requirements, including input and output data such as images and videos ~~

[maxdeichmann]

Thanks for the addition. We will consider this as we build this.

[hammadmlk]

When is this coming?

[maxdeichmann]

We do not have a clear timeline for this unfortunately but this is high on our priority list.