How to extract dynamic "last user message" from chat history for Evaluation?

[VeronikaSus]

Describe your question

Hi everyone, I'm struggling with mapping the correct input for LLM-as-a-judge evaluators in a multi-turn chat application.

My trace input is a standard list of messages (System, User, Assistant, Tool, User...). The length of this array varies per trace.

// Example Trace Input
[
{"role": "system", "content": "..."},
{"role": "user", "content": "First question"},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "Review this please"} // <--- I need this dynamic last message
]

I need to map the last user message to the Evaluator's {{input}} variable. Since the conversation length varies, I cannot use a hardcoded index like input[1].content.

I tried using standard JSONPath features like negative indices: $.input[-1].content, but it seems Langfuse's JSONPath implementation doesn't support negative indexing or filtering.

My Question: What is the best practice for evaluating multi-turn conversations where the user input is at a dynamic index?

Is there a way to target the last item in the input array via Langfuse mapping?

Or should I pass the entire input JSON array to the LLM judge and let the prompt handle the extraction?

Thanks for any tips!

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

https://cloud.langfuse.com/project/cmgxcpp4u02x8ad06o4yxfpwi

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

There is no documented way to select “the last user message” via JSONPath in the evaluator mapping today, so you cannot reliably do this with something like $.input[-1].content in Langfuse. JSONPath is supported for nested extraction, but negative indices / multiple paths are not described as supported in the current docs.(1)(2)

From what is documented, there are two practical patterns:

1. Pass the full messages array into the evaluator and let the judge prompt pick the last user turn.
   In the evaluator template, you can define a variable like {{input}} and map it to the whole trace input (your messages array).(1)(3)
   Then, in the LLM-as-a-judge prompt, instruct the judge model to:

   • interpret {{input}} as the full chat history, and
   • base its evaluation specifically on the last user message in that history.
     This is explicitly aligned with how multi-turn evaluation is handled in the N+1 conversations cookbook, which evaluates responses at specific points in a conversation by feeding the relevant conversation history into the evaluator and letting the evaluation prompt reason over it.(4)

2. Pre-process / reshape your trace input before evaluation.
   When JSONPath alone is not enough (e.g., data is scattered or you need dynamic selection), the recommended approach is to pre-process your data so the relevant piece is available in a single field that you can map directly. For example, you could log an extra field like last_user_message into the trace input or metadata and then map {{input}} to that field with a simple JSONPath.(1)(2)

Given the current behavior:

• There is no documented support for multiple JSONPaths or aggregating several fields into one mapping parameter.(2)
• JSONPath is described as a way to “precisely locate” nested data, but only for a single expression per variable.(1)

So for your case (dynamic last user message in a variable-length messages array), the best-practice options within documented behavior are:

• Recommended: Map the full messages array to {{input}} and handle “last user message” in the LLM-as-a-judge prompt logic.(1)(4)
• Alternative: Adjust your application to store the last user message explicitly (e.g., last_user_message) and map that field directly with JSONPath.(1)(2)

There is no documented way to express “last element of the array” directly in the evaluator mapping UI today.

📚 Sources:

1. LLM-as-a-Judge
2. Having more than one JSONPath when mapping trace outputs to LLM-as-a-Judge parameter
3. langfuse/langfuse/web/src/features/evals/components/inner-evaluator-form.tsx
4. Evaluating Multi-Turn Conversations (N+1)

---

Have another question? Just tag @inkeep.

[marliessophie]

Hey @VeronikaSus was inkeep helpful here? Do you require further assistance?

[pankajrajputai]

hey how can we get set of messages I'm defining evals in which i wanted to define past messages for which i need all messages except last on, how to define this

[marliessophie]

Please check out JSONPath for details on syntax. You should be able to use slice with jsonpath $.content.message[0:-1], assuming you have a content key holding a message array.