Help with the LLM As A Judge Evaluation

[rohitkumar-vc]

Describe your question

I have created a FASTAPI application in which there are nested LLM calls. I want to run evaluator on those nested llm calls. for better understanding of the call structured please refer to the image:

Here nested function calls joke function and then joke function calls the LLM. later the nested function calls poetry function and then in turn it again calls the LLM. I want to run evaluation on the test (name given by me in config run_name langchain).

In each trace object there can be multiple test (LLM Calls), I want for each trace object each LLM within that I should be able to run the evaluator on.

As of now I tried selecting the object as generation then I selected name as test, this works but it is just considering the last LLM call of the trace for the evolution.

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

To evaluate each nested LLM call separately in your trace, you should use observation-level evaluators instead of trace-level evaluators(1).

Based on your setup where you have multiple LLM generations (named "test") within a single trace, here's what you need to do:

Solution: Use Live Observations Evaluation

When setting up your LLM-as-a-Judge evaluator, select "Live Observations" as your evaluation target instead of "Live Traces"(1). This allows you to:

1. Target specific observations - Filter by observation type (Generation) and observation name ("test")(1)
2. Evaluate each LLM call separately - Each generation observation will be evaluated independently, not just the last one(1)
3. Apply flexible filtering - Combine trace-level and observation-level filters to narrow down exactly which observations to evaluate(1)

Configuration Steps

1. Navigate to the Evaluators page and create a new evaluator(1)
2. When choosing which data to evaluate, select "Live Observations"(1)
3. Set filters:
   • Object Type: Generation(1)
   • Object Name: "test" (or whatever name you've given your LLM generations)(1)
4. Map your variables (input/output) from the observation properties(1)

Important Note for SDK Users

If you're using the OpenTelemetry-based SDKs (Python v3+ or JS/TS v4+), observation-level evaluators are the recommended approach(1). If you need to filter observations by trace-level attributes like userId, sessionId, or tags, you must use propagate_attributes() in your instrumentation code to make those attributes available on observations(1).

This approach will evaluate both of your "test" generations (the one in call_joke_llm and the one in call_poetry_llm) separately, rather than just the last one.

📚 Sources:

1. LLM-as-a-Judge

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @rohitkumar-vc, inkeep is correct. Currently the evaluator can only target the first generation with the same name in a trace. However, we are releasing observation-level evals in the next couple of days that allow you to evaluate both LLM observations. Please keep an eye on our changelog for this.