Sequence of the query level evaluators

[shankdubey11]

Describe your question

I am confused that what is the sequence of the evaluators will be while executing the test? Is it is sequential or will be parallel.

For example:
evaluators = [evaA, evaB, evaC, evaD]

One thing I know is, the questions will be running parallely, but don't have idea about the evaluators, can you please help me out?

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the available documentation, evaluators in Langfuse experiments run in parallel for each individual dataset item(1).

When you define multiple evaluators like [evaA, evaB, evaC, evaD], all four evaluators execute concurrently for each test case. The experiment framework handles concurrent execution automatically(1).

The parallel execution applies at two levels:

1. Dataset items run in parallel - Multiple test cases from your dataset execute concurrently (controlled by max_concurrency parameter)(1)
2. Evaluators run in parallel - For each individual item, all evaluators execute concurrently(1)

This means if you have 10 dataset items and 4 evaluators, the framework efficiently processes multiple items at once, and for each item, all 4 evaluators run simultaneously rather than sequentially.

The framework also provides error isolation, so if one evaluator fails, it doesn't stop the other evaluators or terminate the experiment(1).

📚 Sources:

1. Experiments via SDK

---

Have another question? Just tag @inkeep.

[shankdubey11]

Thank you for clarifying that evaluators run in parallel. This brings up a follow-up question regarding dependencies between them.

I have a scenario where the output of one evaluator is needed as an input for other evaluators, all running on the same dataset item. For example:

• evaluatorA runs first to extract a list of keywords from the model's output.
• evaluatorB and evaluatorC then need to use that specific list of keywords to perform their own logic.

Since the evaluators run in parallel, evaluatorB and evaluatorC would not have the output from evaluatorA when they start.

Does Langfuse support this pattern? Is there a way to define an execution dependency or share state between evaluators for a single test case?

[jannikmaierhoefer]

@shankdubey11, Langfuse does not support this pattern yet. Feel free to add it to our public roadmap here: https://langfuse.com/ideas