Evaluators Running Twice on Parent Trace and Child Observations - Creating Duplicates

[PalvikaTR]

Describe the feature or potential improvement

We're experiencing an issue where evaluators are executing twice - once on the parent trace and once on child observations - resulting in duplicate evaluations and duplicated calls to LLM-as-a-judge.

Evaluator settings:

• New dataset run items
• Question is picked from input of Dataset item
• Answer is picked from Trace Output

Current Behavior:
Single trace with parent trace and child observation structure
Evaluators run on both parent and child levels
Duplicate evaluations are created
Existing filters are not preventing this duplication

Expected Behavior:
Evaluators should run only once
No duplicate evaluations

Attempted Solutions:
Tried using various filter combinations without success to filter either parent of child observation
Filters don't distinguish between parent traces and child observations effectively

Additional information

No response

[maxdeichmann]

@PalvikaTR, thanks a lot for pointing us to this issue. We will look into this after the christmas break

[qa-ailabs]

I encountered the same problem; the same trace was run twice in the evaluator.

Evaluator runs on New traces

[PalvikaTR]

Also attaching scores tab screenshot where you would same trace one with no observation id and another one with observation id present on which same evaluator ran twice and gave exact same score to parent and child observation trace, comment/reasoning varies slightly though.

[PalvikaTR]

@maxdeichmann Did you get a chance to look at this? Also want to mention that we are not doing live tracing, we have run experiments on dataset through sdk. Please let me know if you need any other information

[hassiebp]

@PalvikaTR Could you please share the project ID and the Langfuse region you are deployed to?