Langfuse
Inquiry regarding multiple Human Annotations per single trace #11822
-
|
Hello, We are currently setting up the Human Annotation feature in Langfuse and recently conducted a dry run to validate our workflow. Our goal is to have multiple team members evaluate the same trace within the same annotation queue. However, during our testing, we noticed that we could not persist separate evaluations for different users on the same trace. Instead, we observed that when a second user submits an evaluation, it appears to overwrite the previous user's entry. Could you clarify if there is a specific configuration or workflow recommended to allow multiple independent evaluations for a single trace? Thank you for your help. Langfuse Cloud or Self-Hosted?Langfuse Cloud If Self-HostedNo response If Langfuse CloudNo response SDK and integration versionsNo response Pre-Submission Checklist
|
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
Currently, Langfuse does not support multiple users annotating the same trace independently within the same annotation queue.(1) When a second user submits an evaluation on the same trace, it overwrites the previous user's entry rather than creating separate annotations.(1) Workaround: One team has implemented a workaround by creating separate score configs with naming conventions that align with each annotator (e.g., using initials to denote who is assigned).(1) This allows you to have the same scoring rules and setup, but with different score config names to differentiate between annotators.(1) You can then filter traces to compare annotations across users.(1) While this approach doesn't solve the problem at scale, it can be practical as long as your team of reviewers is small enough to manage these score configurations efficiently.(1) This limitation has been noted as a significant gap that affects the value proposition of using Langfuse for human annotation compared to other approaches.(1) 📚 Sources: Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @lucasboulle, inkeep is correct -> "creating separate score configs with naming conventions that align with each annotator". |
Beta Was this translation helpful? Give feedback.
-
|
hey @jannikmaierhoefer, what does 'score config' mean? i was thinking about creating multiple queues (one per annotator) for the same traces. does that make sense? |
Beta Was this translation helpful? Give feedback.
-
|
@lucasboulle a score config defines the score you are attaching and is required for an annotation queue: https://langfuse.com/docs/evaluation/experiments/data-model#score-config Afaik, having multiple queues might not solve the issue as the score will be overwritten. But please test this, as I am not 100% sure about that. |
Beta Was this translation helpful? Give feedback.
Currently, Langfuse does not support multiple users annotating the same trace independently within the same annotation queue.(1) When a second user submits an evaluation on the same trace, it overwrites the previous user's entry rather than creating separate annotations.(1)
Workaround:
One team has implemented a workaround by creating separate score configs with naming conventions that align with each annotator (e.g., using initials to denote who is assigned).(1) This allows you to have the same scoring rules and setup, but with different score config names to differentiate between annotators.(1) You can then filter traces to compare annotations across users.(1)
While this approach doesn'…