Bulk Add Observations to Human Evaluations with Fine-Grained Scoring

[dajiba6]

Describe your question

I've run into an issue where I need to add multiple observations to human evaluations, but currently I can only add them one by one manually from within each trace—this is very slow and inefficient.
if I add an entire trace directly to the evaluation queue, the resulting score and comments apply to the whole trace. This lacks sufficient granularity: since a trace may contain multiple distinct steps (e.g., retrieval, reasoning, tool calls, final response), evaluating it as a single unit makes feedback imprecise and less actionable.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.143.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Langfuse supports adding observations to annotation queues for human evaluation, providing the fine-grained scoring you need(1).

You can bulk add observations to an annotation queue by selecting multiple observations via checkboxes, clicking the "Actions" dropdown menu, then selecting "Add to queue"(1). This allows you to efficiently add multiple observations at once rather than adding them one by one from within each trace(1).

When you add observations to an annotation queue, each observation becomes its own annotation task(1). This means you can score and comment on individual steps within a trace (like retrieval, reasoning, tool calls, or final response) separately, giving you the granularity you're looking for(1).

For programmatic bulk operations, you can manage annotation queues via the API(1). This allows you to scale and automate your annotation workflows(1).

To set this up:

1. Create an annotation queue with the score configs you want to use(1)
2. Bulk select the observations you want to evaluate(1)
3. Add them to the queue via the Actions dropdown(1)
4. Process the queue, scoring each observation individually(1)

This approach addresses both your efficiency concern (bulk adding) and your granularity concern (scoring individual observations rather than entire traces)(1).

📚 Sources:

1. Annotation Queues

---

Have another question? Just tag @inkeep.

[dajiba6]

In the observability UI, the smallest selectable unit is the trace—I can't multi-select individual observations. Is there a way to multi-select traces but, when importing them into human annotation, allow reviewers to add comments and scores at the observation level?

[dajiba6]

I found the solution—the multi-select for observations is in the trace's secondary menu.