Feature Request: Display Full Session Context When Annotating Individual Observations in Annotation Queue

[CrashingRain]

Describe the feature or potential improvement

Problem Statement

When using Annotation Queues to evaluate individual observations (e.g., intent recognition nodes, answer generation nodes) within multi-turn conversational AI applications, annotators cannot see the full session context (previous conversation turns). This severely limits the ability to make accurate, context-aware evaluations.

Current Behavior

What works:

• Can add individual observations to Annotation Queue for evaluation[(1)](https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues)
• Can add sessions to Annotation Queue (added in v3.87.0, July 2025)[(3)](https://langfuse.com/changelog/2025-07-28-sessions-in-annotation-queues)

What doesn't work:

• When annotating an observation, only that observation's data is visible
• Multi-turn conversation history from the parent session is NOT displayed
• Annotators must manually navigate to the session view to see context, then return to the annotation queue

Detailed Use Case

Application Structure:

Session (multi-turn conversation)
├── Trace 1 (QA Round 1)
│   ├── Observation: Intent Recognition
│   ├── Observation: Retrieval
│   └── Observation: Answer Generation
├── Trace 2 (QA Round 2)
│   ├── Observation: Intent Recognition
│   ├── Observation: Retrieval
│   └── Observation: Answer Generation
└── Trace 3 (QA Round 3)
    ├── Observation: Intent Recognition
    ├── Observation: Retrieval
    └── Observation: Answer Generation

Evaluation Requirements:

1. Need to evaluate specific observations (e.g., "Intent Recognition" in Trace 2)
2. Need to see previous conversation turns (Trace 1, and earlier parts of Trace 2) to understand context
3. Need to score different observation types separately (intent recognition vs. answer generation require different scoring criteria)

Why Current Solutions Don't Work:

Approach	What You Can Do	What You Can't Do
Annotate Session	See all traces in session	Cannot score individual observations; only session-level scoring
Annotate Trace	Score the entire trace	Cannot see previous traces in the session; cannot score specific observations
Annotate Observation	Score specific observation	Cannot see any session context or previous conversation turns

Real-World Impact

Example Scenario:

User (Turn 1): "我的血压有点高，医生让我少吃加工食品"
Bot (Turn 1): "听起来确实需要注意，医生有给具体建议吗？"

User (Turn 2): "没听太懂...算了，我压力大，想吃点舒服的。能给我推荐个芝士玉米片的食谱吗？"
Bot (Turn 2): [Intent Recognition Node Output] → "用户想要: 芝士玉米片食谱"

Evaluation Challenge:

• When evaluating the "Intent Recognition" observation in Turn 2, annotators need to see Turn 1 to understand that the user has dietary restrictions
• The intent recognition should ideally flag the conflict between the user's health needs and their request
• Without Turn 1 context, this evaluation is impossible

Proposed Solutions

Option 1: Contextual Annotation View (Preferred)
When annotating an observation in Annotation Queue:

• Display the observation's data (current behavior)
• Add a collapsible "Session Context" panel showing:
  • All previous traces in the same session
  • Chronological conversation flow
  • Ability to expand/collapse for focus

Option 2: Configurable Context Display

• Allow users to configure which parent data to display (similar to JSONPath selectors in LLM-as-a-Judge evaluators)
• Example: Show session.traces[*].input and session.traces[*].output alongside the current observation

Option 3: API Enhancement

• Extend the Annotation Queue API to accept custom context data
• Allow passing additional display data when creating annotation queue items (not just objectId reference)

Related Issues & Discussions

This need has been raised by multiple users:

• [#8485 - Simultaneous Session and Trace Annotation](https://github.com/orgs/langfuse/discussions/8485) (Aug 2025) - User wants to see session context when annotating traces
• [#7593 - Customizable Annotation View for Complex Traces](https://github.com/orgs/langfuse/discussions/7593) (Jul 2025) - Users with LangGraph need custom views to display relevant fields from complex nested traces

Why This Matters

For Conversational AI Evaluation:

• Intent recognition quality depends on conversation history
• Answer relevance requires understanding previous turns
• Safety checks need full context (e.g., user mentioned health restrictions earlier)

For Annotation Efficiency:

• Reduces context-switching between annotation queue and session views
• Improves annotation accuracy and inter-annotator agreement
• Enables scalable evaluation workflows for multi-turn applications

For Langfuse Adoption:

• Many LLM applications are conversational (chatbots, assistants, agents)
• This limitation blocks adoption for context-dependent evaluation use cases
• Competitors may offer better support for multi-turn annotation

Current Workarounds (All Inadequate)

1. Store conversation history in observation metadata - Clutters data model, not scalable
2. Annotate at session level only - Cannot evaluate specific nodes/observations
3. Manual context lookup - Annotators must open session view separately, very inefficient
4. Use external annotation tools - Defeats the purpose of using Langfuse's built-in annotation features

Timeline Context

• July 2025: Session-level annotation support added[(3)](https://langfuse.com/changelog/2025-07-28-sessions-in-annotation-queues)
• August 2025: Community raised need for session context in trace annotation[(7)](https://github.com/orgs/langfuse/discussions/8485)
• February 2026: Still no solution for observation-level annotation with session context

---

Additional information

No response

[marliessophie]

Thanks for sharing this very detailed feature request. I'd be curious to learn more. Would you be happy to hop on a quick 10 minute user interview with me so I can understand what UI would solve your use case? If yes, please send an email to marlies@langfuse.com and we can set up a time.