Description

When using LangChainInstrumentor with async LangChain/LangGraph applications, spans are not properly linked in a parent-child hierarchy. All spans appear as direct children of the root span instead of forming a nested trace tree, making it impossible to visualize the actual exec flow.

Note: Spans share the same Trace ID, but the parent-child relationships are incorrect (flat instead of nested).

Environment

• splunk-otel-instrumentation-langchain: 0.1.x
• langchain-core: 1.x / langgraph: 1.x
• opentelemetry-sdk: 1.39.x
• Async execution via await graph.ainvoke() or await chain.ainvoke()

Steps to Reproduce

1. Instrument an async LangChain/LangGraph application with LangChainInstrumentor
2. Execute an agent using await graph.ainvoke(...)
3. Export spans to Jaeger/Zipkin/Splunk O11y
4. Observe that all spans are siblings with the same parent (flat hierarchy)

Expected Behavior

POST /agents/query (trace_id: 90a9d9...)
└── invoke_agent healthcare_agent (parent: HTTP span)
    ├── step model (parent: agent)
    │   └── chat claude-3-5-sonnet (parent: step)
    └── step tools (parent: agent)
        └── tool query_member_data (parent: step)

Actual Behavior

POST /agents/query (trace_id: 90a9d9...)
├── invoke_agent healthcare_agent (parent: HTTP span)
├── step model (parent: HTTP span)      ← WRONG! Should be child of agent
├── chat unknown_model (parent: HTTP span)  ← WRONG! Should be child of step
├── step tools (parent: HTTP span)      ← WRONG! Should be child of agent
└── tool query_member_data (parent: HTTP span)  ← WRONG! Should be child of tools

All spans have the same Trace ID but all point to the HTTP root span as parent instead of their logical parent.

Root Cause Analysis

In callback_handler.py, the parent_run_id UUID is captured but never resolved to an actual OpenTelemetry Span object. The span emitter code in span.py checks for parent_span:

parent_span = getattr(invocation, "parent_span", None)
parent_ctx = trace.set_span_in_context(parent_span) if parent_span else None

When parent_span is None, parent_ctx becomes None, and the span is created with the current active context. In synchronous code, this works because Python's context propagation maintains the parent. But in async code with await, the context is lost between coroutines, causing all spans to inherit from the outermost context (HTTP request span).

# Current code - parent_run_id captured but not resolved to span
inv = LLMInvocation(
    parent_run_id=parent_run_id,  # UUID stored but not used
    # parent_span is never set!
)

Evidence

Tested with healthcare agent deployed on Snowflake SPCS:

• Trace ID: 90a9d9e471e7ba2a1c88f96e2bc46293
• 14 spans all showing parentId: daaa62fa3f11aae6 (the HTTP root span)
• UI shows flat waterfall with all spans at same indentation level

{
  "spanId": "daaa62fa3f11aae6",
  "parentId": null,
  "operationName": "POST /agents/sf-query"
}

Full trace JSON: trace_flat_hierarchy_bug.json

Impact

• ❌ Trace visualization broken - flat hierarchy instead of nested tree
• ❌ Cannot trace execution flow - unclear which LLM call belongs to which step
• ❌ Agent Flow view broken - Splunk O11y Agent Flow shows incorrect structure