Skip to content

Tempo Search/Drilldown shows “ ” even though root span exists in trace #7494

Description

@vinothlk06

Description
We are observing an issue in Grafana Tempo where some traces appear in Explore Search and Traces Drilldown with:

Service: <root span not yet received>
Name: <empty>

However, when opening the same trace by Trace ID, the trace is fully available and contains a valid root span.
Because of this, Grafana Traces Drilldown is unable to populate the “Trace Service” and “Trace Name” columns for affected traces.
Environment

Tempo Version: 2.10.5
Deployment: tempo-distributed (Helm)
Storage Backend: Azure Blob Storage
Grafana Version: 13.0.1
Single tenant deployment
Metrics Generator enabled
Search backend: Parquet storage (default configuration)

Observed Behavior
For some traces:
Search Results / Drilldown

Service: <root span not yet received>
Name:

Opening the Trace
The trace opens successfully and shows a valid root span, for example:

envoy: ingress
└─ router apps_80 egress
    └─ <Application Name> POST /*

The root span is visible and contains:

service.name = envoy
span.name    = ingress
span.kind    = server

Expected Behavior
If the root span exists in the stored trace and is visible when opening the trace, Search and Drilldown should populate:

Trace Service = envoy
Trace Name    = ingress

instead of:
<root span not yet received>

Same service shows mixed behavior
Using TraceQL:
{ resource.service.name = "envoy" }
Results contain a mix of:
Working traces:

Service: envoy
Name: ingress

and affected traces:

Service: <root span not yet received>
Name:

This occurs for the same service (envoy).
Older traces remain affected
The issue is not transient.
Affected traces from previous days continue to show:
<root span not yet received>
even though opening the trace displays the root span correctly.
Trace data is present
Affected traces:

  • Exist in storage
  • Can be opened by Trace ID
  • Display a complete trace tree
  • Display the root span
  • Display valid service.name
  • Display valid span name
  • Only Search/Drilldown metadata appears incomplete.

Relevant Tempo Configuration
Ingester

trace_idle_period: 60s
max_block_duration: 5m
flush_check_period: 10s
complete_block_timeout: 3m
flush_all_on_shutdown: true

Storage

storage:
 trace:
   backend: azure

Metrics Generator

metrics_generator:
 processor:
   local_blocks:
     flush_to_storage: true

No advanced routing/sampling
Not using:

  • tail_sampling
  • loadbalancing exporter
  • Kafka pipeline
  • Metrics Observed

Tempo is actively ingesting spans and creating traces:
tempo_distributor_spans_received_total
and
tempo_ingester_traces_created_total
Both increase normally.
No evidence of ingestion failure.

1 - sum(rate(tempo_warnings_total{reason="disconnected_trace_flushed_to_wal"}[5m])) / sum(rate(tempo_ingester_traces_created_total{}[5m]))

Image

sum(rate(tempo_warnings_total{}[5m])) by (reason)

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleUsed for stale issues / PRs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions