[Bug]: Query returns empty results for flushed S3 blocks in Standalone mode (v2.10.0) despite correct retention and search config

[furkantarikgocmen]

Describe the bug
We are running Grafana Tempo v2.10.0 in Docker Standalone (monolithic) mode using an S3-compatible backend (RustFS). We are experiencing an issue where traces that have been successfully flushed to the S3 backend are not retrievable via search or direct trace ID lookup.

We can only see "live" traces that are currently in the Ingester (last ~30 minutes). Once the data is flushed to S3 (verified by existence of .parquet files and meta.json in the bucket), it becomes unsearchable via Grafana/Tempo.

We have attempted to tune query_backend_after and query_ingesters_until to bridge the gap between Ingester and Storage, but queries for older time ranges (e.g., last 6 hours) consistently return results only for the recent in-memory period.

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo v2.10.0 using the Docker Compose configuration below.
2. Send traces via OTLP HTTP (port 4318).
3. Wait for max_block_duration (1h) or trigger a flush. Verify data.parquet exists in the S3 bucket.
4. Query Tempo (via Grafana) for a time range covering "Last 3 Hours".
5. Result: Only the last ~30 minutes of traces are returned. Older data, although physically present in S3, is not returned.

Expected behavior
Tempo should seamlessly return traces from both the Ingester (RAM) and the S3 Backend. With query_backend_after: 30m, Tempo should query the S3 backend for any trace older than 30 minutes.

Environment:

• Tempo Version: grafana/tempo:2.10.0
• Deployment: Docker Compose (Standalone / Single Binary)
• Storage Backend: S3 Compatible (RustFS)
• Operating System: Linux

Additional Context

docker-compose.yml:

services:
  tempo:
    image: grafana/tempo:2.10.0
    container_name: tempo
    restart: unless-stopped
    command: ["-config.file=/etc/tempo/config.yml", "-config.expand-env=true"]
    env_file:
      - /etc/docker/envs/infradata/tempo-server.env
    ports:
      - "4318:4318"
      - "3200:3200"
    volumes:
      - ./config.yml:/etc/tempo/config.yml:ro
      - tempo_data:/var/tempo
    # ... networks and labels omitted for brevity

volumes:
  tempo_data:
    name: tempo_data

tempo config:

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 1h
  trace_idle_period: 10s
  flush_check_period: 10s

compactor:
  compaction:
    compaction_window: 24h
    max_block_bytes: 107374182400
    block_retention: 8760h
    compacted_block_retention: 1h

storage:
  trace:
    backend: s3
    wal:
      path: /var/tempo/wal
    blocklist_poll: 10s
    pool:
      max_workers: 100
    s3:
      endpoint: rustfs:9000
      bucket: tempo-traces
      access_key: ${RUSTFS_ROOT_USER}
      secret_key: ${RUSTFS_ROOT_PASSWORD}
      insecure: true
      forcepathstyle: true

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: infra
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://mimir:8080/api/v1/push
        send_exemplars: true
  traces_storage:
    path: /var/tempo/wal

overrides:
  defaults:
    global:
      max_bytes_per_trace: 500000000
    ingestion:
      rate_limit_bytes: 150000000
      burst_size_bytes: 300000000
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]

query_frontend:
  search:
    query_backend_after: 30m
    query_ingesters_until: 30m
    max_duration: 720h
  metrics:
    max_duration: 24h
    query_backend_after: 30m
    concurrent_jobs: 1000
    interval: 30m0s

[electron0zero]

Hey, your config looks correct to me.

My hunch is that the blocklist is not being built or fetched. Tempo's blocklist poller periodically calls ListObjectsV2 with prefix + start-after parameters to discover flushed blocks.

If this API call doesn't return results correctly, blocks exist in backend but Tempo doesn't know about them and can't query them.

RustFS is not one of the backend we test against, so we can't guarantee compatibility, and we have found issues in the past where some backends were not fully compatible with the S3 API.

Could you check:

1. Tempo logs around blocklist polling - look for "failed to poll blocklist", "failed to pull tenant index", or "writing tenant index"
2. Blocklist polling metrics from your Tempo instance - see polling docs for details on what to check

Also take a look at the troubleshooting querying docs for additional debugging steps.

if you want to check that RustFS works fine, you can run these tools against your RustFS instance:

• ceph/s3-tests - S3 API conformance suite
• minio/warp - S3 benchmarking tool to verify RustFS is healthy under load (run on a fresh bucket, it will wipe existing data in the test bucket)

[furkantarikgocmen]

Hey @electron0zero ,
I tried running it with Minio in the same configuration, unfortunately it's still the same. :(

also there is no "failed", "level=error" or "level=warn" logs for tempo, its writing files to minio successfully, seems like tempo did not querying s3 backend

[furkantarikgocmen]

By the way, we are using Loki with rustfs too, and its works as well, same environment, same docker stack, everything same except config, I have no idea what im missing

[electron0zero]

Hey @furkantarikgocmen, a few things to check here:

Try a known-good config

I'd suggest starting from our docker-compose examples as a known-good baseline config and adjusting the config from there and using the default config if you don't need to adjust something.

Check if Tempo's blocklist knows about your blocks by reviewing these metrics

tempodb_blocklist_length                      # should be > 0 if blocks are discovered
tempodb_blocklist_poll_errors_total            # should be 0
tempodb_blocklist_poll_duration_seconds        # should be completing successfully
tempodb_blocklist_tenant_index_age_seconds     # how stale the blocklist is
tempodb_blocklist_tenant_index_builder         # at least one component should have this as 1

See the polling docs for more details.

Use tempo-cli to inspect blocks directly

You can bypass Tempo entirely and query S3 directly with tempo-cli to confirm your blocks are valid and check their time ranges. If blocks show up with valid time ranges but Tempo still can't query them, the blocklist poller might be the problem.

See the tempo-cli docs for available commands.

Check bucket lifecycle policies

Make sure your MinIO/RustFS bucket doesn't have lifecycle rules that could be deleting blocks.

Also see the troubleshooting missing traces docs for additional debugging steps.