Help with service graphs with Tempo..

[LRagji]

Can someone help me with generating service graphs, seems like it newer works. every time i visit Explore on Grafana it says not enabled :( below is my config for tempo

# Tempo configuration file for grafana/otel-lgtm
# Enables service graphs + traces ingestion via OTLP.

server:
  http_listen_port: 3200
  grpc_listen_port: 9096

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "127.0.0.1:4417"
        http:
          endpoint: "127.0.0.1:4418"

ingester:
  trace_idle_period: 1s
  max_block_duration: 1s
  flush_check_period: 1s
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    min_ready_duration: 1s

querier:
  frontend_worker:
    frontend_address: 127.0.0.1:9096

storage:
  trace:
    backend: local
    wal:
      path: /data/tempo/wal
    local:
      path: /data/tempo/blocks

metrics_generator:
  processor:
    service_graphs:
      histogram_buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5]
    span_metrics:
      dimensions:
        - service_name
        - operation
        - status_code
  traces_storage:
    path: /data/tempo/generator/traces
  storage:
    path: /data/tempo/generator/wal
    remote_write:
      - url: http://127.0.0.1:9090/api/v1/write
        send_exemplars: true

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

[martincostello]

The configuration above seems correct based on the documentation for Tempo 2.9.x.

Can you confirm which version of the LGTM container image you're using?

[LRagji]

@martincostello i am using the latest, btw i ran an example from @opentelemetry/instrumentation-express and then the service-maps seems to be displaying, does this mean that spans need to have some special kind of info attached to them like "Kind"?... also there is a bug in @opentelemetry/instrumentation-express which doesn't work with express 5.0 which is confusing the situation more...

[zeitlinger]

does this mean that spans need to have some special kind of info attached to them like "Kind"?

yes, I think it's only generated for client and server spans

... also there is a bug in @opentelemetry/instrumentation-express which doesn't work with express 5.0 which is confusing the situation more...

Our example uses 5.0.0 - did you try that? https://github.com/grafana/docker-otel-lgtm/blob/main/examples/nodejs/package.json#L38

[LRagji]

Ohk let me retry if there have been changes, cause it was not reporting any spans from express library only http library was reporting spans and new express was eating way parent trace ids