Unable to push traces to Tempo Collector using Open Census port: Corrupted Message

[cajual]

Edited for updated information:

I have been unable to successfully push traces to Tempo Collector using Open Census port with the error corrupted message.

Turning on debug gives me:

[     5.266472s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}: tower::balance::p2c::service: updating from discover
69
[     5.266524s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}: linkerd_reconnect: Disconnected backoff=true
68
[     5.266532s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}: linkerd_reconnect: Creating service backoff=true
67
[     5.266541s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}: linkerd_proxy_transport::connect: Connecting server.addr=172.20.141.106:55678
66
[     5.266746s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}: tower::balance::p2c::service: updating from discover
65
[     5.267539s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}:h2: linkerd_proxy_transport::connect: Connected local.addr=10.200.47.23:51016 keepalive=Some(10s)
64
[     5.267578s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}:h2: rustls::client::hs: No cached session for DnsName(DnsName(DnsName("collector.monitoring.serviceaccount.identity.linkerd.cluster.local")))
63
[     5.267654s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}:h2: rustls::client::hs: Not resuming any session
62
[     5.268726s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}: tower::balance::p2c::service: updating from discover
61
[     5.268757s]  WARN ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}: linkerd_reconnect: Failed to connect error=received corrupt message
60
[     5.268765s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}: linkerd_reconnect: Recovering
59
[     5.268771s] DEBUG ThreadId(01) opencensus:opencensus{peer.addr=tempo-distributed-distributor.monitoring:55678}:controller{addr=tempo-distributed-distributor.monitoring:55678}:endpoint{addr=172.20.141.106:55678}: linkerd_reconnect: Disconnected backoff=true

However, I do see that jaeger is working just fine:

❯ linkerd jaeger check
linkerd-jaeger
--------------
√ linkerd-jaeger extension Namespace exists
√ jaeger extension pods are injected
√ jaeger injector pods are running
√ jaeger extension proxies are healthy
√ jaeger extension proxies are up-to-date
√ jaeger extension proxies and cli versions match

Status check results are √

And I do see that the edges are encrypting just fine:

SRC                                   DST                                   SRC_NS           DST_NS       SECURED
my-service                            tempo-distributed-distributor         my-service-ns    monitoring   √
tempo-distributed-distributor         tempo-distributed-metrics-generator   monitoring       monitoring   √
tempo-distributed-metrics-generator   mimir-distributed-nginx               monitoring       monitoring   √
tempo-distributed-querier             tempo-distributed-query-frontend      monitoring       monitoring   √

Which leads me to believe there is either an issue within the Tempo receiver, or Linkerd and Tempo are communicating in the wrong format.

[cajual]

Update 1: Per Dan Williams on Slack:

Yeah, this is what we had. I found this post where the user tomy8s suggested to not export to tempo directly

I reset my values.yaml for Linkerd-Jaeger to include the collector and then point the collector back to my Tempo Distributor:

linkerd-jaeger:
  collector:
    enabled: true
    tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
    config: |
      receivers:
        otlp:
          protocols:
            grpc:
            http:
        opencensus:
        zipkin:
        jaeger:
          protocols:
            grpc:
            thrift_http:
            thrift_compact:
            thrift_binary:
      processors:
        batch:
      extensions:
        health_check:
      exporters:
        jaeger:
          endpoint: tempo-distributed-distributor.monitoring.svc:14250
          tls:
            insecure: true
      service:
        extensions: [health_check]
        pipelines:
          traces:
            receivers: [otlp,opencensus,zipkin,jaeger]
            processors: [batch]
            exporters: [jaeger]
  jaeger:
    enabled: false

However, I had attempted to push directly to Tempo since I wasn't receiving root span context from Traefik in my Tempo query, so instead I decided to push Traefik into the collector as well:

traefik:
  tracing:
    jaeger:
      propagation: b3
      traceContextHeaderName: x-b3-traceid
      disableAttemptReconnecting: true
      samplingType: probabilistic
      samplingParam: 1
      gen128Bit: true
      collector:
         endpoint: http://collector.linkerd-jaeger:14268/api/traces?format=jaeger.thrift

All that was left was to make sure my services (Python) used the otel SDK and included B3 propagation:

...
set_global_textmap(B3MultiFormat())
add_jaeger_tracer(app, settings.jaeger_thrift_url, settings.service_name)
add_prometheus_middleware(app, settings.service_name)
...

After injecting the deployments, injecting the ingress, and adding the middleware, Traefik correctly reports the root span context and Tempo is now providing the full trace stack: