Skip to content

fix(otel): restore thanos discovery and add traffic generator#279

Merged
victoriacheng15 merged 1 commit intomainfrom
fix/thanos-discovery-and-telemetry
Mar 11, 2026
Merged

fix(otel): restore thanos discovery and add traffic generator#279
victoriacheng15 merged 1 commit intomainfrom
fix/thanos-discovery-and-telemetry

Conversation

@victoriacheng15
Copy link
Owner

Summary

Fixed Thanos discovery pipeline and established end-to-end telemetry verification. This update resolves gRPC connectivity between Thanos Query and the Prometheus sidecar and introduces a synthetic traffic generator for pipeline validation.

List of Changes

  • Fix (tofu): Renamed extraArgs to extraFlags in Thanos Helm release to correctly pass the --endpoint argument to the query container, resolving the empty metrics issue.
  • Feat (tofu): Added a managed kubernetes_service_v1 for prometheus-thanos-grpc to ensure reliable headless discovery of the Prometheus sidecar, bypassing Helm chart limitations.
  • Fix (k3s): Disabled redundant dnsDiscovery in Thanos Query values to eliminate SRV lookup errors and prioritize the direct gRPC endpoint connection.
  • Feat (scripts): Created traffic_generator.sh to produce burst and continuous synthetic traffic, enabling empirical verification of the OTLP-to-Thanos telemetry pipeline.

Verification

  • tofu apply successfully provisioned the prometheus-thanos-grpc service.
  • thanos-query logs confirmed successful sidecar discovery: "adding new sidecar...".
  • Executed ./scripts/traffic_generator.sh --burst and verified 20/20 metrics appeared in Thanos via proxy_synthetic_request_total.

@victoriacheng15 victoriacheng15 merged commit edfeac0 into main Mar 11, 2026
1 check passed
@victoriacheng15 victoriacheng15 deleted the fix/thanos-discovery-and-telemetry branch March 11, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant