fix(otel): restore thanos discovery and add traffic generator#279
Merged
victoriacheng15 merged 1 commit intomainfrom Mar 11, 2026
Merged
fix(otel): restore thanos discovery and add traffic generator#279victoriacheng15 merged 1 commit intomainfrom
victoriacheng15 merged 1 commit intomainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixed Thanos discovery pipeline and established end-to-end telemetry verification. This update resolves gRPC connectivity between Thanos Query and the Prometheus sidecar and introduces a synthetic traffic generator for pipeline validation.
List of Changes
extraArgstoextraFlagsin Thanos Helm release to correctly pass the--endpointargument to the query container, resolving the empty metrics issue.kubernetes_service_v1forprometheus-thanos-grpcto ensure reliable headless discovery of the Prometheus sidecar, bypassing Helm chart limitations.dnsDiscoveryin Thanos Query values to eliminate SRV lookup errors and prioritize the direct gRPC endpoint connection.traffic_generator.shto produce burst and continuous synthetic traffic, enabling empirical verification of the OTLP-to-Thanos telemetry pipeline.Verification
tofu applysuccessfully provisioned theprometheus-thanos-grpcservice.thanos-querylogs confirmed successful sidecar discovery: "adding new sidecar..."../scripts/traffic_generator.sh --burstand verified 20/20 metrics appeared in Thanos viaproxy_synthetic_request_total.