query: fix exemplar proxy stripping external label matchers in multi-tier topologies#8752
Merged
GiedriusS merged 4 commits intothanos-io:mainfrom Apr 21, 2026
Conversation
Krishnachaitanyakc
force-pushed
the
implement/query-exemplar-proxy-strips-external-label-matcher
branch
2 times, most recently
from
fc019dc to
01ed452
Compare
Contributor
Author
|
Added an ignore rule in |
Krishnachaitanyakc
force-pushed
the
implement/query-exemplar-proxy-strips-external-label-matcher
branch
from
e6f9265 to
8388bdf
Compare
GiedriusS
reviewed
Apr 14, 2026
Member
GiedriusS
left a comment
GiedriusS
left a comment
There was a problem hiding this comment.
LGTM, let's see if anyone else has any comments.
Contributor
Author
|
@saswatamcode @fpetkovski can you please review this? |
… exemplar proxy In multi-tier Query topologies (Query A → Query B → Sidecars), the exemplar proxy was stripping external label matchers before forwarding to all downstream stores. This stripping is necessary for Sidecars (Prometheus cannot handle external labels), but breaks Query-to-Query forwarding because the intermediate Query node needs the matchers for its own store routing. Add a SupportsExternalLabels field to ExemplarStore that indicates whether a downstream store can handle external label matchers. When constructing exemplar stores from endpoints, set this to true for Query component types. In the proxy fanout loop, only strip external label matchers for stores that do not support them. Fixes thanos-io#8702 Signed-off-by: Kc Balusu <kcbalusu@gmail.com>
Signed-off-by: Kc Balusu <kcbalusu@meta.com>
Add ignore rules for Grafana promtail docs (removed after EOL in March 2026) and db.cs.cmu.edu (DNS/timeout issues from CI). Signed-off-by: Kc Balusu <kcbalusu@meta.com>
Update the sed range in docs/components/receive.md from '1068,1078p' to '1067,1078p' to match the current line numbers in handler.go after upstream changes shifted the writeQuorum function. Signed-off-by: Kc Balusu <kcbalusu@meta.com>
Krishnachaitanyakc
force-pushed
the
implement/query-exemplar-proxy-strips-external-label-matcher
branch
from
8388bdf to
7b851b7
Compare
Can you comment how am I related to this? I might, I just do not know how. |
Contributor
Author
my bad, I tagged you on a different PR in a different repo and it got copy pasted into this. Apologies |
Contributor
Author
|
@saswatamcode @GiedriusS can you please help me getting this reviewed? |
Member
|
Thank you 🙇 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
In multi-tier Query topologies (Query A → Query B → Sidecars), the exemplar proxy unconditionally strips matchers that match external labels before forwarding to all downstream stores. This stripping is correct for Sidecars (Prometheus cannot handle external labels), but breaks Query → Query → Sidecar setups because the intermediate Query node needs those matchers for its own store routing.
For example, when Query A receives
{cluster="A", namespace="foo"}, it stripscluster="A"before forwarding to Query B. Query B then receives{namespace="foo"}and fans out to all sidecars, returning exemplars from every cluster instead of just cluster A.This was introduced by #4123 (fix for #4116).
What this PR does
SupportsExternalLabelsfield toExemplarStorethat indicates whether a downstream store can handle external label matchers.GetExemplarsStores(), setsSupportsExternalLabels: trueforcomponent.Queryendpoints, since Query nodes can handle external labels for their own routing.Verification
TestProxyExternalLabelsPreservedForQueryStores: verifies that a downstream Query node receives the full matchers including external labels, while a non-matching Sidecar is correctly excluded.TestProxyExternalLabelsStrippedForSidecarStores: verifies backward compatibility — Sidecars still receive queries with external label matchers stripped.SupportsExternalLabelsdefaults tofalse).Fixes #8702
Changes have a CHANGELOG entry: yes