|
1 | 1 | --- |
2 | | -title: Span Sampling |
| 2 | +title: Span Sampling & Filtering |
3 | 3 | --- |
4 | 4 |
|
5 | | -# Span Sampling |
| 5 | +Any APIs exposed to the user to sample or filter spans must adhere to the following design principles: |
6 | 6 |
|
7 | | -With the metrics product shifting to a sampling based solution, extrapolation is of the utter most importance to being able to display reliable metrics to our users. We want to account for client sampling in addition to server sampling. |
8 | | -This requires the SDKs to always report the correct sampling rates in each tracing related envelope send to Sentry. |
9 | | -Directional, the goal is to create complete traces by default and wherever possible. We will not optimise for spent-control. |
| 7 | +- The APIs are optimized for trace completness |
| 8 | +- The APIs are optimized for conclusive sampling decisions |
10 | 9 |
|
11 | | -We historically exposed many ways to our users to remove certain transactions or spans from being emitted to Sentry. This resulted in convoluted SDK APIs, weird edge cases in the product and an overall bad user experience. More importantly, these sampling controls will contribute to vastly wrong metrics being extracted from span attributes, hence we need to rework those: |
| 10 | +## Sample root spans with `tracesSampleRate` & `tracesSampler` |
12 | 11 |
|
13 | | -- `beforeSendTransaction` and `beforeSendSpan` will be replaced with `beforeSendSpans`, which encourages users to *mutate* spans, but they cannot be dropped through this callback. |
14 | | -- All SDK integrations that create spans, need to be able to be turned off via a config flag for the purpose of noise reduction or via a new `ignoreSpans` options that accepts a glob pattern. |
15 | | -- Sampling happens exclusively via `tracesSampleRate` or `tracesSampler`. We need to make sure to always prefer the parent sampling decision, either via explicit docs or a new argument for the `tracesSampler` or SDK option. |
16 | | -- Trace propagation is aware of applications or at least organizations and prevents “leaking” traces across this boundary. |
17 | 12 |
|
18 | | -## `beforeSendSpans` |
19 | 13 |
|
20 | | -The primary use-case for this hook will be data scrubbing or mutating certain properties of spans. |
| 14 | +## Filter spans with `ignoreSpans` & integration config |
21 | 15 |
|
22 | | -We are likely only allow to mutate the span’s name, timestamps, status and most attributes. Trace ID, span ID, parent span ID are immutable, as well as certain span attributes, such as segment ID. |
23 | 16 |
|
24 | | -It is yet to be defined which arguments will be passed into the callback or how the hook behaves with transaction envelopes. |
25 | 17 |
|
26 | | -## Span configuration |
| 18 | +## Sanitize span attributes with `beforeSendSpans` |
27 | 19 |
|
28 | | -To reduce noise, users might want to disable certain integrations creating spans. This should ideally be exposed as a global config or at an integration level. Additionally, a new `ignoreSpans` option will allow users to not emit certain spans based on their name & attributes. |
29 | 20 |
|
30 | | -```jsx |
31 | | -Sentry.init({ |
32 | | - dsn: 'foo@bar', |
33 | | - ignoreSpans: [ |
34 | | - 'GET /about', |
35 | | - 'events.signal *', |
36 | | - ], |
37 | | - ignoreSpans: (name, attributes) { |
38 | | - if ( |
39 | | - name === 'server.request' && |
40 | | - attributes['server.address'] === 'https://sentry.io' |
41 | | - ) { |
42 | | - return true |
43 | | - } |
44 | | - }, |
45 | | - integrations: [ |
46 | | - fsIntegration: { |
47 | | - ignoreSpans: [ |
48 | | - 'fs.read', |
49 | | - ], |
50 | | - readSpans: true, |
51 | | - writeSpans: false, |
52 | | - } |
53 | | - ] |
54 | | -}) |
55 | | -``` |
56 | | - |
57 | | -## Parent Sampling Decision |
58 | | - |
59 | | -In today's SDKs, a parent sampling decision received via a `sentry-trace` header or similar can be overruled by setting a `tracesSampler`. |
60 | | -As we need to optimize for trace completeness, we need to explicitly call out the impact of the sampler or change the behaviour to always use the parent’s decision unless explicitly opted-out. |
61 | | - |
62 | | -```jsx |
63 | | -// Explict docs |
64 | | -Sentry.init({ |
65 | | - tracesSampler: ({ name, attributes, parentSampled }) => { |
66 | | - // Continue trace decision, if there is any parentSampled information |
67 | | - // This is crucial for complete traces |
68 | | - if (typeof parentSampled === "boolean") { |
69 | | - return parentSampled; |
70 | | - } |
71 | | - |
72 | | - // Else, use default sample rate (replacing tracesSampleRate) |
73 | | - return 0.5; |
74 | | - }, |
75 | | -}) |
76 | | - |
77 | | -// Not chosen - New top level option |
78 | | -Sentry.init({ |
79 | | - ignoreParentSamplingDecision: true, |
80 | | - tracesSampler: ({ name, attributes, parentSampled }) => { |
81 | | - // Do not sample health checks ever |
82 | | - if (name.includes("healthcheck")) { |
83 | | - // Drop this transaction, by setting its sample rate to 0% |
84 | | - return 0.0; |
85 | | - } |
86 | | - |
87 | | - // Else, use default sample rate (replacing tracesSampleRate) |
88 | | - return 0.2; |
89 | | - }, |
90 | | -}) |
91 | | -``` |
92 | | - |
93 | | -## Parent Sampling Origins |
94 | | - |
95 | | -In order to filter out unrelated 3rd party services that are making requests to a Sentry instrumented app containing a `sentry-trace` header, we’ll implement RFC https://github.com/getsentry/rfcs/pull/137. This feature might be enabled by default if the: |
96 | | - |
97 | | -- SDK knows its org |
98 | | -- The incoming baggage header contains a `sentry-org` entry |
99 | | - |
100 | | -## Sampling Seed in DSC |
101 | | - |
102 | | -To increase the chance of capturing complete traces when users return a new sample rate `tracesSampler` in backend services, we propagate the random value used by the SDK for computing the sampling decision instead of creating a new random value in every service. Therefore, across a trace every SDK uses the *same* random value. |
103 | | - |
104 | | -### Behavior |
105 | | - |
106 | | -A user can also override the parent sample rate in traces sampler. For example, a backend service has a `tracesSampler` that overrides frontend traces. This leads to three scenarios: |
107 | | - |
108 | | -- The new (backend) sample rate is lower than the parent’s (frontend): All traces captured in the backend are complete. There are additional partial traces for the frontend. |
109 | | -- The new (backend) sample rate is higher than the parent’s (fronted): All traces propagated from the frontend are complete. There are additional partial traces for the backend. |
110 | | -- Both sample rates are equal: All traces are complete, the sampling decision is fully inherited. |
111 | | - |
112 | | -The behavior of the static `tracesSampleRate` without the use of `tracesSampler` does not change. We continue to fully inherit sampling decisions for propagated traces and create a new one for started traces. In the future, we might change the default behavior of `tracesSampleRate`, too. |
113 | | - |
114 | | -### SDK Spec |
115 | | - |
116 | | -- sentry baggage gains a new field `sentry-sample_rand` |
117 | | - - when a new trace is started, `sentry-sample_rand` is filled with a truly random number. this also applies when the trace’s sample rate is 1.0 |
118 | | - - for inbound traces without a `sentry-sample_rand` (from old SDKs), the SDK inserts a new truly random number on-the-fly. |
119 | | -- sampling decisions in the SDK that currently compare `sentry-sample_rand` from the trace instead of `math.random()` with the sample rate. |
120 | | - - when traces sampler is invoked, this also applies to the return value of traces sampler. ie. `trace["sentry-sample_rand"] < tracesSampler(context)` |
121 | | - - otherwise, when the SDK is the head of a trace, this applies to sample decisions based on `tracesSampleRate` , i.e. ``trace["sentry-sample_rand"] < config.tracesSampleRate` |
122 | | - - There is no more `math.random()` directly involved in any sampling decision. |
123 | | -- in traces sampler, the most correct way to inherit parent sampling decisions is now to return the parent’s sample **rate** instead of the **decision** as float (`1.0`). This way, we can still extrapolate counts correctly. |
124 | | - |
125 | | - ```jsx |
126 | | - tracesSampler: ({ name, parentSampleRate }) => { |
127 | | - // Inherit the trace parent's sample rate if there is one. Sampling is deterministic |
128 | | - // for one trace, i.e. if the parent was sampled, we will be sampled too at the same |
129 | | - // rate. |
130 | | - if (typeof parentSampleRate === "number") { |
131 | | - return parentSampleRate; |
132 | | - } |
133 | | - |
134 | | - // Else, use default sample rate (replacing tracesSampleRate). |
135 | | - return 0.5; |
136 | | - }, |
137 | | - |
138 | | - ``` |
139 | | - |
140 | | - - if the `sentry-sample_rate` (`parentSampleRate`) is not available for any reason for an inbound trace, but the trace has the sampled flag set to `true`, the SDK injects `parentSampleRate: 1.0` into the callback. |
141 | | - |
142 | | -## Baggage Freeze |
143 | | - |
144 | | ---- |
145 | | - |
146 | | -We accept partial traces under the assumption that the transaction name is mostly changed early in the request cycle. |
147 | | - |
148 | | -# External Resources |
149 | | - |
150 | | -https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling-experimental/ |
0 commit comments