-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Only Spans Docs #11731
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Only Spans Docs #11731
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
dc919c0
Add skeleton
cleptric 83a4125
Add current state of sub pages
cleptric 8bb3a08
Update Span API
cleptric 24ab5ca
Fix formatting
cleptric 091eacd
Add trace propagation
cleptric 88d3ac2
Clean up
cleptric 3ed61ed
Clean up headers
cleptric 072bc24
Add sampling
cleptric f3166a5
Formatting
cleptric 2917836
Add notes
cleptric 7a8e625
Update develop-docs/sdk/telemetry/spans/span-sampling.mdx
cleptric 426c807
Update develop-docs/sdk/telemetry/spans/span-sampling.mdx
cleptric b66bae7
WIP scopes
cleptric File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| title: Spans | ||
| sidebar_order: 8 | ||
| --- | ||
|
|
||
| <PageGrid /> |
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| --- | ||
| title: Span API | ||
| --- | ||
|
|
||
| # Span API | ||
|
|
||
| Spans are measuring the duration of certain operations in an application. | ||
| The topmost member of a span tree is called the root span. This span has no parent span and groups together its children with a representative name for the entire operation, such as `GET /` in case of a request to a backend application. | ||
|
|
||
| ### Creating a root span | ||
|
|
||
| The SDK must expose a method for creating a root span. The user must be able to set certain properties on this root span, such as its name, the type of operation (`op`) and others. | ||
|
|
||
| ```js | ||
| span = sentry.tracing.startSpan() | ||
| ->setName('GET /') | ||
| ->setOp('http.server') | ||
|
|
||
| span.end() | ||
| ``` | ||
|
|
||
| ### Creating nested spans | ||
|
|
||
| To create nested spans, the SDK must expose an explicit way for a user to perform this task. | ||
|
|
||
| Additionally, the SDK may expose alternative APIs to create nested spans, such as allowing a user to wrap an operation into a callback or apply a decorator to certain blocks. These alternative APIs must never create a root span and no-op if no parent span is present. | ||
|
|
||
| ```js | ||
| childSpan = span.startChild() | ||
| ->setName('authentication middleware') | ||
| ->setOp('middleware.handle') | ||
|
|
||
| childSpan.end() | ||
| ``` | ||
|
|
||
| ### Setting the span status | ||
|
|
||
| A span has two statuses, `ok` and `error`. By default, the status of a span is set to `ok`. | ||
| The SDK must allow a user to modify the status of a span. | ||
|
|
||
| ```js | ||
| span.setStatus('error') | ||
| ``` | ||
|
|
||
| ### Setting span attributes | ||
|
|
||
| The SDK must expose a method to allow a user to set data attributes onto a span. | ||
| These attributes should use pre-defined keys whenever possible. | ||
|
|
||
| ```js | ||
| span.setAttribute(SpanAttributes.HTTP_METHOD, 'GET') | ||
| span.setAttribute(SpanAttributes.HTTP_RESPONSE_STATUS_CODE, 200) | ||
| ``` | ||
|
|
||
| ### Receiving the trace parent | ||
|
|
||
| The SDK must expose a method to receive the baggage string. | ||
|
|
||
| ```js | ||
| traceparent = span.getTraceparent() | ||
| ``` | ||
|
|
||
| ### Receiving the baggage | ||
|
|
||
| The SDK must expose a method to receive the baggage string. | ||
|
|
||
| ```js | ||
| baggage = span.getBaggage() | ||
| ``` | ||
|
|
||
| ### Additional, optional span APIs | ||
|
|
||
| `span.setStartTimestamp()` - overwrite the span's start time | ||
| `span.setEndTimestamp()` - overwrites the span's end time |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| --- | ||
| title: Span Propertires | ||
| --- | ||
|
|
||
| # Span Propertires | ||
|
|
||
| Instead of spans containing tags, context, and data, we'll unify all these properties into a new “attributes” property. | ||
| Similar to OTel's semantic conventions, we'll add special meaning to certain attribute keys, such as `sentry.release`, `sentry.op`, etc. | ||
|
|
||
| ```js | ||
| span.setAttribute('http.request.method', 'GET') | ||
| span.setAttribute('user.email', '[email protected]') | ||
| ``` | ||
|
|
||
| We'll map these attributes to their respective existing property in Relay to ease the work required for the product during the transition period. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| --- | ||
| title: Span Protocol | ||
| --- | ||
|
|
||
| # Span protocol | ||
|
|
||
| We'll introduce a new “span” envelope item, which the SDK uses to emit a segment span and its children. And in the future, a batch of spans. | ||
| The payload of each envelope item follows the [OpenTelemetry Protocol](https://opentelemetry.io/docs/specs/otel/protocol/), which introduced typed attributes and will ease the conversion in our POtel SDKs. | ||
|
|
||
| ```json | ||
| { | ||
| "event_id":"9ec79c33ec9942ab8353589fcb2e04dc" | ||
| } | ||
| { | ||
| "type": "span" | ||
| } | ||
| { | ||
| "traceId": "32d3c7cb501fbddbe3ce1016a72d50b5", | ||
| "spanId": "e91d37480970523b", | ||
| "name": "GET /", | ||
| "startTime": "1544712660", | ||
| "endTime": "1544712680", | ||
| "attributes": [ | ||
| { | ||
| "key": "sentry.op", | ||
| "value": { | ||
| "stringValue": "http.sever", | ||
| } | ||
| }, | ||
| { | ||
| "key": "http.response.status_code", | ||
| "value": { | ||
| "intValue": "200", | ||
| } | ||
| } | ||
| } | ||
| } | ||
| { | ||
| "type": "span" | ||
| } | ||
| { | ||
| "traceId": "32d3c7cb501fbddbe3ce1016a72d50b5", | ||
| "spanId": "6b22b3af586e777a", | ||
| "parentSpanId": "e91d37480970523b", | ||
| "name": "UserMiddleware", | ||
| "startTimeUnix": "1544712665", | ||
| "endTimeUnix": "1544712675", | ||
| "attributes": [ | ||
| { | ||
| "key": "sentry.op", | ||
| "value": { | ||
| "stringValue": "middleware.handle", | ||
| } | ||
| }, | ||
| { | ||
| "key": "user.email", | ||
| "value": { | ||
| "stringValue": "[email protected]", | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,150 @@ | ||
| --- | ||
| title: Span Sampling | ||
| --- | ||
|
|
||
| # Span Sampling | ||
|
|
||
| With the metrics product shifting to a sampling based solution, extrapolation is of the utter most importance to being able to display reliable metrics to our users. We want to account for client sampling in addition to server sampling. | ||
| This requires the SDKs to always report the correct sampling rates in each tracing related envelope send to Sentry. | ||
| Directional, the goal is to create complete traces by default and wherever possible. We will not optimise for spent-control. | ||
|
|
||
| We historically exposed many ways to our users to remove certain transactions or spans from being emitted to Sentry. This resulted in convoluted SDK APIs, weird edge cases in the product and an overall bad user experience. More importantly, these sampling controls will contribute to vastly wrong metrics being extracted from span attributes, hence we need to rework those: | ||
|
|
||
| - `beforeSendTransaction` and `beforeSendSpan` will be replaced with `beforeSendSpans`, which encourages users to *mutate* spans, but they cannot be dropped through this callback. | ||
| - All SDK integrations that create spans, need to be able to be turned off via a config flag for the purpose of noise reduction or via a new `ignoreSpans` options that accepts a glob pattern. | ||
| - Sampling happens exclusively via `tracesSampleRate` or `tracesSampler`. We need to make sure to always prefer the parent sampling decision, either via explicit docs or a new argument for the `tracesSampler` or SDK option. | ||
| - Trace propagation is aware of applications or at least organizations and prevents “leaking” traces across this boundary. | ||
|
|
||
| ## `beforeSendSpans` | ||
|
|
||
| The primary use-case for this hook will be data scrubbing or mutating certain properties of spans. | ||
|
|
||
| We are likely only allow to mutate the span’s name, timestamps, status and most attributes. Trace ID, span ID, parent span ID are immutable, as well as certain span attributes, such as segment ID. | ||
|
|
||
| It is yet to be defined which arguments will be passed into the callback or how the hook behaves with transaction envelopes. | ||
|
|
||
| ## Span configuration | ||
|
|
||
| To reduce noise, users might want to disable certain integrations creating spans. This should ideally be exposed as a global config or at an integration level. Additionally, a new `ignoreSpans` option will allow users to not emit certain spans based on their name & attributes. | ||
|
|
||
| ```jsx | ||
| Sentry.init({ | ||
| dsn: 'foo@bar', | ||
| ignoreSpans: [ | ||
| 'GET /about', | ||
| 'events.signal *', | ||
| ], | ||
| ignoreSpans: (name, attributes) { | ||
| if ( | ||
| name === 'server.request' && | ||
| attributes['server.address'] === 'https://sentry.io' | ||
| ) { | ||
| return true | ||
| } | ||
| }, | ||
| integrations: [ | ||
| fsIntegration: { | ||
| ignoreSpans: [ | ||
| 'fs.read', | ||
| ], | ||
| readSpans: true, | ||
| writeSpans: false, | ||
| } | ||
| ] | ||
| }) | ||
| ``` | ||
|
|
||
| ## Parent Sampling Decision | ||
|
|
||
| In today's SDKs, a parent sampling decision received via a `sentry-trace` header or similar can be overruled by setting a `tracesSampler`. | ||
| As we need to optimize for trace completeness, we need to explicitly call out the impact of the sampler or change the behaviour to always use the parent’s decision unless explicitly opted-out. | ||
|
|
||
| ```jsx | ||
| // Explict docs | ||
| Sentry.init({ | ||
| tracesSampler: ({ name, attributes, parentSampled }) => { | ||
| // Continue trace decision, if there is any parentSampled information | ||
| // This is crucial for complete traces | ||
| if (typeof parentSampled === "boolean") { | ||
| return parentSampled; | ||
| } | ||
|
|
||
| // Else, use default sample rate (replacing tracesSampleRate) | ||
| return 0.5; | ||
| }, | ||
| }) | ||
|
|
||
| // Not chosen - New top level option | ||
| Sentry.init({ | ||
| ignoreParentSamplingDecision: true, | ||
| tracesSampler: ({ name, attributes, parentSampled }) => { | ||
| // Do not sample health checks ever | ||
| if (name.includes("healthcheck")) { | ||
| // Drop this transaction, by setting its sample rate to 0% | ||
| return 0.0; | ||
| } | ||
|
|
||
| // Else, use default sample rate (replacing tracesSampleRate) | ||
| return 0.2; | ||
| }, | ||
| }) | ||
| ``` | ||
|
|
||
| ## Parent Sampling Origins | ||
|
|
||
| In order to filter out unrelated 3rd party services that are making requests to a Sentry instrumented app containing a `sentry-trace` header, we’ll implement RFC https://github.com/getsentry/rfcs/pull/137. This feature might be enabled by default if the: | ||
|
|
||
| - SDK knows its org | ||
| - The incoming baggage header contains a `sentry-org` entry | ||
|
|
||
| ## Sampling Seed in DSC | ||
|
|
||
| To increase the chance of capturing complete traces when users return a new sample rate `tracesSampler` in backend services, we propagate the random value used by the SDK for computing the sampling decision instead of creating a new random value in every service. Therefore, across a trace every SDK uses the *same* random value. | ||
|
|
||
| ### Behavior | ||
|
|
||
| A user can also override the parent sample rate in traces sampler. For example, a backend service has a `tracesSampler` that overrides frontend traces. This leads to three scenarios: | ||
|
|
||
| - The new (backend) sample rate is lower than the parent’s (frontend): All traces captured in the backend are complete. There are additional partial traces for the frontend. | ||
| - The new (backend) sample rate is higher than the parent’s (fronted): All traces propagated from the frontend are complete. There are additional partial traces for the backend. | ||
| - Both sample rates are equal: All traces are complete, the sampling decision is fully inherited. | ||
|
|
||
| The behavior of the static `tracesSampleRate` without the use of `tracesSampler` does not change. We continue to fully inherit sampling decisions for propagated traces and create a new one for started traces. In the future, we might change the default behavior of `tracesSampleRate`, too. | ||
|
|
||
| ### SDK Spec | ||
|
|
||
| - sentry baggage gains a new field `sentry-sample_rand` | ||
| - when a new trace is started, `sentry-sample_rand` is filled with a truly random number. this also applies when the trace’s sample rate is 1.0 | ||
| - for inbound traces without a `sentry-sample_rand` (from old SDKs), the SDK inserts a new truly random number on-the-fly. | ||
| - sampling decisions in the SDK that currently compare `sentry-sample_rand` from the trace instead of `math.random()` with the sample rate. | ||
| - when traces sampler is invoked, this also applies to the return value of traces sampler. ie. `trace["sentry-sample_rand"] < tracesSampler(context)` | ||
| - otherwise, when the SDK is the head of a trace, this applies to sample decisions based on `tracesSampleRate` , i.e. ``trace["sentry-sample_rand"] < config.tracesSampleRate` | ||
| - There is no more `math.random()` directly involved in any sampling decision. | ||
| - in traces sampler, the most correct way to inherit parent sampling decisions is now to return the parent’s sample **rate** instead of the **decision** as float (`1.0`). This way, we can still extrapolate counts correctly. | ||
|
|
||
| ```jsx | ||
| tracesSampler: ({ name, parentSampleRate }) => { | ||
| // Inherit the trace parent's sample rate if there is one. Sampling is deterministic | ||
| // for one trace, i.e. if the parent was sampled, we will be sampled too at the same | ||
| // rate. | ||
| if (typeof parentSampleRate === "number") { | ||
| return parentSampleRate; | ||
| } | ||
|
|
||
| // Else, use default sample rate (replacing tracesSampleRate). | ||
| return 0.5; | ||
| }, | ||
|
|
||
| ``` | ||
|
|
||
| - if the `sentry-sample_rate` (`parentSampleRate`) is not available for any reason for an inbound trace, but the trace has the sampled flag set to `true`, the SDK injects `parentSampleRate: 1.0` into the callback. | ||
|
|
||
| ## Baggage Freeze | ||
|
|
||
| --- | ||
|
|
||
| We accept partial traces under the assumption that the transaction name is mostly changed early in the request cycle. | ||
|
|
||
| # External Resources | ||
|
|
||
| https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling-experimental/ |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.