Skip to content

Commit 88e1dda

Browse files
giortzisggiortzisg
committed
feat(dev-docs): add index and subpages for Telemetry Buffer
1 parent ceb2cef commit 88e1dda

File tree

5 files changed

+156
-0
lines changed

5 files changed

+156
-0
lines changed

‎develop-docs/sdk/telemetry/telemetry-buffer/backend-telemetry-buffer.mdx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: Backend Telemetry Buffer
3+
description: Detailed backend telemetry buffer design.
4+
sidebar_order: 1
5+
---
6+
7+
To be defined — full spec lives here.

‎develop-docs/sdk/telemetry/telemetry-buffer/browser-telemetry-buffer.mdx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: Browser Telemetry Buffer
3+
description: Detailed browser telemetry buffer design.
4+
sidebar_order: 2
5+
---
6+
7+
To be defined — full spec lives here.

‎develop-docs/sdk/telemetry/telemetry-buffer/gdx-telemetry-buffer.mdx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: GDX Telemetry Buffer
3+
description: Detailed GDX telemetry buffer design.
4+
sidebar_order: 3
5+
---
6+
7+
To be defined — full spec lives here.

‎develop-docs/sdk/telemetry/telemetry-buffer/index.mdx‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
---
2+
title: Telemetry Buffer
3+
redirect_from:
4+
- /sdk/telemetry/spans/batch-processor/
5+
sidebar_order: 5
6+
---
7+
8+
<Alert level="warning">
9+
🚧 This document is work in progress.
10+
</Alert>
11+
12+
<Alert>
13+
This document uses key words such as "MUST", "SHOULD", and "MAY" as defined in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) to indicate requirement levels.
14+
</Alert>
15+
16+
The telemetry buffer sits between the client and the transport, temporarily buffering high-volume telemetry data such as spans and logs. The client SHOULD continue to pass low-volume telemetry data, such as events, directly to the transport. The telemetry buffer aims to efficiently batch data to reduce the number of outgoing HTTP requests and Sentry envelopes. Without buffering, each span or log would trigger its own request, quickly overwhelming our backends.
17+
18+
```mermaid
19+
flowchart LR
20+
Client[Client] -->|high-volume telemetry data| Buffer[TelemetryBuffer]
21+
Buffer -->|buffered telemetry data| Transport[Transport]
22+
Client -->|low-volume telemetry data| Transport
23+
```
24+
25+
Because telemetry workloads and platform constraints vary widely, buffer requirements differ across environments. For example, backend SDKs need high throughput and backpressure management to handle large data volumes. Mobile SDKs have lower throughput and don't need to worry much about backpressure, but they do need to minimize data loss in the event of abnormal process termination. Browser and GDX SDKs also have different requirements.
26+
27+
Therefore, we recommend implementing different types of telemetry buffers tailored to the platform's needs. As of Nov 5th, 2025, this page is under development, and we're currently refining the requirements for different platforms:
28+
29+
* [Backend Telemetry Buffer](./backend-telemetry-buffer/): Detailed backend design
30+
* [Browser Telemetry Buffer](./browser-telemetry-buffer/): To be defined
31+
* [GDX Telemetry Buffer](./gdx-telemetry-buffer/): To be defined
32+
* [Mobile Telemetry Buffer](./mobile-telemetry-buffer/): To be defined
33+
34+
# Common Requirements
35+
36+
To be defined. Things like common API, client reports, etc.
37+
38+
# BatchProcessor V0
39+
40+
## Overview
41+
42+
The BatchProcessor batches spans and logs into one envelope to reduce the number of HTTP requests. When an SDK implements span streaming or logs, it MUST use a BatchProcessor, which is similar to [OpenTelemetry's Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md). The BatchProcessor holds logs and finished spans in memory and batches them together into envelopes. It uses a combination of time and size-based batching. When writing this, the BatchProcessor only handles spans and logs, but an SDK MAY use it for other telemetry data in the future.
43+
44+
## Specification
45+
46+
Whenever the SDK finishes a span or captures a log, it MUST put it into the BatchProcessor. The SDK MUST NOT put unfinished spans into the BatchProcessor.
47+
48+
The BatchProcessor MUST start a timeout of 5 seconds when the SDK adds the first span or log. When the timeout exceeds, the BatchProcessor MUST send all spans or logs, no matter how many items it contains. The SDK MAY choose a different value for the timeout, but it MUST NOT exceed 30 seconds, as this can lead to problems with the span buffer on the backend, which uses a time interval of 60 seconds for determining segments for spans. The BatchProcessor SHOULD only start a new timeout, when it has spans or logs to send, to avoid running the timeout unnecessarily.
49+
50+
The BatchProcessor MUST send all items after the SDK when containing spans or logs exceeding 1MiB in size. The SDK MAY choose a different value for the max batch size keeping the [envelope max sizes](/sdk/data-model/envelopes/#size-limits) in mind. The SDK MUST calculate the size of a span or a log to manage the BatchProcessor's memory footprint. The SDK MUST serialize the span or log and calculate the size based on the serialized JSON bytes. As serialization is expensive, the BatchProcessor SHOULD keep track of the serialized spans and logs and pass these to the envelope to avoid serializing multiple times.
51+
52+
When the BatchProcessor sends all spans or logs, it MUST reset its timeout and remove all spans and logs. The SDK MUST apply filtering and sampling before adding spans or logs to the BatchProcessor. The SDK MUST apply rate limits to spans and logs after they leave the BatchProcessor to send as much data as possible by dropping data as late as possible.
53+
54+
The BatchProcessor MUST send all spans and logs in memory to avoid data loss in the following scenarios:
55+
56+
1. When the user calls `SentrySDK.flush()`, the BatchProcessor MUST send all data in memory.
57+
2. When the user calls `SentrySDK.close()`, the BatchProcessor MUST send all data in memory.
58+
3. When the application shuts down gracefully, the BatchProcessor SHOULD send all data in memory. This is mostly relevant for mobile SDKs already subscribed to these hooks, such as [applicationWillTerminate](https://developer.apple.com/documentation/uikit/uiapplicationdelegate/applicationwillterminate(_:)) on iOS.
59+
4. When the application moves to the background, the BatchProcessor SHOULD send all data in memory and stop the timer. This is mostly relevant for mobile SDKs.
60+
5. We're working on concept for crashes, and will update the specification when we have more details.
61+
62+
The detailed specification is written in the [Gherkin syntax](https://cucumber.io/docs/gherkin/reference/). The specification uses spans as an example, but the same applies to logs or any other future telemetry data.
63+
64+
65+
```Gherkin
66+
Scenario: No spans in BatchProcessor 1 span added
67+
Given no spans in the BatchProcessor
68+
When the SDK finishes 1 span
69+
Then the SDK puts this span to the BatchProcessor
70+
And starts a timeout of 5 seconds
71+
And doesn't send the span to Sentry
72+
73+
Scenario: Span added before timeout exceeds
74+
Given span A in the BatchProcessor
75+
Given 4.9 seconds pass
76+
When the SDK finishes span B
77+
Then the SDK adds span B to the BatchProcessor
78+
And doesn't reset the timeout
79+
And doesn't send the spans A and B in the BatchProcessor to Sentry
80+
81+
Scenario: Timeout exceeds and no spans or logs to send
82+
Given no spans in the BatchProcessor
83+
When the timeout exceeds
84+
Then the BatchProcessor does nothing
85+
And doesn't start a new timeout
86+
87+
Scenario: Spans with size of 1 MiB - 1 byte added, timeout exceeds
88+
Given spans with size of 1 MiB - 1 byte in the BatchProcessor
89+
When the timeout exceeds
90+
Then the SDK adds all the spans to one envelope
91+
And sends them to Sentry
92+
And resets the timeout
93+
And clears the BatchProcessor
94+
95+
Scenario: Spans with size of 1 MiB - 1 byte added within 4.9 seconds
96+
Given spans with size of 1 MiB - 1 byte in the BatchProcessor
97+
When the SDK finishes another span and puts it into the BatchProcessor
98+
Then the BatchProcessor puts all spans into one envelope
99+
And sends the envelope to Sentry
100+
And resets the timeout
101+
And clears the BatchProcessor
102+
103+
Scenario: Unfinished spans
104+
Given no span is in the BatchProcessor
105+
When the SDK starts a span but doesn't finish it
106+
Then the BatchProcessor is empty
107+
108+
Scenario: Span filtered out
109+
Given no span is in the BatchProcessor
110+
When the finishes a span
111+
And the span is filtered out
112+
Then the BatchProcessor is empty
113+
114+
Scenario: Span not sampled
115+
Given no span is in the BatchProcessor
116+
When the finishes a span
117+
And the span is not sampled
118+
Then the BatchProcessor is empty
119+
120+
Scenario: 1 span added application crashes
121+
Given 1 span in the SpansAggregator
122+
When the SDK detects a crash
123+
Then the SDK does nothing with the items in the BatchProcessor
124+
And loses the spans in the BatchProcessor
125+
126+
```
127+
128+
<PageGrid />

‎develop-docs/sdk/telemetry/telemetry-buffer/mobile-telemetry-buffer.mdx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: Mobile Telemetry Buffer
3+
description: Detailed mobile telemetry buffer design.
4+
sidebar_order: 4
5+
---
6+
7+
To be defined — full spec lives here.

0 commit comments

Comments
 (0)