Skip to content

Commit 5cfa082

Browse files
ref: Rename telemetry buffer pages to processor (#15860)
Rename all the telemetry buffer pages to telemetry processor and clarify that the processor does more than buffering. This is the first step of migrating the [Notion doc](https://www.notion.so/sentry/Telemetry-Processor-2b58b10e4b5d805e819be97fbfc0cd53) concept of the telemetry processor to the develop docs.
1 parent c2265c9 commit 5cfa082

File tree

10 files changed

+139
-100
lines changed

10 files changed

+139
-100
lines changed

develop-docs/sdk/telemetry/telemetry-buffer/browser-telemetry-buffer.mdx

Lines changed: 0 additions & 7 deletions
This file was deleted.

develop-docs/sdk/telemetry/telemetry-buffer/gdx-telemetry-buffer.mdx

Lines changed: 0 additions & 7 deletions
This file was deleted.

develop-docs/sdk/telemetry/telemetry-buffer/index.mdx

Lines changed: 0 additions & 49 deletions
This file was deleted.

develop-docs/sdk/telemetry/telemetry-buffer/backend-telemetry-buffer.mdx renamed to develop-docs/sdk/telemetry/telemetry-processor/backend-telemetry-processor.mdx

Lines changed: 32 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,30 @@
11
---
2-
title: Backend Telemetry Buffer
3-
description: Detailed backend telemetry buffer design.
2+
title: Backend Telemetry Processor
3+
description: Detailed backend telemetry processor design.
44
sidebar_order: 1
55
---
66

7-
## Telemetry Buffer Layer: Prioritized, Bounded, Rate-Aware Envelope Delivery
7+
<Alert level="warning">
8+
🚧 This document is work in progress.
9+
</Alert>
10+
11+
## Telemetry Processor Layer: Prioritized, Bounded, Rate-Aware Envelope Delivery
812

913
### Overview
1014

11-
The buffer system sits between the SDK client and the HTTP transport layer, ensuring that critical telemetry like errors take priority over high-volume data like logs and traces. This prevents important events from getting lost when your application is under heavy load or sending large amounts of telemetry.
15+
The TelemetryProcessor sits between the SDK client and the HTTP transport layer, ensuring that critical telemetry like errors take priority over high-volume data like logs and traces. This prevents important events from getting lost when your application is under heavy load or sending large amounts of telemetry.
1216

1317
### Motivation
1418

15-
- Aggregation lives in a unified buffer layer (this way we avoid creating multiple batch processors for different telemetry types).
19+
- Aggregation lives in a unified layer (this way we avoid creating multiple batch processors for different telemetry types).
1620
- All telemetry types use capture APIs (CaptureX) routed through the Client.
1721
- Rate-limit awareness is built-in across categories.
1822
- Buffers support two modes: normal ring buffer and bucket-by-trace (for spans).
1923
- For spans, dropping an entire trace under pressure is preferable.
2024

2125
### Architecture Overview
2226

23-
Introduce a `Buffer` layer between the `Client` and the `Transport`. This `Buffer` wraps prioritization and scheduling and exposes a minimal API to the SDK:
27+
Introduce a `TelemetryProcessor` layer between the `Client` and the `Transport`. This `TelemetryProcessor` wraps prioritization and scheduling and exposes a minimal API to the SDK:
2428

2529
- Add(item).
2630
- Flush(timeout).
@@ -34,19 +38,19 @@ Introduce a `Buffer` layer between the `Client` and the `Transport`. This `Buffe
3438
3539
3640
┌────────────────────────────────────────────────────────────────────────────┐
37-
Buffer
41+
TelemetryProcessor
3842
│ Add(item) · Flush(timeout) · Close(timeout) │
3943
│ │
4044
│ ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────┐ │
41-
│ │ Error Store │ │ Check-in Store │ │ Log Store │ │
45+
│ │ Error Buffer │ │ Check-in Buffer │ │ Log Buffer │ │
4246
│ │ (CRITICAL) │ │ (HIGH) │ │ (LOW) │ │
4347
│ │ Timeout: N/A │ │ Timeout: N/A │ │ Timeout: 5s │ │
4448
│ │ BatchSize: 1 │ │ BatchSize: 1 │ │ BatchSize: 100 │ │
4549
│ └──────────────────────┘ └──────────────────────┘ └──────────────────┘ │
4650
│ │ │
4751
│ ▼ │
4852
│ ┌─────────────────────────────────────────────────────────────────────┐ │
49-
│ │ Scheduler (Weighted Round-Robin) │ │
53+
│ │ TelemetryScheduler (Weighted Round-Robin) │ │
5054
│ │ - Priority weights: CRITICAL=5, HIGH=4, MEDIUM=3, LOW=2, LOWEST=1 │ │
5155
│ │ - Processes a batch of items based on BatchSize and/or Timeout │ │
5256
│ │ - Builds envelopes from batch │ │
@@ -61,10 +65,10 @@ Introduce a `Buffer` layer between the `Client` and the `Transport`. This `Buffe
6165
└────────────────────────────────────────────────────────────────────────────┘
6266
```
6367

64-
#### How the Buffer works
68+
#### How the Processor works
6569

6670
- **Smart batching**: Logs are batched into single requests; errors, transactions, and monitors are sent immediately.
67-
- **Pre-send rate limiting**: The scheduler checks rate limits before serialization to avoid unnecessary processing. When a telemetry is rate-limited the selected batch should
71+
- **Pre-send rate limiting**: The TelemetryScheduler checks rate limits before serialization to avoid unnecessary processing. When a telemetry is rate-limited the selected batch should
6872
be dropped, to avoid filling up the buffers.
6973
- **Category isolation**: Separate ring buffers for each telemetry type prevent head-of-line blocking.
7074
- **Weighted scheduling**: High-priority telemetry gets sent more frequently via round-robin selection.
@@ -81,9 +85,9 @@ Configurable via weights.
8185

8286
### Components
8387

84-
#### Storage
88+
#### TelemetryBuffers
8589

86-
Each telemetry category maintains a store interface; a fixed-size circular array/ring buffer (not to be confused with the `Buffer` wrapper) that stores items before transmission:
90+
Each telemetry category maintains a buffer interface; a fixed-size circular array/ring buffer that stores items before transmission:
8791

8892
- **Bounded capacity**: Default to 100 items for errors, logs, and monitors; 1000 for transactions. This prevents unbounded memory growth regardless of telemetry volume and backpressure handling.
8993
- **Overflow policies** (optional):
@@ -100,10 +104,10 @@ Each telemetry category maintains a store interface; a fixed-size circular array
100104
- Offer semantics: if not full, append; when full, apply `overflowPolicy`:
101105
- `drop_oldest`: evict the oldest item, insert the new one, and invoke the dropped callback with reason `buffer_full_drop_oldest`.
102106
- `drop_newest`: reject the new item and invoke the dropped callback with reason `buffer_full_drop_newest`.
103-
- Readiness: a store is ready when `size >= batchSize` or when `timeout` has elapsed since `lastFlushTime` (and it is non-empty).
104-
- Polling: `PollIfReady()` returns up to `batchSize` items and updates `lastFlushTime`; `Drain()` empties the store.
107+
- Readiness: a buffer is ready when `size >= batchSize` or when `timeout` has elapsed since `lastFlushTime` (and it is non-empty).
108+
- Polling: `PollIfReady()` returns up to `batchSize` items and updates `lastFlushTime`; `Drain()` empties the buffer.
105109

106-
##### Bucketed-by-trace storage (spans)
110+
##### Bucketed-by-trace buffer (spans)
107111

108112
- **Purpose**: keep spans from the same trace together and flush them as a unit to avoid partial-trace delivery under pressure. This addresses a gap in standard implementations where individual span drops can create incomplete traces.
109113
- **Grouping**: a new bucket is created per trace id; a map (`traceIndex`) provides O(1) lookup.
@@ -119,11 +123,11 @@ Each telemetry category maintains a store interface; a fixed-size circular array
119123
There still remains a small subset of cases that might result in partial traces, where either an old trace bucket was dropped and a new span with the same trace arrived, or we dropped an incoming span of this trace.
120124
The preferred overflow behavior in most cases should be `drop_oldest` since it results in the fewest incomplete traces from the two scenarios.
121125

122-
Stores are mapped to [DataCategories](https://github.com/getsentry/relay/blob/master/relay-base-schema/src/data_category.rs), which determine their scheduling priority and rate limits.
126+
Buffers are mapped to [DataCategories](https://github.com/getsentry/relay/blob/master/relay-base-schema/src/data_category.rs), which determine their scheduling priority and rate limits.
123127

124-
#### Scheduler
128+
#### TelemetryScheduler
125129

126-
The scheduler runs as a background worker, coordinating the flow of telemetry from storage to the transport:
130+
The TelemetryScheduler runs as a background worker, coordinating the flow of telemetry from buffers to the transport:
127131

128132
- **Initialization**: Constructs a weighted priority cycle (e.g., `[CRITICAL×5, HIGH×4, MEDIUM×3, ...]`) based on configured weights.
129133
- **Event loop**: Wakes when explicitly signaled from the `captureX` methods on the client when new data is available (if the language does not support this, then a periodic ticker can be used).
@@ -191,7 +195,7 @@ type Storage[T any] interface {
191195
}
192196

193197

194-
// Single item store
198+
// Single item buffer
195199
func (b *RingBuffer[T]) PollIfReady() []T {
196200
b.mu.Lock()
197201
defer b.mu.Unlock()
@@ -226,7 +230,7 @@ func (b *RingBuffer[T]) PollIfReady() []T {
226230
return result
227231
}
228232

229-
// Bucketed store
233+
// Bucketed buffer
230234
func (b *BucketedBuffer[T]) PollIfReady() []T {
231235
b.mu.Lock()
232236
defer b.mu.Unlock()
@@ -257,10 +261,10 @@ func (b *BucketedBuffer[T]) PollIfReady() []T {
257261

258262
```
259263

260-
#### Scheduler Processing
264+
#### TelemetryScheduler Processing
261265

262266
```go
263-
func (s *Scheduler) run() {
267+
func (s *TelemetryScheduler) run() {
264268
for {
265269
s.mu.Lock()
266270

@@ -274,7 +278,7 @@ func (s *Scheduler) run() {
274278
}
275279
}
276280

277-
func (s *Scheduler) hasWork() bool {
281+
func (s *TelemetryScheduler) hasWork() bool {
278282
for _, buffer := range s.buffers {
279283
if buffer.IsReadyToFlush() {
280284
return true
@@ -283,15 +287,15 @@ func (s *Scheduler) hasWork() bool {
283287
return false
284288
}
285289

286-
func (s *Scheduler) processNextBatch() {
290+
func (s *TelemetryScheduler) processNextBatch() {
287291
if len(s.currentCycle) == 0 {
288292
return
289293
}
290294

291295
priority := s.currentCycle[s.cyclePos]
292296
s.cyclePos = (s.cyclePos + 1) % len(s.currentCycle)
293297

294-
var bufferToProcess Storage[protocol.EnvelopeItemConvertible]
298+
var bufferToProcess TelemetryBuffer[protocol.EnvelopeItemConvertible]
295299
var categoryToProcess ratelimit.Category
296300
for category, buffer := range s.buffers {
297301
if buffer.Priority() == priority && buffer.IsReadyToFlush() {
@@ -311,8 +315,8 @@ func (s *Scheduler) processNextBatch() {
311315
#### Flushing
312316

313317
```go
314-
func (s *Scheduler) flush() {
315-
// should process all store buffers and send to transport
318+
func (s *TelemetryScheduler) flush() {
319+
// should process all buffers and send to transport
316320
for category, buffer := range s.buffers {
317321
if !buffer.IsEmpty() {
318322
s.processItems(buffer, category, true)

develop-docs/sdk/telemetry/telemetry-buffer/batch-processor.mdx renamed to develop-docs/sdk/telemetry/telemetry-processor/batch-processor.mdx

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,12 @@
22
title: Batch Processor (deprecated)
33
redirect_from:
44
- /sdk/telemetry/spans/batch-processor/
5+
- /sdk/telemetry/telemetry-buffer/batch-processor/
56
sidebar_order: 10
67
---
78

89
<Alert level="warning">
9-
The BatchProcessor is deprecated. Please use the [Telemetry Buffer](/sdk/telemetry/telemetry-buffer/) instead.
10+
The BatchProcessor is deprecated. Please use the [Telemetry Processor](/sdk/telemetry/telemetry-processor/) instead.
1011
</Alert>
1112

1213
<Alert>
@@ -15,7 +16,7 @@ sidebar_order: 10
1516

1617
# BatchProcessor (deprecated)
1718

18-
This section covers the initial specification of the BatchProcessor, which some SDKs use as a reference when implementing logs. This exists only as a reference until we fully spec out the [telemetry buffer](/sdk/telemetry/telemetry-buffer/) across all platforms.
19+
This section covers the initial specification of the BatchProcessor, which some SDKs use as a reference when implementing logs. This exists only as a reference until we fully spec out the [telemetry processor](/sdk/telemetry/telemetry-processor/) across all platforms.
1920

2021
## Overview
2122

@@ -37,7 +38,7 @@ The BatchProcessor **MUST** forward all spans and logs in memory to the transpor
3738
2. When the user calls `SentrySDK.close()`, the BatchProcessor **MUST** forward all data in memory to the transport. SDKs **SHOULD** keep their existing closing behavior.
3839
3. When the application shuts down gracefully, the BatchProcessor **SHOULD** forward all data in memory to the transport. The transport **SHOULD** keep its existing behavior, which usually stores the data to disk as an envelope. It is not required to call a transport `flush`. This is mostly relevant for mobile SDKs already subscribed to these hooks, such as [applicationWillTerminate](https://developer.apple.com/documentation/uikit/uiapplicationdelegate/applicationwillterminate(_:)) on iOS.
3940
4. When the application moves to the background, the BatchProcessor **SHOULD** forward all the data in memory to the transport and stop the timer. The transport **SHOULD** keep its existing behavior, which usually stores the data to disk as an envelope. It is not required to call the transport `flush`. This is mostly relevant for mobile SDKs.
40-
5. Mobile SDKs **MUST** minimize data loss when sudden process terminations occur. Refer to the [Mobile Telemetry Buffer](/sdk/telemetry/telemetry-buffer/mobile-telemetry-buffer) section for more details.
41+
5. Mobile SDKs **MUST** minimize data loss when sudden process terminations occur. Refer to the [Mobile Telemetry Processor](/sdk/telemetry/telemetry-processor/mobile-telemetry-processor) section for more details.
4142

4243
The detailed specification is written in the [Gherkin syntax](https://cucumber.io/docs/gherkin/reference/). The specification uses spans as an example, but the same applies to logs or any other future telemetry data.
4344

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: Browser Telemetry Processor
3+
description: Detailed browser telemetry processor design.
4+
sidebar_order: 2
5+
---
6+
7+
To be defined — full spec lives here.
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: GDX Telemetry Processor
3+
description: Detailed GDX telemetry processor design.
4+
sidebar_order: 3
5+
---
6+
7+
To be defined — full spec lives here.

0 commit comments

Comments
 (0)