Skip to content

Commit f19ebbb

Browse files
docs(sdks): Batch Processor (#13306)
This PR migrates the SpansAggregator RFC to the develop docs. As logs also have to use some aggregation, we renamed the SpansAggregator to BatchProcessor so it can be used with any type of telemetry data.
1 parent ff2ad87 commit f19ebbb

File tree

1 file changed

+83
-0
lines changed

1 file changed

+83
-0
lines changed
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
---
2+
title: Batch Processor
3+
---
4+
5+
<Alert level="warning">
6+
🚧 This document is work in progress.
7+
</Alert>
8+
9+
<Alert>
10+
This document uses key words such as "MUST", "SHOULD", and "MAY" as defined in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) to indicate requirement levels.
11+
</Alert>
12+
13+
The BatchProcessor batches spans and logs into one envelope to reduce the number of HTTP requests. When an SDK implements span streaming or logs, it MUST use a BatchProcessor, which is similar to [OpenTelemetry's Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md). The BatchProcessor holds logs and finished spans in memory and batches them together into envelopes. It uses a combination of time and size-based batching. When writing this, the BatchProcessor only handles spans and logs, but an SDK MAY use it for other telemetry data in the future.
14+
15+
## Specification
16+
17+
Whenever the SDK finishes a span or captures a log, it MUST put it into the BatchProcessor. The SDK MUST NOT put unfinished spans into the BatchProcessor.
18+
19+
The BatchProcessor MUST start a timeout of 5 seconds when the SDK adds the first span or log. When the timeout exceeds, the BatchProcessor MUST send all spans or logs, no matter how many items it contains. The SDK MAY choose a different value for the timeout, but it MUST NOT exceed 30 seconds, as this can lead to problems with the span buffer on the backend, which uses a time interval of 60 seconds for determining segments for spans.
20+
21+
The BatchProcessor MUST send all items after the SDK when containing spans or logs exceeding 1MiB in size. The SDK MAY choose a different value for the max batch size keeping the [envelope max sizes](/sdk/data-model/envelopes/#size-limits) in mind. The SDK MUST calculate the size of a span or a log to manage the BatchProcessor's memory footprint. The SDK MUST serialize the span or log and calculate the size based on the serialized JSON bytes. As serialization is expensive, the BatchProcessor SHOULD keep track of the serialized spans and logs and pass these to the envelope to avoid serializing multiple times.
22+
23+
When the BatchProcessor sends all spans or logs, it MUST reset its timeout and remove all spans and logs. The SDK MUST apply filtering and sampling before adding spans or logs to the BatchProcessor. The SDK MUST apply rate limits to spans and logs after they leave the BatchProcessor to send as much data as possible by dropping data as late as possible.
24+
25+
The detailed specification is written in the [Gherkin syntax](https://cucumber.io/docs/gherkin/reference/). The specification uses spans as an example, but the same applies to logs or any other future telemetry data.
26+
27+
28+
```Gherkin
29+
Scenario: No spans in BatchProcessor 1 span added
30+
Given no spans in the BatchProcessor
31+
When the SDK finishes 1 span
32+
Then the SDK puts this span to the BatchProcessor
33+
And starts a timeout of 5 seconds
34+
And doesn't send the span to Sentry
35+
36+
Scenario: Span added before timeout exceeds
37+
Given span A in the BatchProcessor
38+
Given 4.9 seconds pass
39+
When the SDK finishes span B
40+
Then the SDK adds span B to the BatchProcessor
41+
And doesn't reset the timeout
42+
And doesn't send the spans A and B in the BatchProcessor to Sentry
43+
44+
Scenario: Spans with size of 1 MiB - 1 byte added, timeout exceeds
45+
Given spans with size of 1 MiB - 1 byte in the BatchProcessor
46+
When the timeout exceeds
47+
Then the SDK adds all the spans to one envelope
48+
And sends them to Sentry
49+
And resets the timeout
50+
And clears the BatchProcessor
51+
52+
Scenario: Spans with size of 1 MiB - 1 byte added within 4.9 seconds
53+
Given spans with size of 1 MiB - 1 byte in the BatchProcessor
54+
When the SDK finishes another span and puts it into the BatchProcessor
55+
Then the BatchProcessor puts all spans into one envelope
56+
And sends the envelope to Sentry
57+
And resets the timeout
58+
And clears the BatchProcessor
59+
60+
Scenario: Unfinished spans
61+
Given no span is in the BatchProcessor
62+
When the SDK starts a span but doesn't finish it
63+
Then the BatchProcessor is empty
64+
65+
Scenario: Span filtered out
66+
Given no span is in the BatchProcessor
67+
When the finishes a span
68+
And the span is filtered out
69+
Then the BatchProcessor is empty
70+
71+
Scenario: Span not sampled
72+
Given no span is in the BatchProcessor
73+
When the finishes a span
74+
And the span is not sampled
75+
Then the BatchProcessor is empty
76+
77+
Scenario: 1 span added application crashes
78+
Given 1 span in the SpansAggregator
79+
When the SDK detects a crash
80+
Then the SDK does nothing with the items in the BatchProcessor
81+
And loses the spans in the BatchProcessor
82+
83+
```

0 commit comments

Comments
 (0)