Skip to content

Commit ec88682

Browse files
simplify
1 parent 740c4d7 commit ec88682

File tree

1 file changed

+18
-21
lines changed

1 file changed

+18
-21
lines changed

develop-docs/sdk/telemetry/spans/batch-processor.mdx

Lines changed: 18 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -121,64 +121,61 @@ When the timeout expires or the cache file hits the size limit, the BatchProcess
121121

122122
### 3. DoubleRotatingBuffer
123123

124-
SDKs should only consider implementing this option when options [1](#1-flush-all-data) or [2](#2-file-stream-cache) are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or to consistently store envelopes to disk during such terminations, as can occur with Android or Apple devices.
124+
SDKs should only consider implementing this option when options [1](#1-flush-all-data) or [2](#2-file-stream-cache) are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or consistently store envelopes to disk during such terminations.
125125

126126
The BatchProcessor uses two buffers to minimize data loss in the event of an abnormal process termination:
127127
* **Crash-Safe List**: A list stored in a crash-safe space to prevent data loss during detectable abnormal process terminations.
128128
* **Async IO Cache**: When a process terminates without the SDK being able to detect it, the crash-safe list loses all its elements. Therefore, the BatchProcessor uses a second buffer, the async IO cache, that stores elements to disk on a background thread to avoid blocking the calling thread, which ensures minimal data loss when such terminations occur.
129129

130-
Furthermore, the BatchProcessor MUST prevent data loss when flushing. Therefore, it uses a double-buffering solution, meaning the two buffers alternate. The crash-safe list has two lists, and the async IO buffer has two files. When list1 is full, the BatchProcessor stores any new incoming items in list2 until it successfully stores items from list1 to disk as an envelope. Then it can delete items in list1. The same applies to the IO buffer.
130+
As the BatchProcessor MUST prevent data loss during flushing, it uses a double-buffering solution. The crash-safe list has two lists `crash-safe-list-1` and `crash-safe-list-2`, and the async IO cache has two files `async-io-cache-1` and `async-io-cache-2`. When `crash-safe-list-1` is full, the BatchProcessor stores any new incoming items in `crash-safe-list-2` until it successfully stores items from `crash-safe-list-1` to disk as an envelope. Then it can delete items in `crash-safe-list-1`. The same applies to the async IO cache.
131131

132132
#### BatchProcessor Files
133133

134134
The SDK SHOULD store the BatchProcessor files in a folder that is a sibling of the `envelopes` or `replay` folder, named `batch-processor`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. The `batch-processor` folder MAY contain the following files:
135135

136-
- `file-buffer1` and `file-buffer2` - The active IO buffers for the BatchProcessor.
136+
- `async-io-cache-1` and `async-io-cache-2` - The async IO cache files.
137137
- `detected-termination-x` - The file containing items from a previous detected abnormal termination.
138-
- `envelope-to-flush-x` - The envelope that the BatchProcessor is about to move to the envelopes cache folder, so the SDK can send it to Sentry, where `x` is the an increasing index of the file starting from 0.
138+
- `envelope-x` - The envelope that the BatchProcessor is about to move to the envelopes cache folder, so the SDK can send it to Sentry, where `x` is the an increasing index of the file starting from 0.
139139

140140

141141
#### Receiving Items
142142

143-
The BatchProcessor has two lists `crash-safe-list1` and `crash-safe-list2` and two files `file-buffer1` and `file-buffer2`. When it receives items, it performs the following steps
143+
The BatchProcessor has two lists `crash-safe-list-1` and `crash-safe-list-2` and two files `async-io-cache-1` and `async-io-cache-2`. When it receives items, it performs the following steps:
144144

145145
1. Put the item into the crash-safe `crash-safe-list1` on the calling thread.
146-
2. On a background thread, store the item in the `file-buffer1`.
146+
2. On a background thread, store the item in the `async-io-cache-1`.
147147

148148
#### Flushing
149149

150150
When the `crash-safe-list1` exceeds the [above described](#specification) 1MiB in size or the timeout exceeds, the BatchProcessor performs the following flushing steps:
151151

152-
1. Store new incoming items to the `crash-safe-list2` and `file-buffer2`.
153-
2. Put the items of `crash-safe-list1` into an envelope named `envelope-to-flush-x`.
154-
3. Delete the items in `crash-safe-list1` and `file-buffer1`.
155-
4. Move the `envelope-to-flush-x` to the envelopes cache folder, in which all the other envelopes are stored, so the SDK can send it to Sentry.
152+
1. Store new incoming items to the `crash-safe-list-2` and `async-io-cache-2`.
153+
2. Put the items of `crash-safe-list-1` into an envelope named `envelope-x`.
154+
3. Delete the items in `crash-safe-list-1` and `async-io-cache-1`.
155+
4. Move the `envelope-x` to the envelopes cache folder, in which all the other envelopes are stored, so the SDK can send it to Sentry.
156156

157-
The BatchProcessor stores the `envelope-to-flush-x` not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items `crash-safe-list1` and `file-buffer1`, the SDKs might send duplicate items.
157+
The BatchProcessor stores the `envelope-x` not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items `crash-safe-list-1` and `async-io-cache-1`, the SDKs might send duplicate items.
158158

159159

160160
#### Abnormal Process Termination
161161

162-
When SDKs detect an abnormal process termination, they MUST write the items in both `crash-safe-list1` and `crash-safe-list2` to the `detected-termination-x` file where `x` is the an increasing index of the file starting from 0.
162+
When SDKs detect an abnormal process termination, they MUST write the items in both `crash-safe-list-1` and `crash-safe-list-2` to the `detected-termination-x` file where `x` is the an increasing index of the file starting from 0.
163163

164164
When the process terminates abnormally and the SDKs can't detect it, the SDKs lose items in the crash safe lists, which we accept over blocking the calling thread that could be the main thread.
165165

166166
#### SDK Initialization
167167

168168
Whenever the SDKs initialize, they must check if there is any data in the batch processor folder that needs to be recovered. SDKs MUST perform the following steps when initializing:
169169

170-
1. If there are items in the `file-buffer1` or `file-buffer2` file, store all items into a file named `undetected-termination-x`.
171-
2. Create new `file-buffer1` and `file-buffer2` files and store new items to this file.
172-
3. Load all items from the `undetected-termination-x` and `detected-termination-x` and deduplicate them based on the IDs of the items.
173-
4. Put the deduplicated items into the `envelope-to-flush-x` in the batch processor cache folder.
174-
5. Delete the `undetected-termination-x` and `detected-termination-x` files.
175-
6. Move the `envelope-to-flush-x` to the envelopes cache folder.
176-
177-
As abnormal terminations can occur at any time, there may be multiple `undetected-termination-x` and `detected-termination-x` files. SDKs MUST handle multiple file pairs at each of the above-described steps. For example, if there are two pairs of `undetected-termination-x` and `detected-termination-x`, the SDKs should perform steps 3 to 6 for both pairs.
170+
1. Load all items from `async-io-cache-1`, `async-io-cache-2` and `detected-termination-x` files into memory if there are any. If the application terminates normally, these files don't exist.
171+
2. Deduplicate the items based on the IDs of the items and store the deduplicated items in the `envelope-x` file.
172+
3. Create new `async-io-cache-1` and `async-io-cache-2` files, and delete the `detected-termination-x` file.
173+
4. Now the BatchProcessor can start receiving new items again.
174+
5. Move the `envelope-x` to the envelopes cache folder.
178175

179176
#### SDK Closes
180177

181-
Whenever the users closes the SDK or the application terminates normally, the BatchProcessor MUST perform the steps described in the [Flushing](#flushing) section and the SDK MUST delete all items in the `file-buffer1` and `file-buffer2` files.
178+
Whenever the users closes the SDK or the application terminates normally, the BatchProcessor MUST perform the steps described in the [Flushing](#flushing) section and the SDK MUST delete all items in the `async-io-cache-1` and `async-io-cache-2` files.
182179

183180
#### Miscellaneous
184181

0 commit comments

Comments
 (0)