You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: develop-docs/sdk/telemetry/spans/batch-processor.mdx
+18-21Lines changed: 18 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -121,64 +121,61 @@ When the timeout expires or the cache file hits the size limit, the BatchProcess
121
121
122
122
### 3. DoubleRotatingBuffer
123
123
124
-
SDKs should only consider implementing this option when options [1](#1-flush-all-data) or [2](#2-file-stream-cache) are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or to consistently store envelopes to disk during such terminations, as can occur with Android or Apple devices.
124
+
SDKs should only consider implementing this option when options [1](#1-flush-all-data) or [2](#2-file-stream-cache) are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or consistently store envelopes to disk during such terminations.
125
125
126
126
The BatchProcessor uses two buffers to minimize data loss in the event of an abnormal process termination:
127
127
***Crash-Safe List**: A list stored in a crash-safe space to prevent data loss during detectable abnormal process terminations.
128
128
***Async IO Cache**: When a process terminates without the SDK being able to detect it, the crash-safe list loses all its elements. Therefore, the BatchProcessor uses a second buffer, the async IO cache, that stores elements to disk on a background thread to avoid blocking the calling thread, which ensures minimal data loss when such terminations occur.
129
129
130
-
Furthermore, the BatchProcessor MUST prevent data loss when flushing. Therefore, it uses a double-buffering solution, meaning the two buffers alternate. The crash-safe list has two lists, and the async IO buffer has two files. When list1 is full, the BatchProcessor stores any new incoming items in list2 until it successfully stores items from list1 to disk as an envelope. Then it can delete items in list1. The same applies to the IO buffer.
130
+
As the BatchProcessor MUST prevent data loss during flushing, it uses a double-buffering solution. The crash-safe list has two lists`crash-safe-list-1` and `crash-safe-list-2`, and the async IO cache has two files`async-io-cache-1` and `async-io-cache-2`. When `crash-safe-list-1` is full, the BatchProcessor stores any new incoming items in `crash-safe-list-2` until it successfully stores items from `crash-safe-list-1` to disk as an envelope. Then it can delete items in `crash-safe-list-1`. The same applies to the async IO cache.
131
131
132
132
#### BatchProcessor Files
133
133
134
134
The SDK SHOULD store the BatchProcessor files in a folder that is a sibling of the `envelopes` or `replay` folder, named `batch-processor`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. The `batch-processor` folder MAY contain the following files:
135
135
136
-
-`file-buffer1` and `file-buffer2` - The active IO buffers for the BatchProcessor.
136
+
-`async-io-cache-1` and `async-io-cache-2` - The async IO cache files.
137
137
-`detected-termination-x` - The file containing items from a previous detected abnormal termination.
138
-
-`envelope-to-flush-x` - The envelope that the BatchProcessor is about to move to the envelopes cache folder, so the SDK can send it to Sentry, where `x` is the an increasing index of the file starting from 0.
138
+
-`envelope-x` - The envelope that the BatchProcessor is about to move to the envelopes cache folder, so the SDK can send it to Sentry, where `x` is the an increasing index of the file starting from 0.
139
139
140
140
141
141
#### Receiving Items
142
142
143
-
The BatchProcessor has two lists `crash-safe-list1` and `crash-safe-list2` and two files `file-buffer1` and `file-buffer2`. When it receives items, it performs the following steps
143
+
The BatchProcessor has two lists `crash-safe-list-1` and `crash-safe-list-2` and two files `async-io-cache-1` and `async-io-cache-2`. When it receives items, it performs the following steps:
144
144
145
145
1. Put the item into the crash-safe `crash-safe-list1` on the calling thread.
146
-
2. On a background thread, store the item in the `file-buffer1`.
146
+
2. On a background thread, store the item in the `async-io-cache-1`.
147
147
148
148
#### Flushing
149
149
150
150
When the `crash-safe-list1` exceeds the [above described](#specification) 1MiB in size or the timeout exceeds, the BatchProcessor performs the following flushing steps:
151
151
152
-
1. Store new incoming items to the `crash-safe-list2` and `file-buffer2`.
153
-
2. Put the items of `crash-safe-list1` into an envelope named `envelope-to-flush-x`.
154
-
3. Delete the items in `crash-safe-list1` and `file-buffer1`.
155
-
4. Move the `envelope-to-flush-x` to the envelopes cache folder, in which all the other envelopes are stored, so the SDK can send it to Sentry.
152
+
1. Store new incoming items to the `crash-safe-list-2` and `async-io-cache-2`.
153
+
2. Put the items of `crash-safe-list-1` into an envelope named `envelope-x`.
154
+
3. Delete the items in `crash-safe-list-1` and `async-io-cache-1`.
155
+
4. Move the `envelope-x` to the envelopes cache folder, in which all the other envelopes are stored, so the SDK can send it to Sentry.
156
156
157
-
The BatchProcessor stores the `envelope-to-flush-x` not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items `crash-safe-list1` and `file-buffer1`, the SDKs might send duplicate items.
157
+
The BatchProcessor stores the `envelope-x` not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items `crash-safe-list-1` and `async-io-cache-1`, the SDKs might send duplicate items.
158
158
159
159
160
160
#### Abnormal Process Termination
161
161
162
-
When SDKs detect an abnormal process termination, they MUST write the items in both `crash-safe-list1` and `crash-safe-list2` to the `detected-termination-x` file where `x` is the an increasing index of the file starting from 0.
162
+
When SDKs detect an abnormal process termination, they MUST write the items in both `crash-safe-list-1` and `crash-safe-list-2` to the `detected-termination-x` file where `x` is the an increasing index of the file starting from 0.
163
163
164
164
When the process terminates abnormally and the SDKs can't detect it, the SDKs lose items in the crash safe lists, which we accept over blocking the calling thread that could be the main thread.
165
165
166
166
#### SDK Initialization
167
167
168
168
Whenever the SDKs initialize, they must check if there is any data in the batch processor folder that needs to be recovered. SDKs MUST perform the following steps when initializing:
169
169
170
-
1. If there are items in the `file-buffer1` or `file-buffer2` file, store all items into a file named `undetected-termination-x`.
171
-
2. Create new `file-buffer1` and `file-buffer2` files and store new items to this file.
172
-
3. Load all items from the `undetected-termination-x` and `detected-termination-x` and deduplicate them based on the IDs of the items.
173
-
4. Put the deduplicated items into the `envelope-to-flush-x` in the batch processor cache folder.
174
-
5. Delete the `undetected-termination-x` and `detected-termination-x` files.
175
-
6. Move the `envelope-to-flush-x` to the envelopes cache folder.
176
-
177
-
As abnormal terminations can occur at any time, there may be multiple `undetected-termination-x` and `detected-termination-x` files. SDKs MUST handle multiple file pairs at each of the above-described steps. For example, if there are two pairs of `undetected-termination-x` and `detected-termination-x`, the SDKs should perform steps 3 to 6 for both pairs.
170
+
1. Load all items from `async-io-cache-1`, `async-io-cache-2` and `detected-termination-x` files into memory if there are any. If the application terminates normally, these files don't exist.
171
+
2. Deduplicate the items based on the IDs of the items and store the deduplicated items in the `envelope-x` file.
172
+
3. Create new `async-io-cache-1` and `async-io-cache-2` files, and delete the `detected-termination-x` file.
173
+
4. Now the BatchProcessor can start receiving new items again.
174
+
5. Move the `envelope-x` to the envelopes cache folder.
178
175
179
176
#### SDK Closes
180
177
181
-
Whenever the users closes the SDK or the application terminates normally, the BatchProcessor MUST perform the steps described in the [Flushing](#flushing) section and the SDK MUST delete all items in the `file-buffer1` and `file-buffer2` files.
178
+
Whenever the users closes the SDK or the application terminates normally, the BatchProcessor MUST perform the steps described in the [Flushing](#flushing) section and the SDK MUST delete all items in the `async-io-cache-1` and `async-io-cache-2` files.
0 commit comments