Change to DoubleRotatingBuffer

Author: philipphofmann

develop-docs/sdk/telemetry/spans/batch-processor.mdx

@@ -105,11 +105,11 @@ Each SDK environment is unique. Therefore, SDKs have three options to choose fro
 
 When the SDK detects a sudden process termination, it MUST put all remaining items in the BatchProcessor into one envelope and flush it. If your SDK has an offline cache, it MAY flush the envelope to disk and skip sending it to Sentry, if it ensures to send the envelope the next time the SDK starts. The BatchProcessor MUST keep its existing logic described in the [specification](#specification) above.
 
-Suppose your SDK can't reliably detect sudden process terminations, or it can't reliably flush envelopes to Sentry or disk when a sudden process termination happens. In that case, it SHOULD implement the [FileStream Cache](#2-file-stream-cache) or the [FIFO Queue with Async FileStream Cache](#3-fifo-queue-with-async-file-stream-cache). It's acceptable to start with this option as a best effort interim solution before adding one of the more complex options.
+Suppose your SDK can't reliably detect sudden process terminations, or it can't reliably flush envelopes to Sentry or disk when a sudden process termination happens. In that case, it SHOULD implement the [FileStream Cache](#2-file-stream-cache) or the [DoubleRotatingBuffer](#3-doublerotatingbuffer). It's acceptable to start with this option as a best effort interim solution before adding one of the more complex options.
 
 ### 2. FileStream Cache
 
-SDKs for which blocking the main thread is a nogo, such as Android and Apple, SDKs MUST NOT implement this option. They SHOULD implement the [FIFO Queue with Async FileStream Cache](#3-fifo-queue-with-async-file-stream-cache).
+SDKs for which blocking the main thread is a nogo, such as Android and Apple, SDKs MUST NOT implement this option. They SHOULD implement the [DoubleRotatingBuffer](#3-doublerotatingbuffer).
 
 With this option, the BatchProcessor stores the data on the calling thread directly to disk. The SDK SHOULD store the BatchProcessor files in a folder that is a sibling of the `envelopes` or `replay` folder, named `batch-processor`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. In the `batch-processor` folder, the SDK MUST store two types of cache files:
 
@@ -119,68 +119,71 @@ With this option, the BatchProcessor stores the data on the calling thread direc
 When the timeout expires or the cache file hits the size limit, the BatchProcessor renames the `cache` file to `flushing`, creates a new `cache` file for incoming data, converts the data in the `flushing` file to an envelope, sends it to Sentry, and then deletes the `flushing` file. When the SDK starts again, it MUST check if there are any cache files in the cache directory (both `cache` and `flushing`) and if so, it MUST load the data from the files and send it to Sentry.
 
 
-### 3. FIFO Queue with Async FileStream Cache
+### 3. DoubleRotatingBuffer
 
 SDKs should only consider implementing this option when options 1 or 2 are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or to consistently store envelopes to disk during such terminations, as can occur with Android or Apple devices.
 
-With this option, the BatchProcessor first stores its data in a thread-safe FIFO queue, residing in an async-safe memory space, allowing the crash reporter to write to disk when a crash occurs. Furthermore, the BatchProcessor stores data asynchronously into a file, allowing it to recover after an abnormal termination, for which the crash handler can't run.
+The BatchProcessor uses two buffers to minimize data loss in the event of an abnormal process termination:
+* **Crash-safe list**: A list stored in a crash-safe space to prevent data loss during detectable abnormal process terminations.
+* **Async IO cache**: When a process terminates abruptly, the crash-safe list loses all its elements. Therefore, the BatchProcessor uses a second buffer, the async IO cache, that stores elements to disk on a background thread to avoid blocking the calling thread.
 
-The BatchProcessor maintains its logic of batching multiple logs and spans together into a single envelope to avoid multiple HTTP requests.
+Furthermore, the BatchProcessor MUST prevent data loss when flushing. Therefore, it uses a double-buffering solution, meaning the two buffers alternate. The crash-safe list has two lists, and the async IO buffer has two files. When list1 is full, the BatchProcessor stores items in list2 until it successfully stores items in list1 to disk as an envelope. Then it can delete items in list1. The same applies to the IO buffer.
 
-Hybrid SDKs pass every log and span down to the native SDKs, which will put every log and span in their BatchProcessor and its cache when logs and spans are ready for sending, meaning after they go through beforeLog, integrations, processors, etc.
+When the SDK detects an abnormal process termination, it stores items in the crash-safe list to disk. The next time the SDK starts, it sends these items. When an undetectable process termination occurs, the SDK loses items from the crash-safe list that have not yet been stored in the async IO buffer, which we intentionally accept rather than blocking the calling thread. Furthermore, the SDK must deduplicate items in the stored crash-safe list and the IO buffer by item ID to avoid sending duplicates.
 
-#### Receiving Data
+#### BatchProcessor Files
 
-When the BatchProcessor receives data, it performs the following steps
+The SDK SHOULD store the BatchProcessor files in a folder that is a sibling of the `envelopes` or `replay` folder, named `batch-processor`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. The `batch-processor` folder SHOULD contain the following files:
 
-1. Put the item into the FIFO queue on the calling thread.
-2. On a background thread, serialize the next item of the FIFO queue and store it in the `cache` file.
-3. Remove the log from the FIFO queue.
-4. If the queue isn't empty, go back to step 2.
+- `file-buffer1` and `file-buffer2` - The active IO buffers for the BatchProcessor.
+- `detected-termination-x` - The file containing items from a previous detected abnormal termination.
+- `envelope-to-flush-x` - The envelope that the BatchProcessor is about to move to the envelopes cache folder, so the SDK can send it to Sentry.
 
-The FIFO queue has a `max-item-count` of 64. When the FIFO queue exceeds `max-item-count`, the BatchProcessor MUST drop items and record client reports with the category `queue_overflow` for every dropped item. SDKs MAY choose a different `max-item-count` value, if needed. SDKs MUST NOT expose the `max-item-count` value to users as an option. We can make this option public in the future, if required.
 
-#### Abnormal Process Termination
+#### Receiving Items
 
-When SDKs detect an abnormal process termination, they MUST write the items in the FIFO queue to the `abnormal-termination-x` file where `x` is the an increasing index of the file starting from 0.
+The BatchProcessor has two lists `crash-safe-list1` and `crash-safe-list2` and two files `file-buffer1` and `file-buffer2`. When it receives items, it performs the following steps
 
-When the process terminates abnormally and the SDKs can't detect it, the SDKs lose items in the FIFO queue, which we accept over blocking the calling thread that could be the main thread.
+1. Put the item into the crash-safe `crash-safe-list1` on the calling thread.
+2. On a background thread, store the item in the `file-buffer1`.
 
-No matter the abnormal process termination, SDKs MUST send items both in the `abnormal-termination-x` and `cache` files on the next SDK start. Please refer to the [SDK start](#sdk-start) section for the detailed specification.
+#### Flushing
 
-#### Cache File Location
+When the `crash-safe-list1` exceeds the [above described](#specification) 1MiB in size, the BatchProcessor performs the following flushing steps:
 
-The SDK SHOULD store the BatchProcessor cache files in a folder that is a sibling of the `envelopes` or `replay` folder, named `batch-processor`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. The `batch-processor` folder MAY contain the follow files, where `x` is the an increasing index of the file starting from 0:
+1. Store new incoming items in `crash-safe-list2` and `file-buffer2`.
+2. Put the items of `crash-safe-list1` into an envelope named `envelope-to-flush-x`, where `x` is the an increasing index of the file starting from 0, in the same folder as the BatchProcessor files.
+3. Delete the items in `crash-safe-list1` and `file-buffer1`.
+4. Move the envelope to the envelopes cache folder.
 
-- `cache` - The active writing cache file for the BatchProcessor.
-- `cache-to-flush-x` - The cache file that the BatchProcessor is about to convert to an envelope.
-- `abnormal-termination-fifo-queue-x` - The file containing the data from the FIFO queue when an abnormal process termination occurs.
-- `abnormal-termination-cache-x` - The cache file containing items from a previous abnormal termination.
-- `envelope-to-flush-x` - The envelope that the BatchProcessor is about to move to the envelopes cache folder, so the SDK can send it to Sentry.
+The BatchProcessor stores the envelope-to-flush not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items `crash-safe-list1` and `file-buffer1`, the SDKs would send duplicate items.
 
-#### Flushing
 
-The BatchProcessor MUST keep two cache files. When the BatchProcessor sends the items from `cache`, it renames it to `cache-to-flush-x` and creates a new `cache` to avoid losing items if an abnormal termination occurs when flushing the items. To avoid sending duplicate items if an abnormal termination occurs between storing the envelope and deleting the cache file, the BatchProcessor MUST first store the envelope to the same folder as the BatchProcessor files. After deleting the `cache-to-flush-x`, SDKs MUST move the envelope to the envelope cache folder. As moving files is usually atomic, SDKs avoid sending duplicated items in the described scenario. These are the flushing steps:
+#### Abnormal Process Termination
+
+When SDKs detect an abnormal process termination, they MUST write the items in both `crash-safe-list1` and `crash-safe-list2` to the `detected-abnormal-termination-x` file where `x` is the an increasing index of the file starting from 0.
 
-1. Rename `cache` to `cache-to-flush-x` where `x` is the an increasing index of the cache file starting from 0.
-2. Create a new `cache` file and store new items to this file.
-3. Move the items in `cache-to-flush-x` to an envelope named `envelope-to-flush-x`. SDKs should utilize a file stream or a similar mechanism when moving the items to prevent loading all items into memory and causing a short-term memory spike. SDKs MUST not put the envelope into the envelope cache folder yet. Instead, they MUST store the envelope to the same folder as the other BatchProcessor files.
-4. Delete the file `cache-to-flush-x`.
-5. Move the `envelope-to-flush-x` to the envelopes cache folder.
+When the process terminates abnormally and the SDKs can't detect it, the SDKs lose items in the crash safe lists, which we accept over blocking the calling thread that could be the main thread.
 
-#### SDK start
+#### SDK Initialization
 
-Whenever the SDK starts, it must check if there is any data in the batch processor folder that needs to be recovered. SDKs MUST perform the following steps when starting:
+Whenever the SDKs initialize, they must check if there is any data in the batch processor folder that needs to be recovered. SDKs MUST perform the following steps when initializing:
 
-1. If there are items in the `cache` file, rename the `cache` file to `abnormal-termination-cache-x`.
-2. Create a new `cache` file and store new items to this file.
-3. If there is an `abnormal-termination-fifo-queue-x` file, deduplicate data from both the `abnormal-termination-cache-x` and `abnormal-termination-fifo-queue-x` file based on the IDs of the items.
+1. If there are items in the `file-buffer1` or `file-buffer2` file, store all items into a file named `undetected-termination-x`.
+2. Create new `file-buffer1` and `file-buffer2` files and store new items to this file.
+3. Load all items from the `undetected-termination-x` and `detected-termination-x` and deduplicate them based on the IDs of the items.
 4. Put the deduplicated items into the `envelope-to-flush-x` in the batch processor cache folder.
-5. Delete the `abnormal-termination-cache-x` and `abnormal-termination-fifo-queue-x` files.
+5. Delete the `undetected-termination-x` and `detected-termination-x` files.
 6. Move the `envelope-to-flush-x` to the envelopes cache folder.
 
-As abnormal terminations can occur at any time, there may be multiple `abnormal-termination-cache-x`, `abnormal-termination-fifo-queue-x` or `envelope-to-flush-x` files. SDKs MUST handle multiple file pairs at each of the above-described steps. For example, if there are two pairs of `abnormal-termination-cache-x` and `abnormal-termination-fifo-queue-x`, the SDKs should perform steps 3 to 6 for both pairs.
+As abnormal terminations can occur at any time, there may be multiple `undetected-termination-x` and `detected-termination-x` files. SDKs MUST handle multiple file pairs at each of the above-described steps. For example, if there are two pairs of `undetected-termination-x` and `detected-termination-x`, the SDKs should perform steps 3 to 6 for both pairs.
 
 #### SDK Closes
 
-Whenever the users closes the SDK, the BatchProcessor MUST perform the steps described in the [Flushing](#flushing) section.
+Whenever the users closes the SDK or the application terminates normally, the BatchProcessor MUST perform the steps described in the [Flushing](#flushing) section and the SDK MUST delete all items in the `file-buffer1` and `file-buffer2` files.
+
+#### Miscellaneous
+
+The BatchProcessor maintains its logic of batching multiple logs and spans together into a single envelope to avoid multiple HTTP requests.
+
+Hybrid SDKs pass every log and span down to the native SDKs, which will put every log and span in their BatchProcessor and its cache when logs and spans are ready for sending, meaning after they go through beforeLog, integrations, processors, etc.