move it to doc comment

Author: lalitb

opentelemetry-sdk/src/logs/log_processor.rs

@@ -264,94 +264,92 @@ type LogsData = Box<(LogRecord, InstrumentationScope)>;
 ///     .with_log_processor(processor)
 ///     .build();
 ///
+/// **Memory Management in BatchLogProcessor**
 ///
-// **Memory Management in BatchLogProcessor**
-//
-// The `BatchLogProcessor` manages memory through the following stages of log processing:
-//
-// 1. **Record Ingestion**:
-//    - Each `LogRecord` is **cloned** upon entering the processor.
-//    - `LogRecordAttributes` utilize a hybrid memory model:
-//      - First 5 attributes are **stack-allocated**.
-//      - Adding additional attribtues trigger **heap allocation** in a dynamically growing vector.
-//    - The `LogRecord` and its associated `InstrumentationScope` are **boxed together**
-//      to allocate them on the heap before entering the queue. This means:
-//      - The `LogRecord`'s inline attributes (if any) are moved to the heap as part of the boxed structure.
-//      - Any dynamically allocated data already on the heap (e.g., strings, overflow attributes) remains unaffected.
-//      - Ownership of the boxed data is transferred to the queue, ensuring it can be processed independently of the original objects.
-//
-// 2. **Queue Management**:
-//    - Uses **two bounded synchronous channels** (`sync_channel`):
-//      - One for **log records** (`logs_sender` and `logs_receiver`).
-//      - Another for **control messages** (`message_sender` and `message_receiver`).
-//
-//    - **Log Record Queue**:
-//      - Stores log records as **heap-allocated** `Box<(LogRecord, InstrumentationScope)>`.
-//      - The queue size is configurable and defined by `max_queue_size`.
-//      - If the queue is full:
-//        - New log records are **dropped**, and a warning is logged the first time this happens.
-//        - Dropped records are counted for reporting during shutdown.
-//
-//    - **Control Message Queue**:
-//      - Stores control messages (`BatchMessage`) to manage operations like exporting, force flushing, setting resources, and shutting down.
-//      - The control message queue has a fixed size (e.g., 64 messages).
-//      - Control messages are processed with higher priority, ensuring operational commands are handled promptly.
-//      - The use of a separate control queue ensures that critical commands, such as `Shutdown`, are not lost even if the log record queue is full.
-//      - Messages supported include:
-//        - `ExportLog`: Triggers an immediate export of log records.
-//        - `ForceFlush`: Flushes all buffered log records to the exporter.
-//        - `SetResource`: Updates the exporter with a new resource.
-//        - `Shutdown`: Cleans up and flushes logs before terminating the processor.
-//
-// 3. **Worker Thread Storage**:
-//    - The worker thread maintains a pre-allocated `Vec` of boxed record pairs:
-//      - The vector’s capacity is fixed at `max_export_batch_size`.
-//    - Records are **moved** (not cloned) from the log record queue to the vector for processing.
-//
-// 4. **Export Process**:
-//    - During the export process:
-//        - The worker thread retrieves records from the log record queue until `max_export_batch_size` is reached or the queue is empty.
-//        - The retrieved records are processed in batches and passed to the exporter.
-//    - The exporter's `export()` method receives references to the log records and `InstrumentationScope`.
-//        - If the exporter requires retaining the log records (e.g., for retries or asynchronous operations), it must **clone** the records inside the `export()` implementation.
-//    - After successful export:
-//        - The original boxed records are dropped, releasing heap memory.
-//    - Export is triggered in the following scenarios:
-//        - When the batch size reaches `max_export_batch_size`, resulting in `ExportLog` control message being sent to the worker thread.
-//        - When the scheduled delay timer expires.
-//        - When `force_flush` is called by the application, resulting in a `ForceFlush` control message being sent to the worker thread.
-//        - During processor shutdown initiated by the application, resulting in a `Shutdown` control message being sent to the worker thread.
-//    - Generation of `ExportLog` control message:
-//        - The `ExportLog` control message is generated by the application thread when a new record is added to the log record queue, and the current batch size reaches `max_export_batch_size`.
-//        - To prevent redundant messages, the `ExportLog` message is only sent if the previous one has been processed by the worker thread.
-//        - Upon receiving this message, the worker thread immediately processes and exports the current batch, overriding any scheduled delay.
-//
-// 5. **Memory Limits**:
-//    - **Worst-Case Memory Usage**:
-//      - **Log Record Queue Memory** = `max_queue_size * size of boxed (LogRecord + InstrumentationScope)`.
-//      - **Batch Memory** = `max_export_batch_size * size of boxed (LogRecord + InstrumentationScope)`.
-//    - **Control Message Queue Memory**:
-//      - Fixed at 64 messages, with negligible memory overhead.
-//    - **Total Maximum Memory:**
-//      - When both the log record queue and batch vector are full:
-//        ```
-//        (max_queue_size + max_export_batch_size) * size of boxed (LogRecord + InstrumentationScope)
-//        ```
-//      - The average size of a `LogRecord` is ~300 bytes ( assuming 4 attributes), and the `InstrumentationScope` is ~50 bytes assuming no attributes.
-//        - For `max_queue_size = 2048` and `max_export_batch_size = 512`, the total memory usage is ~900 KB as below:
-//          Calculation: `(2048 + 512) * (300 + 50) = 2560 * 350 = 896000 bytes = 896 KB`.
-//
-// 6. **Key Notes on Memory Behavior**:
-//    - Boxing a `LogRecord` and `InstrumentationScope` moves the record to the heap,
-//      including stack-allocated attributes.
-//    - During the export process, records are moved from the log record queue to the worker thread’s vector.
-//    - No additional cloning or copying occurs during the export process, minimizing memory overhead while ensuring efficient handling of log records.
-//
-// 7. **Control Queue Prioritization**:
-//    - Control messages take precedence over log record processing to ensure timely execution of critical operations.
-//    - For instance, a `Shutdown` message is processed before continuing with log exports, guaranteeing graceful cleanup.
-//    - The use of a separate control queue ensures responsiveness to operational commands without the risk of losing critical messages, even if the log record queue is full.
-
+/// The `BatchLogProcessor` manages memory through the following stages of log processing:
+///
+/// 1. **Record Ingestion**:
+///    - Each `LogRecord` is **cloned** upon entering the processor.
+///    - `LogRecordAttributes` utilize a hybrid memory model:
+///      - First 5 attributes are **stack-allocated**.
+///      - Adding additional attribtues trigger **heap allocation** in a dynamically growing vector.
+///    - The `LogRecord` and its associated `InstrumentationScope` are **boxed together**
+///      to allocate them on the heap before entering the queue. This means:
+///      - The `LogRecord`'s inline attributes (if any) are moved to the heap as part of the boxed structure.
+///      - Any dynamically allocated data already on the heap (e.g., strings, overflow attributes) remains unaffected.
+///      - Ownership of the boxed data is transferred to the queue, ensuring it can be processed independently of the original objects.
+///
+/// 2. **Queue Management**:
+///    - Uses **two bounded synchronous channels** (`sync_channel`):
+///      - One for **log records** (`logs_sender` and `logs_receiver`).
+///      - Another for **control messages** (`message_sender` and `message_receiver`).
+///
+///    - **Log Record Queue**:
+///      - Stores log records as **heap-allocated** `Box<(LogRecord, InstrumentationScope)>`.
+///      - The queue size is configurable and defined by `max_queue_size`.
+///      - If the queue is full:
+///        - New log records are **dropped**, and a warning is logged the first time this happens.
+///        - Dropped records are counted for reporting during shutdown.
+///
+///    - **Control Message Queue**:
+///      - Stores control messages (`BatchMessage`) to manage operations like exporting, force flushing, setting resources, and shutting down.
+///      - The control message queue has a fixed size (e.g., 64 messages).
+///      - Control messages are processed with higher priority, ensuring operational commands are handled promptly.
+///      - The use of a separate control queue ensures that critical commands, such as `Shutdown`, are not lost even if the log record queue is full.
+///      - Messages supported include:
+///        - `ExportLog`: Triggers an immediate export of log records.
+///        - `ForceFlush`: Flushes all buffered log records to the exporter.
+///        - `SetResource`: Updates the exporter with a new resource.
+///        - `Shutdown`: Cleans up and flushes logs before terminating the processor.
+///
+/// 3. **Worker Thread Storage**:
+///    - The worker thread maintains a pre-allocated `Vec` of boxed record pairs:
+///      - The vector’s capacity is fixed at `max_export_batch_size`.
+///    - Records are **moved** (not cloned) from the log record queue to the vector for processing.
+///
+/// 4. **Export Process**:
+///    - During the export process:
+///        - The worker thread retrieves records from the log record queue until `max_export_batch_size` is reached or the queue is empty.
+///        - The retrieved records are processed in batches and passed to the exporter.
+///    - The exporter's `export()` method receives references to the log records and `InstrumentationScope`.
+///        - If the exporter requires retaining the log records (e.g., for retries or asynchronous operations), it must **clone** the records inside the `export()` implementation.
+///    - After successful export:
+///        - The original boxed records are dropped, releasing heap memory.
+///    - Export is triggered in the following scenarios:
+///        - When the batch size reaches `max_export_batch_size`, resulting in `ExportLog` control message being sent to the worker thread.
+///        - When the scheduled delay timer expires.
+///        - When `force_flush` is called by the application, resulting in a `ForceFlush` control message being sent to the worker thread.
+///        - During processor shutdown initiated by the application, resulting in a `Shutdown` control message being sent to the worker thread.
+///    - Generation of `ExportLog` control message:
+///        - The `ExportLog` control message is generated by the application thread when a new record is added to the log record queue, and the current batch size reaches `max_export_batch_size`.
+///        - To prevent redundant messages, the `ExportLog` message is only sent if the previous one has been processed by the worker thread.
+///        - Upon receiving this message, the worker thread immediately processes and exports the current batch, overriding any scheduled delay.
+///
+/// 5. **Memory Limits**:
+///    - **Worst-Case Memory Usage**:
+///      - **Log Record Queue Memory** = `max_queue_size * size of boxed (LogRecord + InstrumentationScope)`.
+///      - **Batch Memory** = `max_export_batch_size * size of boxed (LogRecord + InstrumentationScope)`.
+///    - **Control Message Queue Memory**:
+///      - Fixed at 64 messages, with negligible memory overhead.
+///    - **Total Maximum Memory:**
+///      - When both the log record queue and batch vector are full:
+///        ```
+///        (max_queue_size + max_export_batch_size) * size of boxed (LogRecord + InstrumentationScope)
+///        ```
+///      - The average size of a `LogRecord` is ~300 bytes ( assuming 4 attributes), and the `InstrumentationScope` is ~50 bytes assuming no attributes.
+///        - For `max_queue_size = 2048` and `max_export_batch_size = 512`, the total memory usage is ~900 KB as below:
+///          Calculation: `(2048 + 512) * (300 + 50) = 2560 * 350 = 896000 bytes = 896 KB`.
+///
+/// 6. **Key Notes on Memory Behavior**:
+///    - Boxing a `LogRecord` and `InstrumentationScope` moves the record to the heap,
+///      including stack-allocated attributes.
+///    - During the export process, records are moved from the log record queue to the worker thread’s vector.
+///    - No additional cloning or copying occurs during the export process, minimizing memory overhead while ensuring efficient handling of log records.
+///
+/// 7. **Control Queue Prioritization**:
+///    - Control messages take precedence over log record processing to ensure timely execution of critical operations.
+///    - For instance, a `Shutdown` message is processed before continuing with log exports, guaranteeing graceful cleanup.
+///    - The use of a separate control queue ensures responsiveness to operational commands without the risk of losing critical messages, even if the log record queue is full.
 pub struct BatchLogProcessor {
     logs_sender: SyncSender<LogsData>, // Data channel to store log records and instrumentation scopes
     message_sender: SyncSender<BatchMessage>, // Control channel to store control messages for the worker thread