Storage irrecoverable (#1106)

* administration:buffering-and-storage adding irrecoverable definition

Signed-off-by: RicardoAAD <[email protected]>

* administration:buffering-and-storage fixing wording corrupted to irrecoverable

Signed-off-by: RicardoAAD <[email protected]>

* administration:buffering-and-storage modifying descirption of the option storage.delete_irrecoverable_chunks

Signed-off-by: RicardoAAD <[email protected]>

* Added link to irrecoverable chunks section next to the definition of delete_irrecoverable_chunks option.

Signed-off-by: RicardoAAD <[email protected]>

* administration:buffering-and-storage.md: fixing typo

Signed-off-by: RicardoAAD <[email protected]>

---------

Signed-off-by: RicardoAAD <[email protected]>

Author: RicardoAAD

administration/buffering-and-storage.md

@@ -14,6 +14,16 @@ Understanding the chunks, buffering and backpressure concepts is critical for a
 
 When an input plugin \(source\) emits records, the engine groups the records together in a _Chunk_. A Chunk size usually is around 2MB. By configuration, the engine decides where to place this Chunk, the default is that all chunks are created only in memory.
 
+#### Irrecoverable Chunks
+
+There are two scenarios where fluent-bit marks chunks as irrecoverable:
+
+* When Fluent Bit encounters a bad layout in a chunk. A bad layout is a chunk that does not conform to the expected format. [Chunk definition](https://github.com/fluent/fluent-bit/blob/master/CHUNKS.md)
+
+* When Fluent Bit encounters an incorrect or invalid chunk header size.
+
+In both scenarios Fluent-Bit will log an error message and then discard the irrecoverable chunks.
+
 #### Buffering and Memory
 
 As mentioned above, the Chunks generated by the engine are placed in memory but this is configurable.
@@ -105,7 +115,7 @@ The Service section refers to the section defined in the main [configuration fil
 | storage.max\_chunks\_up | If the input plugin has enabled `filesystem` storage type, this property sets the maximum number of Chunks that can be `up` in memory. *This is the setting to use to control memory usage when you enable `storage.type filesystem`*. | 128 |
 | storage.backlog.mem\_limit | If _storage.path_ is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called _backlog_ data. _Backlog chunks_ are filesystem chunks that were left over from a previous Fluent Bit run; chunks that could not be sent before exit that Fluent Bit will pick up when restarted. Fluent Bit will check the `storage.backlog.mem_limit` value against the current memory usage from all `up` chunks for the input. If the `up` chunks currently consume less memory than the limit, it will bring the _backlog_ chunks up into memory so they can be sent by outputs. | 5M |
 | storage.metrics | If `http_server` option has been enabled in the main `[SERVICE]` section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the [Monitoring](monitoring.md) section. | off |
-| storage.delete_irrecoverable_chunks | When enabled, corrupted chunks will be deleted during runtime, and the previously corrupted chunks in the storage path will be deleted when Fluent-Bit starts. | Off |
+| storage.delete_irrecoverable_chunks | When enabled, [irrecoverable chunks](./buffering-and-storage.md#irrecoverable-chunks) will be deleted during runtime, and any other irrecoverable chunk located in the configured storage path directory will be deleted when Fluent-Bit starts. | Off |
 
 a Service section will look like this: