You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's possible for logs or data to be ingested or created faster than the ability toflush it to some destinations. A common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates _backpressure_, leading to high memory consumption in the service.
5
+
It's possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. A common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates _backpressure_, leading to high memory consumption in the service.
6
6
7
7
To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
8
8
9
9
As described in the [Buffering](../concepts/buffering.md) concepts section, Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory and filesystem (optional).
10
10
11
11
The default `storage.type memory` buffer can be restricted with `Mem_Buf_Limit`. If memory reaches this limit and you reach a backpressure scenario, you won't be able to ingest more data until the data chunks that are in memory can be flushed. The input pauses and Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1334) a `[warn] [input] {input name or alias} paused (mem buf overlimit)` log message.
12
12
13
-
Depending on the input plugin in use, this might cause incoming data to be discarded (for example, TCP input plugin). The tail plugin can handle pauses without data ingloss, storing its current file offset and resuming reading later. When buffer memory is available, the input resumes accepting logs. Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1277) a `[info] [input] {input name or alias} resume (mem buf overlimit)` message.
13
+
Depending on the input plugin in use, this might cause incoming data to be discarded (for example, TCP input plugin). The tail plugin can handle pauses without data loss, storing its current file offset and resuming reading later. When buffer memory is available, the input resumes accepting logs. Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1277) a `[info] [input] {input name or alias} resume (mem buf overlimit)` message.
14
14
15
-
Mitigate the risk of data loss by configuring secondary storage on the filesystem using the `storage.type` of `filesystem` (as described in [Buffering & Storage](buffering-and-storage.md)). Initially, logs will be buffered to both memory and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer.
15
+
Mitigate the risk of data loss by configuring secondary storage on the filesystem using the `storage.type` of `filesystem` (as described in [Buffering and Storage](buffering-and-storage.md)). Initially, logs will be buffered to both memory and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer.
16
16
17
17
## `Mem_Buf_Limit`
18
18
@@ -38,7 +38,7 @@ In a few seconds, if the scheduler was able to flush the initial 700 KB of
38
38
39
39
- Upon data buffer release (700 KB), the internal counters get updated.
40
40
- Counters now are set at 500 KB.
41
-
- Because 500 KB isless than 1 MB, it checks the input plugin state.
41
+
- Because 500 KB is less than 1 MB, it checks the input plugin state.
42
42
- If the plugin is paused, it invokes a `resume` callback.
43
43
- The input plugin can continue appending more data.
44
44
@@ -50,7 +50,7 @@ The setting behaves similar to the `Mem_Buf_Limit` scenario when the non-default
50
50
51
51
When (default) `storage.pause_on_chunks_overlimit` is disabled, the input won't pause when the memory limit is reached. Instead, it switches to buffering logs only in the filesystem. Limit the disk spaced used for filesystem buffering with `storage.total_limit_size`.
52
52
53
-
See [Buffering & Storage](buffering-and-storage.md) docs for more information.
53
+
See [Buffering and Storage](buffering-and-storage.md) docs for more information.
@@ -92,7 +92,7 @@ By default, the HTTP input plugin uses plain TCP. Run the following command to e
92
92
```
93
93
94
94
{% hint style="info" %}
95
-
See Tips & Trick section below for details on generating `self_signed.crt` and `self_signed.key` files shown in these examples.
95
+
See the Tips and Tricks section below for details on generating `self_signed.crt` and `self_signed.key` files shown in these examples.
96
96
{% endhint %}
97
97
98
98
In the previous command, the two properties `tls` and `tls.verify` are set for demonstration purposes. Always enable verification in production environments.
Copy file name to clipboardExpand all lines: concepts/buffering.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,4 +14,4 @@ Fluent Bit buffering strategies are designed to solve problems associated with b
14
14
15
15
These mechanisms aren't mutually exclusive. When data is ready to be processed or delivered it's always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.
16
16
17
-
To learn more about the buffering configuration in Fluent Bit, see [Buffering & Storage](../administration/buffering-and-storage.md).
17
+
To learn more about the buffering configuration in Fluent Bit, see [Buffering and Storage](../administration/buffering-and-storage.md).
0 commit comments