Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions administration/backpressure.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# Backpressure

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=63e37cfe-9ce3-4a18-933a-76b9198958c1" />

It's possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. A common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates _backpressure_, leading to high memory consumption in the service.

To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
Expand Down Expand Up @@ -70,4 +68,4 @@ With `storage.type filesystem` and `storage.max_chunks_up`, the following log me
```text
[input] {input name or alias} paused (storage buf overlimit)
[input] {input name or alias} resume (storage buf overlimit)
```
```
165 changes: 81 additions & 84 deletions administration/buffering-and-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,27 +56,26 @@ Choose your preferred format for an example input definition:

```yaml
pipeline:
inputs:
- name: tcp
listen: 0.0.0.0
port: 5170
format: none
tag: tcp-logs
mem_buf_limit: 50MB
inputs:
- name: tcp
listen: 0.0.0.0
port: 5170
format: none
tag: tcp-logs
mem_buf_limit: 50MB
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[INPUT]
Name tcp
Listen 0.0.0.0
Port 5170
Format none
Tag tcp-logs
Mem_Buf_Limit 50MB
Name tcp
Listen 0.0.0.0
Port 5170
Format none
Tag tcp-logs
Mem_Buf_Limit 50MB
```

{% endtab %}
Expand All @@ -89,8 +88,9 @@ If this input uses more than 50&nbsp;MB memory to buffer logs, you will get a wa
```

{% hint style="info" %}
`mem_buf_Limit` applies only when `storage.type` is set to the default value of
`memory`.

`m em_buf_Limit` applies only when `storage.type` is set to the default value of `memory`.

{% endhint %}

#### Filesystem buffering
Expand Down Expand Up @@ -156,28 +156,27 @@ A Service section will look like this:

```yaml
service:
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.backlog.mem_limit: 5M
storage.backlog.flush_on_shutdown: off
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.backlog.mem_limit: 5M
storage.backlog.flush_on_shutdown: off
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_Level info
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M
storage.backlog.flush_on_shutdown off
flush 1
log_Level info
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M
storage.backlog.flush_on_shutdown off
```

{% endtab %}
Expand All @@ -201,44 +200,43 @@ The following example configures a service offering filesystem buffering capabil

```yaml
service:
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.max_chunks_up: 128
storage.backlog.mem_limit: 5M
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.max_chunks_up: 128
storage.backlog.mem_limit: 5M

pipeline:
inputs:
- name: cpu
storage.type: filesystem
inputs:
- name: cpu
storage.type: filesystem

- name: mem
storage.type: memory
- name: mem
storage.type: memory
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_Level info
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.max_chunks_up 128
storage.backlog.mem_limit 5M
flush 1
log_Level info
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.max_chunks_up 128
storage.backlog.mem_limit 5M

[INPUT]
name cpu
storage.type filesystem
name cpu
storage.type filesystem

[INPUT]
name mem
storage.type memory
name mem
storage.type memory
```

{% endtab %}
Expand All @@ -259,50 +257,49 @@ The following example creates records with CPU usage samples in the filesystem w

```yaml
service:
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.max_chunks_up: 128
storage.backlog.mem_limit: 5M
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.max_chunks_up: 128
storage.backlog.mem_limit: 5M

pipeline:
inputs:
- name: cpu
storage.type: filesystem

outputs:
- name: stackdriver
match: '*'
storage.total_limit_size: 5M
inputs:
- name: cpu
storage.type: filesystem

outputs:
- name: stackdriver
match: '*'
storage.total_limit_size: 5M
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_Level info
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.max_chunks_up 128
storage.backlog.mem_limit 5M
flush 1
log_Level info
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.max_chunks_up 128
storage.backlog.mem_limit 5M

[INPUT]
name cpu
storage.type filesystem
name cpu
storage.type filesystem

[OUTPUT]
name stackdriver
match *
storage.total_limit_size 5M
name stackdriver
match *
storage.total_limit_size 5M
```

{% endtab %}
{% endtabs %}

If Fluent Bit is offline because of a network issue, it will continue buffering CPU samples, keeping a maximum of 5&nbsp;MB of the newest data.
If Fluent Bit is offline because of a network issue, it will continue buffering CPU samples, keeping a maximum of 5&nbsp;MB of the newest data.
Loading