Skip to content

Commit 386c684

Browse files
eschabelleschabell
authored
Fixing admin docs, standardization of coding examples. Fixes #2007. (#2008)
* Fixing admin troubleshooting docs, standardization. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin transport security docs, standardization code examples. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin scheduling and retries docs, standardization code examples. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin networking docs, standardization code examples. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin monitoring docs, standardization code examples. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin memory managment docs, standardization code examples. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin transport security docs, standardization code examples. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin troubleshooting docs, standardization. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin buffering and storage docs, standardization. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin backpressure docs, standardization. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> * Fixing admin multiline-parsing docs, standardization. Part of issue #2007. Signed-off-by: Eric D. Schabell <[email protected]> --------- Signed-off-by: Eric D. Schabell <[email protected]>
1 parent e5932f0 commit 386c684

File tree

9 files changed

+488
-591
lines changed

9 files changed

+488
-591
lines changed

‎administration/backpressure.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
# Backpressure
22

3-
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=63e37cfe-9ce3-4a18-933a-76b9198958c1" />
4-
53
It's possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. A common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates _backpressure_, leading to high memory consumption in the service.
64

75
To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
@@ -70,4 +68,4 @@ With `storage.type filesystem` and `storage.max_chunks_up`, the following log me
7068
```text
7169
[input] {input name or alias} paused (storage buf overlimit)
7270
[input] {input name or alias} resume (storage buf overlimit)
73-
```
71+
```

‎administration/buffering-and-storage.md

Lines changed: 81 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -56,27 +56,26 @@ Choose your preferred format for an example input definition:
5656

5757
```yaml
5858
pipeline:
59-
inputs:
60-
- name: tcp
61-
listen: 0.0.0.0
62-
port: 5170
63-
format: none
64-
tag: tcp-logs
65-
mem_buf_limit: 50MB
59+
inputs:
60+
- name: tcp
61+
listen: 0.0.0.0
62+
port: 5170
63+
format: none
64+
tag: tcp-logs
65+
mem_buf_limit: 50MB
6666
```
6767
6868
{% endtab %}
69-
7069
{% tab title="fluent-bit.conf" %}
7170
7271
```text
7372
[INPUT]
74-
Name tcp
75-
Listen 0.0.0.0
76-
Port 5170
77-
Format none
78-
Tag tcp-logs
79-
Mem_Buf_Limit 50MB
73+
Name tcp
74+
Listen 0.0.0.0
75+
Port 5170
76+
Format none
77+
Tag tcp-logs
78+
Mem_Buf_Limit 50MB
8079
```
8180

8281
{% endtab %}
@@ -89,8 +88,9 @@ If this input uses more than 50&nbsp;MB memory to buffer logs, you will get a wa
8988
```
9089

9190
{% hint style="info" %}
92-
`mem_buf_Limit` applies only when `storage.type` is set to the default value of
93-
`memory`.
91+
92+
`m em_buf_Limit` applies only when `storage.type` is set to the default value of `memory`.
93+
9494
{% endhint %}
9595

9696
#### Filesystem buffering
@@ -156,28 +156,27 @@ A Service section will look like this:
156156

157157
```yaml
158158
service:
159-
flush: 1
160-
log_level: info
161-
storage.path: /var/log/flb-storage/
162-
storage.sync: normal
163-
storage.checksum: off
164-
storage.backlog.mem_limit: 5M
165-
storage.backlog.flush_on_shutdown: off
159+
flush: 1
160+
log_level: info
161+
storage.path: /var/log/flb-storage/
162+
storage.sync: normal
163+
storage.checksum: off
164+
storage.backlog.mem_limit: 5M
165+
storage.backlog.flush_on_shutdown: off
166166
```
167167
168168
{% endtab %}
169-
170169
{% tab title="fluent-bit.conf" %}
171170
172171
```text
173172
[SERVICE]
174-
flush 1
175-
log_Level info
176-
storage.path /var/log/flb-storage/
177-
storage.sync normal
178-
storage.checksum off
179-
storage.backlog.mem_limit 5M
180-
storage.backlog.flush_on_shutdown off
173+
flush 1
174+
log_Level info
175+
storage.path /var/log/flb-storage/
176+
storage.sync normal
177+
storage.checksum off
178+
storage.backlog.mem_limit 5M
179+
storage.backlog.flush_on_shutdown off
181180
```
182181

183182
{% endtab %}
@@ -201,44 +200,43 @@ The following example configures a service offering filesystem buffering capabil
201200

202201
```yaml
203202
service:
204-
flush: 1
205-
log_level: info
206-
storage.path: /var/log/flb-storage/
207-
storage.sync: normal
208-
storage.checksum: off
209-
storage.max_chunks_up: 128
210-
storage.backlog.mem_limit: 5M
203+
flush: 1
204+
log_level: info
205+
storage.path: /var/log/flb-storage/
206+
storage.sync: normal
207+
storage.checksum: off
208+
storage.max_chunks_up: 128
209+
storage.backlog.mem_limit: 5M
211210

212211
pipeline:
213-
inputs:
214-
- name: cpu
215-
storage.type: filesystem
212+
inputs:
213+
- name: cpu
214+
storage.type: filesystem
216215

217-
- name: mem
218-
storage.type: memory
216+
- name: mem
217+
storage.type: memory
219218
```
220219
221220
{% endtab %}
222-
223221
{% tab title="fluent-bit.conf" %}
224222
225223
```text
226224
[SERVICE]
227-
flush 1
228-
log_Level info
229-
storage.path /var/log/flb-storage/
230-
storage.sync normal
231-
storage.checksum off
232-
storage.max_chunks_up 128
233-
storage.backlog.mem_limit 5M
225+
flush 1
226+
log_Level info
227+
storage.path /var/log/flb-storage/
228+
storage.sync normal
229+
storage.checksum off
230+
storage.max_chunks_up 128
231+
storage.backlog.mem_limit 5M
234232

235233
[INPUT]
236-
name cpu
237-
storage.type filesystem
234+
name cpu
235+
storage.type filesystem
238236

239237
[INPUT]
240-
name mem
241-
storage.type memory
238+
name mem
239+
storage.type memory
242240
```
243241

244242
{% endtab %}
@@ -259,50 +257,49 @@ The following example creates records with CPU usage samples in the filesystem w
259257

260258
```yaml
261259
service:
262-
flush: 1
263-
log_level: info
264-
storage.path: /var/log/flb-storage/
265-
storage.sync: normal
266-
storage.checksum: off
267-
storage.max_chunks_up: 128
268-
storage.backlog.mem_limit: 5M
260+
flush: 1
261+
log_level: info
262+
storage.path: /var/log/flb-storage/
263+
storage.sync: normal
264+
storage.checksum: off
265+
storage.max_chunks_up: 128
266+
storage.backlog.mem_limit: 5M
269267

270268
pipeline:
271-
inputs:
272-
- name: cpu
273-
storage.type: filesystem
274-
275-
outputs:
276-
- name: stackdriver
277-
match: '*'
278-
storage.total_limit_size: 5M
269+
inputs:
270+
- name: cpu
271+
storage.type: filesystem
272+
273+
outputs:
274+
- name: stackdriver
275+
match: '*'
276+
storage.total_limit_size: 5M
279277
```
280278
281279
{% endtab %}
282-
283280
{% tab title="fluent-bit.conf" %}
284281
285282
```text
286283
[SERVICE]
287-
flush 1
288-
log_Level info
289-
storage.path /var/log/flb-storage/
290-
storage.sync normal
291-
storage.checksum off
292-
storage.max_chunks_up 128
293-
storage.backlog.mem_limit 5M
284+
flush 1
285+
log_Level info
286+
storage.path /var/log/flb-storage/
287+
storage.sync normal
288+
storage.checksum off
289+
storage.max_chunks_up 128
290+
storage.backlog.mem_limit 5M
294291

295292
[INPUT]
296-
name cpu
297-
storage.type filesystem
293+
name cpu
294+
storage.type filesystem
298295

299296
[OUTPUT]
300-
name stackdriver
301-
match *
302-
storage.total_limit_size 5M
297+
name stackdriver
298+
match *
299+
storage.total_limit_size 5M
303300
```
304301

305302
{% endtab %}
306303
{% endtabs %}
307304

308-
If Fluent Bit is offline because of a network issue, it will continue buffering CPU samples, keeping a maximum of 5&nbsp;MB of the newest data.
305+
If Fluent Bit is offline because of a network issue, it will continue buffering CPU samples, keeping a maximum of 5&nbsp;MB of the newest data.

0 commit comments

Comments
 (0)