You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: administration/networking.md
+8-4Lines changed: 8 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -73,9 +73,13 @@ plugin issues a retry.
73
73
74
74
### Listener backlog
75
75
76
-
When Fluent Bit listens for incoming connections (for example, in input plugins like HTTP, TCP, OpenTelemetry, Forward, Syslog, etc.), the operating system maintains a queue of pending connections. The `net.backlog` option controls the maximum number of pending connections that can be queued before new connection attempts are refused. Increasing this value can help Fluent Bit handle bursts of incoming connections more gracefully. The default value is `128`.
76
+
When Fluent Bit listens for incoming connections (for example, in input plugins like HTTP, TCP, OpenTelemetry, Forward, and Syslog), the operating system maintains a queue of pending connections. The `net.backlog` option controls the maximum number of pending connections that can be queued before new connection attempts are refused. Increasing this value can help Fluent Bit handle bursts of incoming connections more gracefully. The default value is `128`.
77
77
78
-
> **Note:** On Linux, the effective backlog value may be capped by the kernel parameter `net.core.somaxconn`. If you need to allow a higher number of pending connections, you may need to increase this system setting.
78
+
{% hint style="info" %}
79
+
80
+
On Linux, the effective backlog value might be capped by the kernel parameter `net.core.somaxconn`. If you need to allow a greater number of pending connections, you can increase this system setting.
81
+
82
+
{% endhint %}
79
83
80
84
## Configuration options
81
85
@@ -95,7 +99,7 @@ that rely on networking I/O:
95
99
|`net.keepalive_max_recycle`| Set maximum number of times a keepalive connection can be used before it's retired. |`2000`|
96
100
|`net.max_worker_connections`| Set maximum number of TCP connections that can be established per worker. |`0` (unlimited) |
97
101
|`net.source_address`| Specify network address to bind for data traffic. |_none_|
98
-
|`net.backlog`| Set the maximum number of pending connections for listening sockets. This option is vailable on versions >= 4.0.4. |`128`|
102
+
|`net.backlog`| Set the maximum number of pending connections for listening sockets. This option requires Fluent Bit version 4.0.4 or later. |`128`|
99
103
100
104
## Example
101
105
@@ -183,4 +187,4 @@ If the `net.keepalive` option isn't enabled, Fluent Bit closes the TCP connectio
183
187
and netcat quits.
184
188
185
189
After the five records arrive, the connection idles. After 10 seconds, the connection
Copy file name to clipboardExpand all lines: pipeline/filters/lua.md
+20-12Lines changed: 20 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -99,15 +99,15 @@ function cb_print(tag, timestamp, record)
99
99
end
100
100
```
101
101
102
-
####Function arguments
102
+
### Function arguments
103
103
104
104
| Name | Description |
105
105
| ---- | ----------- |
106
106
|`tag`| Name of the tag associated with the incoming record. |
107
107
|`timestamp`| Unix timestamp with nanoseconds associated with the incoming record. The original format is a double (`seconds.nanoseconds`). |
108
108
|`record`| Lua table with the record content. |
109
109
110
-
####Return values
110
+
### Return values
111
111
112
112
Each callback must return three values:
113
113
@@ -117,13 +117,13 @@ Each callback must return three values:
117
117
|`timestamp`| double | If `code` equals `1`, the original record timestamp will be replaced with this new value. |
118
118
|`record`| table | If `code` equals `1`, the original record information will be replaced with this new value. The `record` value must be a valid Lua table. This value can be an array of tables (for example, an array of objects in JSON format), and in that case the input record is effectively split into multiple records. |
119
119
120
-
## Lua Extended callback with Groups and Metadata support
120
+
## Lua extended callback with groups and metadata support
121
121
122
122
{% hint style="info" %}
123
-
This feature is available in Fluent Bit version 4.0.4 and later.
123
+
This feature requires Fluent Bit version 4.0.4 or later.
124
124
{% endhint %}
125
125
126
-
For more advanced use cases, especially when working with structured formats like OpenTelemetry Logs, Fluent Bit supports an extended callback prototype that provides access to group metadata and record metadata.
126
+
For more advanced use cases, especially when working with structured formats like OpenTelemetry logs, Fluent Bit supports an extended callback prototype that provides access to group metadata and record metadata.
127
127
128
128
### Extended function signature
129
129
@@ -140,8 +140,8 @@ end
140
140
| ---- | ----------- |
141
141
|`tag`| Name of the tag associated with the incoming record. |
142
142
|`timestamp`| Unix timestamp with nanoseconds associated with the incoming record. |
143
-
|`group`| A read-only table containing group-level metadata (e.g., OpenTelemetry resource or scope info). This will be an empty table if the log is not part of a group. |
144
-
|`metadata`| A table representing the record-specific metadata. You may modify this if needed. |
143
+
|`group`| A read-only table containing group-level metadata (for example, OpenTelemetry resource or scope info). This will be an empty table if the log is not part of a group. |
144
+
|`metadata`| A table representing the record-specific metadata. You can modify this if needed. |
145
145
|`record`| Lua table with the record content. |
146
146
147
147
#### Extended return values
@@ -159,8 +159,8 @@ Each extended callback must return four values:
159
159
160
160
At load time, the Lua filter automatically detects which callback prototype to use based on the number of parameters:
161
161
162
-
-**3 arguments**: Uses the classic mode (`tag`, `timestamp`, `record`)
This ensures backward compatibility with existing Lua scripts.
166
166
@@ -182,7 +182,11 @@ function cb_metadata(tag, ts, group, metadata, record)
182
182
end
183
183
```
184
184
185
-
> **Note:** The metadata and record arrays must have the same length.
185
+
{% hint style="info" %}
186
+
187
+
The metadata and record arrays must have the same length.
188
+
189
+
{% endhint %}
186
190
187
191
### OpenTelemetry example
188
192
@@ -262,7 +266,11 @@ pipeline:
262
266
}
263
267
```
264
268
265
-
> **Important:** Group metadata is read-only and should not be modified. If you don't need group or metadata support, you can continue using the 3-argument prototype.
269
+
{% hint style="info" %}
270
+
271
+
Group metadata is read-only and should not be modified. If you don't need group or metadata support, you can continue using the three-argument prototype.
Copy file name to clipboardExpand all lines: pipeline/inputs/tail.md
+19-18Lines changed: 19 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -42,37 +42,38 @@ The plugin supports the following configuration parameters:
42
42
43
43
The Tail plugin uses buffers to efficiently read and process log files. Understanding how these buffers work helps optimize memory usage and performance.
44
44
45
-
### File buffers vs Fluent Bit chunks
45
+
### File buffers versus Fluent Bit chunks
46
46
47
-
When a file is opened for monitoring, the Tail plugin allocates a buffer in memory of `buffer_chunk_size` bytes (defaults to 32KB). This buffer is used to read data from the file. If a single record (line) is longer than `buffer_chunk_size`, the buffer will grow up to `buffer_max_size` to accommodate it.
47
+
When a file is opened for monitoring, the Tail plugin allocates a buffer in memory of `buffer_chunk_size` bytes (defaults to 32 KB). This buffer is used to read data from the file. If a single record (line) is longer than `buffer_chunk_size`, the buffer will grow up to `buffer_max_size` to accommodate it.
48
48
49
-
> **Note:** These buffers are per-file. If you're monitoring many files, each file gets its own buffer, which can significantly increase memory usage.
49
+
{% hint style="info" %}
50
+
51
+
These buffers are per-file. If you're monitoring many files, each file gets its own buffer, which can significantly increase memory usage.
52
+
53
+
{% endhint %}
50
54
51
55
### From buffers to chunks
52
56
53
-
Inside each file buffer, multiple lines/records might exist. The plugin processes these records and converts them to msgpack format (binary serialization). This msgpack data is then appended to what Fluent Bit calls a **Chunk** - a collection of serialized records that belong to the same tag.
57
+
Inside each file buffer, multiple lines or records might exist. The plugin processes these records and converts them to MessagePack format (binary serialization). This MessagePack data is then appended to what Fluent Bit calls a _chunk_, which is a collection of serialized records that belong to the same tag.
54
58
55
-
While Fluent Bit has a soft limit of 2MB for chunks, input plugins like Tail can generate msgpack buffers larger than 2MB, and the final chunk can exceed this soft limit.
59
+
Although Fluent Bit has a soft limit of 2 MB for chunks, input plugins like Tail can generate MessagePack buffers larger than 2 MB, and the final chunk can exceed this soft limit.
56
60
57
61
### Memory protection with `mem_buf_limit`
58
62
59
-
If Fluent Bit is not configured to use filesystem buffering, it needs mechanisms to protect against high memory consumption during backpressure scenarios (e.g., when destination endpoints are down or network issues occur). The `mem_buf_limit` option restricts how much memory in chunks an input plugin can use.
63
+
If Fluent Bit is not configured to use filesystem buffering, it needs mechanisms to protect against high memory consumption during backpressure scenarios (for example, when destination endpoints are down or network issues occur). The `mem_buf_limit` option restricts how much memory in chunks an input plugin can use.
60
64
61
65
When filesystem buffering is enabled, memory management works differently. For more details, see [Buffering and Storage](../../administration/buffering-and-storage.md).
62
66
63
67
## Database file
64
68
65
-
{% hint style="info" %}
66
-
**File positioning behavior:**
69
+
File positioning behavior varies based on the presence or absence of a database file.
67
70
68
-
-**With database file**: The plugin restores the last known position (offset) from the database. If no previous position exists and `read_from_head` is false, it starts monitoring from the end of the file.
71
+
If a database file is present, the plugin restores the last known position (offset) from the database. If no previous position exists and `read_from_head` is `false`, it starts monitoring from the end of the file.
69
72
70
-
-**Without database file**:
71
-
- If `read_from_head` is true: The plugin reads from the beginning of the file
72
-
- If `read_from_head` is false: The plugin starts monitoring from the end of the file (classic "tail" behavior)
73
+
If no database file is present, positioning behavior depends on the value of `read_from_head`:
73
74
74
-
This means that without a database and with `read_from_head`set to false, only new content written after Fluent Bit starts will be monitored.
75
-
{% endhint %}
75
+
- When `read_from_head`is `true`, the plugin reads from the beginning of the file.
76
+
- When `read_from_head` is `false`, the plugin starts monitoring from the end of the file (classic "tail" behavior). This means that only new content written after Fluent Bit starts will be monitored.
Fluent Bit 1.8 and later supports multiline core capabilities for the Tail input plugin. Fluent Bit supports the both the old and new configuration mechanisms. To avoid breaking changes, users are encouraged to use the latest one. The two mechanisms are:
130
131
131
-
- Multiline Core
132
-
- Old Multiline
132
+
- Multiline core
133
+
- Old multiline
133
134
134
-
### Multiline Core
135
+
### Multiline core
135
136
136
-
The new multiline core is exposed by the following configuration:
137
+
Multiline core is exposed by the following configuration:
0 commit comments