-
Notifications
You must be signed in to change notification settings - Fork 541
pipeline: filters: lua: general cleanup #1706
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
alexakreizinger
merged 2 commits into
master
from
alexakreizinger/sc-136151/update-fluent-bit-docs-pipeline-filters-lua
Jun 2, 2025
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,44 +1,44 @@ | ||||||
| # Lua | ||||||
|
|
||||||
| The **Lua** filter allows you to modify the incoming records (even split one record into multiple records) using custom [Lua](https://www.lua.org/) scripts. | ||||||
| The _Lua_ filter lets you modify incoming records (or split one record into multiple records) using custom [Lua](https://www.lua.org/) scripts. | ||||||
|
|
||||||
| <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=f519378e-536c-4b25-8949-ee6ed8d8d6c1" /> | ||||||
|
|
||||||
| Due to the necessity to have a flexible filtering mechanism, it is now possible to extend Fluent Bit capabilities by writing custom filters using Lua programming language. A Lua-based filter takes two steps: | ||||||
| A Lua-based filter requires two steps: | ||||||
|
|
||||||
| 1. Configure the Filter in the main configuration | ||||||
| 2. Prepare a Lua script that will be used by the Filter | ||||||
| 1. Configure the filter in the main configuration | ||||||
| 2. Prepare a Lua script for the filter to use | ||||||
|
|
||||||
| ## Configuration Parameters <a id="config"></a> | ||||||
| ## Configuration parameters | ||||||
|
|
||||||
| The plugin supports the following configuration parameters: | ||||||
|
|
||||||
| | Key | Description | | ||||||
| | :--- | :--- | | ||||||
| | script | Path to the Lua script that will be used. This can be a relative path against the main configuration file. | | ||||||
| | call | Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the **script** parameter defined above. | | ||||||
| | type\_int\_key | If these keys are matched, the fields are converted to integer. If more than one key, delimit by space. Note that starting from Fluent Bit v1.6 integer data types are preserved and not converted to double as in previous versions. | | ||||||
| | type\_array\_key| If these keys are matched, the fields are handled as array. If more than one key, delimit by space. It is useful the array can be empty. | | ||||||
| | protected\_mode | If enabled, Lua script will be executed in protected mode. It prevents Fluent Bit from crashing when invalid Lua script is executed or the triggered Lua function throws exceptions. Default is true. | | ||||||
| | time\_as\_table | By default when the Lua script is invoked, the record timestamp is passed as a *floating number* which might lead to precision loss when it is converted back. If you desire timestamp precision, enabling this option will pass the timestamp as a Lua table with keys `sec` for seconds since epoch and `nsec` for nanoseconds. | | ||||||
| | code | Inline LUA code instead of loading from a path via `script`. | | ||||||
| | enable_flb_null| If enabled, null will be converted to flb_null in Lua. It is useful to prevent removing key/value since nil is a special value to remove key value from map in Lua. Default is false. | | ||||||
| | --- | ----------- | | ||||||
| | `script` | Path to the Lua script that will be used. This can be a relative path against the main configuration file. | | ||||||
| | `call` | The Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the `script` parameter. | | ||||||
| | `type_int_key` | If these keys are matched, the fields are converted to integers. If more than one key, delimit by space. | | ||||||
| | `type_array_key` | If these keys are matched, the fields are handled as array. If more than one key, delimit by space. The array can be empty. | | ||||||
| | `protected_mode` | If enabled, the Lua script will be executed in protected mode. It prevents Fluent Bit from crashing when an invalid Lua script is executed or the triggered Lua function throws exceptions. Default value: `true`. | | ||||||
| | `time_as_table` | By default, when the Lua script is invoked, the record timestamp is passed as a floating number, which might lead to precision loss when it is converted back. If you need timestamp precision, enabling this option will pass the timestamp as a Lua table with keys `sec` for seconds since epoch and `nsec` for nanoseconds. | | ||||||
| | `code` | Inline Lua code instead of loading from a path defined in `script`. | | ||||||
| | `enable_flb_null` | If enabled, `null` will be converted to `flb_null` in Lua. This helps prevent removing key/value since `nil` is a special value to remove key/value from map in Lua. Default value: `false`. | | ||||||
|
|
||||||
| ## Getting Started <a id="getting_started"></a> | ||||||
| ## Get started | ||||||
|
|
||||||
| In order to test the filter, you can run the plugin from the command line or through the configuration file. The following examples use the [dummy](../inputs/dummy.md) input plugin for data ingestion, invoke Lua filter using the [test.lua](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua) script and call the [cb\_print\(\)](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua#L29) function which only prints the same information to the standard output: | ||||||
| To test the Lua filter, you can run the plugin from the command line or through the configuration file. The following examples use the [dummy](../inputs/dummy.md) input plugin for data ingestion, invoke Lua filter using the [test.lua](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua) script, and call the [`cb_print()`](https://github.com/fluent/fluent-bit/blob/master/scripts/test.lua#L29) function, which only prints the same information to the standard output. | ||||||
|
|
||||||
| ### Command Line | ||||||
| ### Command line | ||||||
|
|
||||||
| From the command line you can use the following options: | ||||||
|
|
||||||
| ```bash | ||||||
| $ fluent-bit -i dummy -F lua -p script=test.lua -p call=cb_print -m '*' -o null | ||||||
alexakreizinger marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
| ``` | ||||||
|
|
||||||
| ### Configuration File | ||||||
| ### Configuration file | ||||||
|
|
||||||
| In your main configuration file append the following _Input_, _Filter_ & _Output_ sections: | ||||||
| In your main configuration file, append the following `Input`, `Filter` and `Output` sections: | ||||||
alexakreizinger marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| {% tabs %} | ||||||
| {% tab title="fluent-bit.conf" %} | ||||||
|
|
@@ -76,16 +76,16 @@ pipeline: | |||||
| {% endtabs %} | ||||||
|
|
||||||
|
|
||||||
| ## Lua Script Filter API <a id="lua_script"></a> | ||||||
| ## Lua script filter API | ||||||
|
|
||||||
| The life cycle of a filter have the following steps: | ||||||
| The life cycle of a filter has the following steps: | ||||||
|
|
||||||
| 1. Upon Tag matching by this filter, it may process or bypass the record. | ||||||
| 2. If tag matched, it will accept the record and invoke the function defined in the `call` property which basically is the name of a function defined in the Lua `script`. | ||||||
| 3. Invoke Lua function and pass each record in JSON format. | ||||||
| 4. Upon return, validate return value and continue the pipeline. | ||||||
| 1. Upon tag matching by this filter, it might process or bypass the record. | ||||||
| 2. If the tag matched, it will accept the record and invoke the function defined in the `call` property, which is the name of a function defined in the Lua `script`. | ||||||
| 3. It invokes the Lua function and passes each record in JSON format. | ||||||
| 4. Upon return, it validates the return value and continues the pipeline. | ||||||
alexakreizinger marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| ## Callback Prototype | ||||||
| ## Callback prototype | ||||||
|
|
||||||
| The Lua script can have one or multiple callbacks that can be used by this filter. The function prototype is as follows: | ||||||
|
|
||||||
|
|
@@ -96,23 +96,23 @@ function cb_print(tag, timestamp, record) | |||||
| end | ||||||
| ``` | ||||||
|
|
||||||
| #### Function Arguments | ||||||
| #### Function arguments | ||||||
|
|
||||||
| | name | description | | ||||||
| | :--- | :--- | | ||||||
| | tag | Name of the tag associated with the incoming record. | | ||||||
| | timestamp | Unix timestamp with nanoseconds associated with the incoming record. The original format is a double (seconds.nanoseconds) | | ||||||
| | record | Lua table with the record content | | ||||||
| | Name | Description | | ||||||
| | ---- | ----------- | | ||||||
| | `tag` | Name of the tag associated with the incoming record. | | ||||||
| | `timestamp` | Unix timestamp with nanoseconds associated with the incoming record. The original format is a double (`seconds.nanoseconds`). | | ||||||
| | `record` | Lua table with the record content. | | ||||||
|
|
||||||
| #### Return Values | ||||||
| #### Return values | ||||||
|
|
||||||
| Each callback **must** return three values: | ||||||
| Each callback must return three values: | ||||||
|
|
||||||
| | name | data type | description | | ||||||
| | :--- | :--- | :--- | | ||||||
| | code | integer | The code return value represents the result and further action that may follows. If _code_ equals -1, means that the record will be dropped. If _code_ equals 0, the record will not be modified, otherwise if _code_ equals 1, means the original timestamp and record have been modified so it must be replaced by the returned values from _timestamp_ (second return value) and _record_ (third return value). If _code_ equals 2, means the original timestamp is not modified and the record has been modified so it must be replaced by the returned values from _record_ (third return value). The _code_ 2 is supported from v1.4.3. | | ||||||
| | timestamp | double | If code equals 1, the original record timestamp will be replaced with this new value. | | ||||||
| | record | table | If code equals 1, the original record information will be replaced with this new value. Note that the _record_ value **must** be a valid Lua table. This value can be an array of tables (i.e., array of objects in JSON format), and in that case the input record is effectively split into multiple records. (see below for more details) | | ||||||
| | Name | Data type | Description | | ||||||
| | ---- | --------- | ----------- | | ||||||
| | `code` | integer | The code return value represents the result and further actions that might follow. If `code` equals `-1`, this means that the record will be dropped. If `code` equals `0`, the record won't be modified. Otherwise, if `code` equals `1`, this means the original timestamp and record have been modified, so it must be replaced by the returned values from `timestamp` (second return value) and `record` (third return value). If `code` equals `2`, this means the original timestamp won't be modified and the record has been modified, so it must be replaced by the returned values from `record` (third return value). | | ||||||
| | `timestamp` | double | If `code` equals `1`, the original record timestamp will be replaced with this new value. | | ||||||
| | `record` | table | If `code` equals `1`, the original record information will be replaced with this new value. The `record` value must be a valid Lua table. This value can be an array of tables (for example, an array of objects in JSON format), and in that case the input record is effectively split into multiple records. | | ||||||
|
|
||||||
| ## Features | ||||||
|
|
||||||
|
|
@@ -176,36 +176,29 @@ pipeline: | |||||
| {% endtab %} | ||||||
| {% endtabs %} | ||||||
|
|
||||||
| ### Number Type | ||||||
| ### Number type | ||||||
|
|
||||||
| Lua treats numbers as a `double` type, which means an `integer` type | ||||||
| containing data like user IDs and log levels will be converted to a `double`. | ||||||
| To avoid type conversion, use the `type_int_key` property. | ||||||
| Lua treats numbers as a `double` type, which means an `integer` type containing data like user IDs and log levels will be converted to a `double`. To avoid type conversion, use the `type_int_key` property. | ||||||
|
|
||||||
| ### Protected Mode | ||||||
| ### Protected mode | ||||||
|
|
||||||
| Fluent Bit supports protected mode to prevent crashes if it executes an invalid Lua script. | ||||||
| See [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html) in | ||||||
| the Lua documentation for more information. | ||||||
| Fluent Bit supports protected mode to prevent crashes if it executes an invalid Lua script. See [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html) in the Lua documentation for more information. | ||||||
|
|
||||||
| ## Code examples | ||||||
|
|
||||||
| ## Code Examples | ||||||
|
|
||||||
| For functional examples of this interface, please refer to the code samples provided in the source code of the project located here: | ||||||
|
|
||||||
| For functional examples of this interface, refer to the code samples provided in the source code of the project located here: | ||||||
|
||||||
| For functional examples of this interface, refer to the code samples provided in the source code of the project located here: | |
| For functional examples of this interface, refer to the code samples provided in the source code of the project located here: |
missing location link?
alexakreizinger marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
alexakreizinger marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.