Skip to content

Pipeline: filters: modify: style updates #1689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 29, 2025
Merged

Pipeline: filters: modify: style updates #1689

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cb79196
Pipeline: filters: modify: style updates
esmerel May 28, 2025
f0a2fb4
Apply suggestions from code review
esmerel May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 68 additions & 54 deletions pipeline/filters/modify.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Modify

The _Modify Filter_ plugin allows you to change records using rules and conditions.
The _Modify Filter_ plugin lets you change records using rules and conditions.

## Example usage

As an example using JSON notation to,
As an example, use JSON notation to:

* Rename `Key2` to `RenamedKey`
* Add a key `OtherKey` with value `Value3` if `OtherKey` does not yet exist
- Rename `Key2` to `RenamedKey`
- Add a key `OtherKey` with value `Value3` if `OtherKey` doesn't yet exist

_Example \(input\)_
Example (input):

```text
{
Expand All @@ -18,7 +18,7 @@ _Example \(input\)_
}
```

_Example \(output\)_
Example (output):

```text
{
Expand All @@ -28,56 +28,60 @@ _Example \(output\)_
}
```

## Configuration Parameters
## Configuration parameters

The Modify filter supports multiple parameters.

### Rules

The plugin supports the following rules:

| Operation | Parameter 1 | Parameter 2 | Description |
| :--- | :--- | :--- | :--- |
| Set | STRING:KEY | STRING:VALUE | Add a key/value pair with key `KEY` and value `VALUE`. If `KEY` already exists, _this field is overwritten_ |
| Add | STRING:KEY | STRING:VALUE | Add a key/value pair with key `KEY` and value `VALUE` if `KEY` does not exist |
| Remove | STRING:KEY | NONE | Remove a key/value pair with key `KEY` if it exists |
| Remove\_wildcard | WILDCARD:KEY | NONE | Remove all key/value pairs with key matching wildcard `KEY` |
| Remove\_regex | REGEXP:KEY | NONE | Remove all key/value pairs with key matching regexp `KEY` |
| Rename | STRING:KEY | STRING:RENAMED\_KEY | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists AND `RENAMED_KEY` _does not exist_ |
| Hard\_rename | STRING:KEY | STRING:RENAMED\_KEY | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists. If `RENAMED_KEY` already exists, _this field is overwritten_ |
| Copy | STRING:KEY | STRING:COPIED\_KEY | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists AND `COPIED_KEY` _does not exist_ |
| Hard\_copy | STRING:KEY | STRING:COPIED\_KEY | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists. If `COPIED_KEY` already exists, _this field is overwritten_ |
| Move\_to\_start | WILDCARD:KEY | NONE | Move key/value pairs with keys matching KEY to the start of the message |
| Move\_to\_end | WILDCARD:KEY | NONE | Move key/value pairs with keys matching KEY to the end of the message |

* Rules are case insensitive, parameters are not
* Any number of rules can be set in a filter instance.
* Rules are applied in the order they appear, with each rule operating on the result of the previous rule.
| `Set` | `STRING:KEY` | `STRING:VALUE` | Add a key/value pair with key `KEY` and value `VALUE`. If `KEY` already exists, this field is overwritten. |
| `Add` | `STRING:KEY` | `STRING:VALUE` | Add a key/value pair with key `KEY` and value `VALUE` if `KEY` doesn't exist. |
| `Remove` | `STRING:KEY` | _none_ | Remove a key/value pair with key `KEY` if it exists. |
| `Remove_wildcard` | `WILDCARD:KEY` | _none_ | Remove all key/value pairs with key matching wildcard `KEY`. |
| `Remove_regex` | `REGEXP:KEY` | _none_ | Remove all key/value pairs with key matching regexp `KEY`. |
| `Rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists and `RENAMED_KEY` doesn't exist. |
| `Hard_rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists. If `RENAMED_KEY` already exists, this field is overwritten. |
| `Copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists and `COPIED_KEY` doesn't exist. |
| `Hard_copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists. If `COPIED_KEY` already exists, this field is overwritten. |
| `Move_to_start` | `WILDCARD:KEY` | _none_ | Move key/value pairs with keys matching `KEY` to the start of the message. |
| `Move_to_end` | `WILDCARD:KEY` | _none_ | Move key/value pairs with keys matching `KEY` to the end of the message. |

- Rules are case insensitive, but parameters aren't.
- Any number of rules can be set in a filter instance.
- Rules are applied in the order they appear, with each rule operating on the result of the previous rule.

### Conditions

The plugin supports the following conditions:

| Condition | Parameter | Parameter 2 | Description |
| :--- | :--- | :--- | :--- |
| Key\_exists | STRING:KEY | NONE | Is `true` if `KEY` exists |
| Key\_does\_not\_exist | STRING:KEY | NONE | Is `true` if `KEY` does not exist |
| A\_key\_matches | REGEXP:KEY | NONE | Is `true` if a key matches regex `KEY` |
| No\_key\_matches | REGEXP:KEY | NONE | Is `true` if no key matches regex `KEY` |
| Key\_value\_equals | STRING:KEY | STRING:VALUE | Is `true` if `KEY` exists and its value is `VALUE` |
| Key\_value\_does\_not\_equal | STRING:KEY | STRING:VALUE | Is `true` if `KEY` exists and its value is not `VALUE` |
| Key\_value\_matches | STRING:KEY | REGEXP:VALUE | Is `true` if key `KEY` exists and its value matches `VALUE` |
| Key\_value\_does\_not\_match | STRING:KEY | REGEXP:VALUE | Is `true` if key `KEY` exists and its value does not match `VALUE` |
| Matching\_keys\_have\_matching\_values | REGEXP:KEY | REGEXP:VALUE | Is `true` if all keys matching `KEY` have values that match `VALUE` |
| Matching\_keys\_do\_not\_have\_matching\_values | REGEXP:KEY | REGEXP:VALUE | Is `true` if all keys matching `KEY` have values that do not match `VALUE` |

* Conditions are case insensitive, parameters are not
* Any number of conditions can be set.
* Conditions apply to the whole filter instance and all its rules. _Not_ to individual rules.
* All conditions have to be `true` for the rules to be applied.
* You can set [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) as `STRING:KEY` for nested key.

## Example \#1 - Add and Rename

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the following \(example\),
| `Key_exists` | `STRING:KEY` | _none_ | Is `true` if `KEY` exists. |
| `Key_does_not_exist` | `STRING:KEY` | _none_ | Is `true` if `KEY` doesn't exist. |
| `A_key_matches` | `REGEXP:KEY` | _none_ | Is `true` if a key matches regex `KEY`. |
| `No_key_matches` | `REGEXP:KEY` | _none_ | Is `true` if no key matches regex `KEY`. |
| `Key_value_equals` | `STRING:KEY` | `STRING:VALUE` | Is `true` if `KEY` exists and its value is `VALUE`. |
| `Key_value_does_not_equal` | `STRING:KEY` | `STRING:VALUE` | Is `true` if `KEY` exists and its value isn't `VALUE`. |
| `Key_value_matches` | `STRING:KEY` | `REGEXP:VALUE` | Is `true` if key `KEY` exists and its value matches `VALUE`. |
| `Key_value_does_not_match` | `STRING:KEY` | `REGEXP:VALUE` | Is `true` if key `KEY` exists and its value doesn't match `VALUE`. |
| `Matching_keys_have_matching_values` | `REGEXP:KEY` | `REGEXP:VALUE` | Is `true` if all keys matching `KEY` have values that match `VALUE`. |
| `Matching_keys_do_not_have_matching_values` | `REGEXP:KEY` | `REGEXP:VALUE` | Is `true` if all keys matching `KEY` have values that don't match `VALUE`. |

- Conditions are case insensitive, but parameters aren't.
- Any number of conditions can be set.
- Conditions apply to the whole filter instance and all its rules. _Not_ to individual rules.
- All conditions must be `true` for the rules to be applied.
- You can set [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) as `STRING:KEY` for nested key.

## Example 1 - add and rename

To start filtering records, you can run the filter from the command line or through
the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md),
which outputs data similar to the following:

```text
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
Expand All @@ -86,9 +90,9 @@ In order to start filtering records, you can run the filter from the command lin
[3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
```

### Using command Line
### Using command line

> Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.

```text
bin/fluent-bit -i mem \
Expand All @@ -105,10 +109,11 @@ bin/fluent-bit -i mem \
-o stdout
```

### Configuration File
### Configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[INPUT]
Name mem
Expand All @@ -129,9 +134,11 @@ bin/fluent-bit -i mem \
Rename Swap.total SWAPTOTAL
Add Mem.total TOTALMEM
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
Expand All @@ -153,12 +160,13 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}

### Result

The output of both the command line and configuration invocations should be identical and result in the following output.
The output of both the command line and configuration invocations should be identical and result in the following output:

```text
[2018/04/06 01:35:13] [ info] [engine] started
Expand All @@ -168,12 +176,13 @@ The output of both the command line and configuration invocations should be iden
[3] mem.local: [1522980613.000122671, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
```

## Example \#2 - Conditionally Add and Remove
## Example 2 - conditionally add and remove

### Configuration File
### Use a configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[INPUT]
Name mem
Expand Down Expand Up @@ -211,9 +220,11 @@ The output of both the command line and configuration invocations should be iden
Name stdout
Match *
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
Expand Down Expand Up @@ -242,10 +253,11 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}

### Result
### Add and remove result

```text
[2018/06/14 07:37:34] [ info] [engine] started (pid=1493)
Expand All @@ -255,12 +267,13 @@ pipeline:
[3] mem.local: [1528925858.000152319, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
```

## Example \#3 - Emoji
## Example 3 - emoji

### Configuration File
### Emoji configuration File

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[INPUT]
Name mem
Expand All @@ -283,9 +296,11 @@ pipeline:
Set ❄️ is_cold
Set 💦 is_wet
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
Expand All @@ -310,11 +325,11 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}


### Result
### Emoji example result

```text
[2018/06/14 07:46:11] [ info] [engine] started (pid=21875)
Expand All @@ -324,4 +339,3 @@ pipeline:
[3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
[0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
```