Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 16 additions & 27 deletions pipeline/processors/conditional-processing.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,16 @@
# Conditional processing

Conditional processing lets you selectively apply [processors](README.md) to
logs based on the value of fields that those logs contain. This feature lets you
create processing pipelines that only process records that meet certain
criteria, and ignore the rest.
Conditional processing lets you selectively apply [processors](README.md) to logs based on the value of fields within those logs. This feature lets you create processing pipelines that only process records that meet certain criteria, and ignore the rest.

Conditional processing is available in Fluent Bit version 4.0 and greater.

## Configuration

You can turn a standard processor into a conditional processor by adding a
`condition` block to the processor's YAML configuration settings.
You can turn a standard processor into a conditional processor by adding a `condition` block to the processor's YAML configuration settings.

{% hint style="info" %}
- Conditional processing is only available for [YAML configuration files](../../administration/configuring-fluent-bit/yaml/README.md), not [classic configuration files](../../administration/configuring-fluent-bit/classic-mode/README.md).
- Conditional Processing feature is not supported when using [Filter as Processor](./filters.md).
- Only [YAML configuration files](../administration/configuring-fluent-bit/yaml/README.md) support processors, including conditional processors.
- Conditional processing isn't supported if you're using a [filter as a processor](../pipeline/processors/filters).
{% endhint %}

These `condition` blocks use the following syntax:
Expand Down Expand Up @@ -45,19 +41,14 @@ pipeline:
{% endtab %}
{% endtabs %}

Each processor can only have a single `condition` block, but that condition can
include multiple rules. These rules are stored as items in the `condition.rules`
array.
Each processor can only have a single `condition` block, but that condition can include multiple rules. These rules are stored as items in the `condition.rules` array.

### Condition evaluation

The `condition.op` parameter specifies the condition's evaluation logic. It has
two possible values:
The `condition.op` parameter specifies the condition's evaluation logic. It can have one of the following values:

- `and`: A log entry meets this condition when all the rules in the `condition.rules`
are [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy).
- `or`: A log entry meets this condition when one or more rules in the `condition.rules`
array are [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy).
- `and`: A log entry meets this condition when all the rules in the `condition.rules` array are [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy).
- `or`: A log entry meets this condition when one or more rules in the `condition.rules` array are [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy).

### Rules

Expand Down Expand Up @@ -104,8 +95,7 @@ The `conditions.rules.op` parameter has the following possible values:

### Basic condition

This example applies a condition that only processes logs that contain the
string `{"request": {"method": "POST"`:
This example applies a condition that only processes logs that contain the string `{"request": {"method": "POST"`:

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand All @@ -116,7 +106,7 @@ pipeline:
- name: dummy
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource"}}'
tag: request.log

processors:
logs:
- name: content_modifier
Expand Down Expand Up @@ -147,7 +137,7 @@ pipeline:
- name: dummy
dummy: '{"request": {"method": "POST", "path": "/api/v1/sensitive-data"}}'
tag: request.log

processors:
logs:
- name: content_modifier
Expand Down Expand Up @@ -181,7 +171,7 @@ pipeline:
- name: dummy
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource", "status_code": 200, "response_time": 150}}'
tag: request.log

processors:
logs:
- name: content_modifier
Expand Down Expand Up @@ -215,7 +205,7 @@ pipeline:
- name: dummy
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource"}}'
tag: request.log

processors:
logs:
- name: content_modifier
Expand Down Expand Up @@ -246,7 +236,7 @@ pipeline:
- name: dummy
dummy: '{"log": "Error: Connection refused", "level": "error", "service": "api-gateway"}'
tag: app.log

processors:
logs:
- name: content_modifier
Expand All @@ -262,7 +252,7 @@ pipeline:
- field: "$service"
op: in
value: ["api-gateway", "authentication", "database"]

- name: content_modifier
action: insert
key: paging_required
Expand All @@ -281,5 +271,4 @@ pipeline:
{% endtab %}
{% endtabs %}

This configuration adds an `alert` field to error logs from critical services,
and adds a `paging_required` field to errors that contain specific critical patterns.
This configuration adds an `alert` field to error logs from critical services, and adds a `paging_required` field to errors that contain specific critical patterns.
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Spelling-exceptions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,7 @@ Timeshift
tolerations
tooltip
tooltips
truthy
uber
unaggregated
unary
Expand Down