Skip to content

Commit 0644fb7

Browse files
Jorge Niedbalskialexakreizinger
andauthored
Update pipeline/processors/conditional-processing.md
Co-authored-by: Alexa Kreizinger <[email protected]> Signed-off-by: Jorge Niedbalski <[email protected]>
1 parent c524d9e commit 0644fb7

File tree

1 file changed

+70
-42
lines changed

1 file changed

+70
-42
lines changed

pipeline/processors/conditional-processing.md

Lines changed: 70 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -2,62 +2,90 @@
22

33
Conditional processing allows you to selectively apply processors to log records based on field values. This feature enables you to create processing pipelines that apply processors only to records that match specific criteria.
44

5-
## Configuration format
5+
## Configuration
66

7-
Conditional processing is available for processors in the YAML configuration format. To apply a processor conditionally, you add a `condition` block to the processor configuration:
7+
You can turn a standard processor into a conditional processor by adding a `condition` block to the
8+
processor's YAML configuration settings.
9+
10+
{% hint style="info" %}
11+
Conditional processing is only available for
12+
[YAML configuration files](../../administration/configuring-fluent-bit/yaml/README.md),
13+
not [classic configuration files](../../administration/configuring-fluent-bit/classic-mode/README.md).
14+
{% endhint %}
15+
16+
17+
These `condition` blocks use the following syntax:
818

919
```yaml
10-
- name: processor_name
11-
# Regular processor configuration...
12-
condition:
13-
op: and|or
14-
rules:
15-
- field: "$field_name"
16-
op: comparison_operator
17-
value: comparison_value
18-
# Additional rules...
20+
pipeline:
21+
inputs:
22+
<...>
23+
processors:
24+
logs:
25+
- name: {processor_name}
26+
<...>
27+
condition:
28+
op: {and|or}
29+
rules:
30+
- field: {field_name1}
31+
op: {comparison_operator}
32+
value: {comparison_value1}
33+
- field: {field_name2}
34+
op: {comparison_operator}
35+
value: {comparison_value2}
36+
<...>
1937
```
2038

21-
### Condition operators
39+
Each processor can only have a single `condition` block, but can have multiple rules within that condition.
40+
These rules are stored as items in the `condition.rules` array.
2241

23-
The `op` field in the condition block specifies the logical operator to apply across all rules:
42+
### Condition evaluation
2443

25-
| Operator | Description |
26-
| --- | --- |
27-
| `and` | All rules must evaluate to true for the condition to be true |
28-
| `or` | At least one rule must evaluate to true for the condition to be true |
44+
The `condition.op` parameter specifies the condition's evaluation logic. It has two possible values:
45+
46+
- `and`: All rules in the `condition.rules` array must evaluate to `true` for the condition to be met.
47+
- `or`: One or more rules in the `conditions.rules` array must evaluate to `true` for the condition to be met.
2948

3049
### Rules
3150

32-
Each rule consists of:
51+
Each item in the `condition.rules` array must include values for the following parameters:
3352

34-
- `field`: The field to evaluate (must use [record accessor syntax](/administration/configuring-fluent-bit/classic-mode/record-accessor.md) with `$` prefix)
35-
- `op`: The comparison operator
36-
- `value`: The value to compare against
53+
| Parameter | Description |
54+
| --- | --- |
55+
| `field` | The field within your logs to evaluate. The value of this parameter must use [the correct syntax](#field-access) to access the fields inside logs. |
56+
| `op` | The [comparison operator](#comparison-operators) to evaluate whether the rule is true. This parameter (`condition.rules.op`) is distinct from the `condition.op` parameter and has different possible values. |
57+
| `value` | The value of the specified log field to use in your comparison. Optionally, you can provide [an array that contains multiple values](#array-of-values). |
3758

38-
### Comparison operators
59+
Rules are evaluated against each log that passes through your data pipeline. For example, given a rule with these parameters:
3960

40-
The following comparison operators are supported:
61+
```
62+
- field: "$status"
63+
op: eq
64+
value: 200
65+
```
4166

42-
| Operator | Description |
43-
| --- | --- |
44-
| `eq` | Equal to |
45-
| `neq` | Not equal to |
46-
| `gt` | Greater than |
47-
| `lt` | Less than |
48-
| `gte` | Greater than or equal to |
49-
| `lte` | Less than or equal to |
50-
| `regex` | Matches regular expression |
51-
| `not_regex` | Does not match regular expression |
52-
| `in` | Value is in the specified array |
53-
| `not_in` | Value is not in the specified array |
54-
55-
### Field access
56-
57-
You can access record fields using [record accessor syntax](/administration/configuring-fluent-bit/classic-mode/record-accessor.md):
58-
59-
- Simple fields: `$field`
60-
- Nested fields: `$parent['child']['subchild']`
67+
This rule evaluates to `true` for a log that contains the string `'status':200`, but evaluates to `false` for a log that contains the string `'status':403`.
68+
69+
#### Field access
70+
71+
The `conditions.rules.field` parameter uses [record accessor syntax](/administration/configuring-fluent-bit/classic-mode/record-accessor.md) to reference fields inside logs.
72+
73+
You can use `$field` syntax to access a top-level field, and `$field['child']['subchild']` to access nested fields.
74+
75+
#### Comparison operators
76+
77+
The `conditions.rules.op` parameter has the following possible values:
78+
79+
- `eq`: equal to
80+
- `neq`: not equal to
81+
- `gt`: greater than
82+
- `lt`: less than
83+
- `gte`: greater than or equal to
84+
- `lte`: less than or equal to
85+
- `regex`: matches a regular expression
86+
- `not_regex`: does not match a regular expression
87+
- `in`: is included in the specified array
88+
- `not_in`: is not included in the specified array
6189

6290
## Examples
6391

0 commit comments

Comments
 (0)