From 608cb10dee78f0716ff83aaccbccc85be8476dd4 Mon Sep 17 00:00:00 2001 From: Michael Voelker Date: Wed, 29 Jul 2020 21:40:50 +0200 Subject: [PATCH 1/3] filter_rewrite_tag: add and-combination for rules description for and-combination for rewrite_tag filter rules PR: https://github.com/fluent/fluent-bit/pull/2399 Signed-off-by: Michael Voelker --- pipeline/filters/rewrite-tag.md | 54 ++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/pipeline/filters/rewrite-tag.md b/pipeline/filters/rewrite-tag.md index e1e1a329d..13667b0ec 100644 --- a/pipeline/filters/rewrite-tag.md +++ b/pipeline/filters/rewrite-tag.md @@ -26,7 +26,7 @@ The `rewrite_tag` filter supports the following configuration parameters: | Key | Description | | :--- | :--- | -| Rule | Defines the matching criteria and the format of the Tag for the matching record. The Rule format have four components: `KEY REGEX NEW_TAG KEEP`. For more specific details of the Rule format and it composition read the next section. | +| Rule | Defines the matching criteria and the format of the Tag for the matching record. The Rule format have four mandatory components: `KEY REGEX NEW_TAG KEEP` and an optional component: `AND_COMBINE`. For more specific details of the Rule format and it composition read the next section. | | Emitter\_Name | When the filter emits a record under the new Tag, there is an internal emitter plugin that takes care of the job. Since this emitter expose metrics as any other component of the pipeline, you can use this property to configure an optional name for it. | | Emitter\_Storage.type | Define a buffering mechanism for the new records created. Note these records are part of the emitter plugin. This option support the values `memory` \(default\) or `filesystem`. If the destination for the new records generated might face backpressure due to latency or slow network, we strongly recommend enabling the `filesystem` mode. | | Emitter\_Mem\_Buf\_Limit | Set a limit on the amount of memory the tag rewrite emitter can consume if the outputs provide backpressure. The default for this limit is `10M`. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to `10M` then the pipeline will pause if the buffer exceeds `10M`. The pipeline will remain paused until the output drains the buffer below the `10M` limit. | @@ -151,6 +151,58 @@ Fluent Bit v1.x.x ... [0] from.test_tag.new.fluent.bit.out: [1580436933.000050569, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}] ``` +## Configuration Example with many Rules +In case of many rules, they are passed through in order until one rule matches. With`AND_COMBINE`value`true` as optional fifth +component, the rule is combined with the following rule like an 'and' combination. Only if first and following rule match, the message is retagged with the tag in the last matched rule. +An `AND_COMBINE` in the last rule is ignored. +``` +[SERVICE] + Flush 5 + Log_Level info + +[INPUT] + Name tail + Tag tail + Path /var/tmp/loginput.txt + +[FILTER] + Name rewrite_tag + Match tail + Rule $log ^(1)$ newtag_or false + Rule $log ^(.*and)$ newtag_and_1 false true + Rule $log ^(1.*)$ newtag_and_2 false + Rule $log ^(42)$ newtag_or false + Rule $log ^(9)$ newtag_and_3 false true + +[OUTPUT] + Name stdout + Match * +``` +inputfile /var/tmp/loginput.txt +``` +1 +2 +3 +9 +10and +10 +42 +``` + +the logmessages will be rewritten: +``` +fluent-bit_1 | [0] tail: [1596050753.241336500, {"log"=>"2"}] +fluent-bit_1 | [1] tail: [1596050753.241356700, {"log"=>"3"}] +fluent-bit_1 | [2] tail: [1596050753.241410100, {"log"=>"10"}] +fluent-bit_1 | [0] newtag_or: [1596050753.237370100, {"log"=>"1"}] +fluent-bit_1 | [1] newtag_or: [1596050753.241427200, {"log"=>"42"}] +fluent-bit_1 | [0] newtag_and_3: [1596050753.241374500, {"log"=>"9"}] +fluent-bit_1 | [0] newtag_and_2: [1596050753.241392800, {"log"=>"10and"}] +``` + + + + ## Monitoring From 70e87d78200fd7e6adae4826293ecabd11981ce6 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Tue, 21 Oct 2025 09:36:21 -0700 Subject: [PATCH 2/3] Apply suggestions from code review Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/filters/rewrite-tag.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/pipeline/filters/rewrite-tag.md b/pipeline/filters/rewrite-tag.md index 0312c23b4..9bb683ec7 100644 --- a/pipeline/filters/rewrite-tag.md +++ b/pipeline/filters/rewrite-tag.md @@ -176,11 +176,11 @@ $ fluent-bit -c example.conf ... [0] from.test_tag.new.fluent.bit.out: [1580436933.000050569, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}] ``` -## Configuration Example with many Rules -In case of many rules, they are passed through in order until one rule matches. With`AND_COMBINE`value`true` as optional fifth -component, the rule is combined with the following rule like an 'and' combination. Only if first and following rule match, the message is retagged with the tag in the last matched rule. +## Configuration example with multiple rules +In cases using multiple rules, the rules are passed through in order until one matches. With `AND_COMBINE` using the value `true` as optional fifth component, the rule is combined with the following rule like an 'and' combination. I f the first and following rule match, the message is retagged with the tag in the last matched rule. + An `AND_COMBINE` in the last rule is ignored. -``` +```text [SERVICE] Flush 5 Log_Level info @@ -204,7 +204,7 @@ An `AND_COMBINE` in the last rule is ignored. Match * ``` inputfile /var/tmp/loginput.txt -``` +```text 1 2 3 @@ -214,7 +214,7 @@ inputfile /var/tmp/loginput.txt 42 ``` -the logmessages will be rewritten: +The log messages will be rewritten: ``` fluent-bit_1 | [0] tail: [1596050753.241336500, {"log"=>"2"}] fluent-bit_1 | [1] tail: [1596050753.241356700, {"log"=>"3"}] @@ -225,10 +225,6 @@ fluent-bit_1 | [0] newtag_and_3: [1596050753.241374500, {"log"=>"9"}] fluent-bit_1 | [0] newtag_and_2: [1596050753.241392800, {"log"=>"10and"}] ``` - - - - ## Monitoring As described in the [Monitoring](../../administration/monitoring.md) section, every component of the pipeline of Fluent Bit exposes metrics. The basic metrics exposed by this filter are `drop_records` and `add_records`, which summarize the total of dropped records from the incoming data chunk or the new records added. From e40adf1f0d52547e7a45f930da8d32de688bbf28 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Tue, 21 Oct 2025 09:39:28 -0700 Subject: [PATCH 3/3] Apply suggestions from code review Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/filters/rewrite-tag.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline/filters/rewrite-tag.md b/pipeline/filters/rewrite-tag.md index 9bb683ec7..6f81a1cac 100644 --- a/pipeline/filters/rewrite-tag.md +++ b/pipeline/filters/rewrite-tag.md @@ -177,7 +177,7 @@ $ fluent-bit -c example.conf [0] from.test_tag.new.fluent.bit.out: [1580436933.000050569, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}] ``` ## Configuration example with multiple rules -In cases using multiple rules, the rules are passed through in order until one matches. With `AND_COMBINE` using the value `true` as optional fifth component, the rule is combined with the following rule like an 'and' combination. I f the first and following rule match, the message is retagged with the tag in the last matched rule. +In cases using multiple rules, the rules are passed through in order until one matches. With `AND_COMBINE` using the value `true` as optional fifth component, the rule is combined with the following rule like an 'and' combination. If the first and following rule match, the message is retagged with the tag in the last matched rule. An `AND_COMBINE` in the last rule is ignored. ```text @@ -203,7 +203,7 @@ An `AND_COMBINE` in the last rule is ignored. Name stdout Match * ``` -inputfile /var/tmp/loginput.txt +Use the input file `/var/tmp/loginput.txt`: ```text 1 2