Skip to content

Commit a950e72

Browse files
esmerelesmerel
authored
Merge pull request #1697 from fluent/lynettemiles/sc-136159/update-fluent-bit-docs-pipeline-filters-rewrite
2 parents 402d9b6 + 68d3a3b commit a950e72

File tree

1 file changed

+58
-59
lines changed

1 file changed

+58
-59
lines changed

‎pipeline/filters/rewrite-tag.md

Lines changed: 58 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -4,44 +4,43 @@ description: Powerful and flexible routing
44

55
# Rewrite Tag
66

7-
Tags are what makes [routing](../../concepts/data-pipeline/router.md) possible. Tags are set in the configuration of the Input definitions where the records are generated, but there are certain scenarios where might be useful to modify the Tag in the pipeline so we can perform more advanced and flexible routing.
7+
Tags make [routing](../../concepts/data-pipeline/router.md) possible. Tags are set in the configuration of the `INPUT` definitions where the records are generated. There are scenarios where you might want to modify the tag in the pipeline to perform more advanced and flexible routing.
88

9-
The `rewrite_tag` filter, allows to re-emit a record under a new Tag. Once a record has been re-emitted, the original record can be preserved or discarded.
9+
The _Rewrite Tag_ filter lets you re-emit a record under a new tag. Once a record has been re-emitted, the original record can be preserved or discarded.
1010

11-
## How it Works
11+
The Rewrite Tag filter defines rules that match specific record key content against a regular expression. If a match exists, a new record with the defined tag will be emitted, entering from the beginning of the pipeline. Multiple rules can be specified and are processed in order until one of them matches.
1212

13-
The way it works is defining rules that matches specific record key content against a regular expression, if a match exists, **a new record with the defined Tag will be emitted, entering from the beginning of the pipeline.** Multiple rules can be specified and they are processed in order until one of them matches.
13+
The new tag definition can be composed of:
1414

15-
The new Tag to define can be composed by:
15+
- Alphanumeric characters
16+
- Numbers
17+
- The original tag string or part of it
18+
- Regular expressions groups capture
19+
- Any key or sub-key of the processed record
20+
- Environment variables
1621

17-
* Alphabet characters & Numbers
18-
* Original Tag string or part of it
19-
* Regular Expressions groups capture
20-
* Any key or sub-key of the processed record
21-
* Environment variables
22-
23-
## Configuration Parameters
22+
## Configuration parameters
2423

2524
The `rewrite_tag` filter supports the following configuration parameters:
2625

2726
| Key | Description |
2827
| :--- | :--- |
29-
| Rule | Defines the matching criteria and the format of the Tag for the matching record. The Rule format have four components: `KEY REGEX NEW_TAG KEEP`. For more specific details of the Rule format and it composition read the next section. |
30-
| Emitter\_Name | When the filter emits a record under the new Tag, there is an internal emitter plugin that takes care of the job. Since this emitter expose metrics as any other component of the pipeline, you can use this property to configure an optional name for it. |
31-
| Emitter\_Storage.type | Define a buffering mechanism for the new records created. Note these records are part of the emitter plugin. This option support the values `memory` \(default\) or `filesystem`. If the destination for the new records generated might face backpressure due to latency or slow network, we strongly recommend enabling the `filesystem` mode. |
32-
| Emitter\_Mem\_Buf\_Limit | Set a limit on the amount of memory the tag rewrite emitter can consume if the outputs provide backpressure. The default for this limit is `10M`. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to `10M` then the pipeline will pause if the buffer exceeds `10M`. The pipeline will remain paused until the output drains the buffer below the `10M` limit. |
28+
| `Rule` | Defines the matching criteria and the format of the tag for the matching record. The Rule format has four components: `KEY REGEX NEW_TAG KEEP`. |
29+
| `Emitter_Name` | Use this property to configure an optional name for the internal emitter plugin that handles filters emitting a record under the new tag. This emitter exposes metrics like any other component of the pipeline. |
30+
| `Emitter_Storage.type` | Define a buffering mechanism for the new records created. These records are part of the emitter plugin. Supported values are `memory` (default) and `filesystem`. If the destination for the new records generated might face backpressure due to latency or slow network, Fluent Bit strongly recommends enabling the `filesystem` mode. |
31+
| `Emitter_Mem_Buf_Limit` | Set a limit on the amount of memory the tag rewrite emitter can consume if the outputs provide backpressure. The default value is `10M`. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to `10M` then the pipeline will pause if the buffer exceeds `10M`. The pipeline will remain paused until the output drains the buffer below the `10M` limit. |
3332

3433
## Rules
3534

36-
A rule aims to define matching criteria and specify how to create a new Tag for a record. You can define one or multiple rules in the same configuration section. The rules have the following format:
35+
A rule defines matching criteria and specifies how to create a new tag for a record. You can define one or multiple rules in the same configuration section. The rules have the following format:
3736

3837
```text
3938
$KEY REGEX NEW_TAG KEEP
4039
```
4140

4241
### Key
4342

44-
The key represents the name of the _record key_ that holds the _value_ that we want to use to match our regular expression. A key name is specified and prefixed with a `$`. Consider the following structured record \(formatted for readability\):
43+
The key represents the name of the _record key_ that holds the `value` to use to match the regular expression. A key name is specified and prefixed with a `$`. Consider the following structured record (formatted for readability):
4544

4645
```javascript
4746
{
@@ -54,71 +53,72 @@ The key represents the name of the _record key_ that holds the _value_ that we w
5453
}
5554
```
5655

57-
If we wanted to match against the value of the key `name` we must use `$name`. The key selector is flexible enough to allow to match nested levels of sub-maps from the structure. If we wanted to check the value of the nested key `s2` we can do it specifying `$ss['s1']['s2']`, for short:
56+
To match against the value of the key `name`, you must use `$name`. The key selector is flexible enough to allow to match nested levels of sub-maps from the structure. To capture the value of the nested key `s2`, specify `$ss['s1']['s2']`, for short:
5857

59-
* `$name` = "abc-123"
60-
* `$ss['s1']['s2']` = "flb"
58+
-`$name` = "abc-123"
59+
-`$ss['s1']['s2']` = "flb"
6160

62-
Note that a key must point to a value that contains a string, it's **not valid** for numbers, booleans, maps or arrays.
61+
A key must point to a value that contains a string. It's not valid for numbers, Booleans, maps, or arrays.
6362

64-
### Regex
63+
### Regular expressions
6564

66-
Using a simple regular expression we can specify a matching pattern to use against the value of the key specified above, also we can take advantage of group capturing to create custom placeholder values.
65+
Use a regular expression to specify a matching pattern to use against the value of the key specified previously. You can take advantage of group capturing to create custom placeholder values.
6766

68-
If we wanted to match any record that it `$name` contains a value of the format `string-number` like the example provided above, we might use:
67+
To match any record that it `$name` contains a value of the format `string-number` like the previous example, you might use:
6968

7069
```text
7170
^([a-z]+)-([0-9]+)$
7271
```
7372

74-
Note that in our example we are using parentheses, this teams that we are specifying groups of data. If the pattern matches the value a placeholder will be created that can be consumed by the NEW\_TAG section.
73+
This example uses parentheses to specify groups of data. If the pattern matches the value a placeholder will be created that can be consumed by the `NEW_TAG` section.
7574

76-
If `$name` equals `abc-123` , then the following **placeholders** will be created:
75+
If `$name` equals `abc-123` , then the following placeholders will be created:
7776

78-
* `$0` = "abc-123"
79-
* `$1` = "abc"
80-
* `$2` = "123"
77+
-`$0` = "abc-123"
78+
-`$1` = "abc"
79+
-`$2` = "123"
8180

82-
If the Regular expression do not matches an incoming record, the rule will be skipped and the next rule \(if any\) will be processed.
81+
If the regular expression doesn't match an incoming record, the rule will be skipped and the next rule (if present) will be processed.
8382

84-
### New Tag
83+
### `New_Tag`
8584

86-
If a regular expression has matched the value of the defined key in the rule, we are ready to compose a new Tag for that specific record. The tag is a concatenated string that can contain any of the following characters: `a-z`,`A-Z`, `0-9` and `.-,`.
85+
If a regular expression has matched the value of the defined key in the rule, you can compose a new tag for that specific record. The tag is a concatenated string that can contain any of the following characters: `a-z`,`A-Z`, `0-9` and `.-,`.
8786

88-
A Tag can take any string value from the matching record, the original tag it self, environment variable or general placeholder.
87+
A tag can take any string value from the matching record, the original tag it self, environment variables, or general placeholders.
8988

9089
Consider the following incoming data on the rule:
9190

92-
* Tag = aa.bb.cc
93-
* Record = `{"name": "abc-123", "ss": {"s1": {"s2": "flb"}}}`
94-
* Environment variable $HOSTNAME = fluent
91+
- Tag = aa.bb.cc
92+
- Record = `{"name": "abc-123", "ss": {"s1": {"s2": "flb"}}}`
93+
- Environment variable $HOSTNAME = fluent
9594

96-
With such information we could create a very custom Tag for our record like the following:
95+
With this information you can create a custom tag for your record like the following:
9796

9897
```text
9998
newtag.$TAG.$TAG[1].$1.$ss['s1']['s2'].out.${HOSTNAME}
10099
```
101100

102-
the expected Tag to generated will be:
101+
the expected generated tag will be:
103102

104103
```text
105104
newtag.aa.bb.cc.bb.abc.flb.out.fluent
106105
```
107106

108-
We make use of placeholders, record content and environment variables.
107+
This make use of placeholders, record content, and environment variables.
109108

110109
### Keep
111110

112-
If a rule matches a rule the filter will emit a copy of the record with the new defined Tag. The property keep takes a boolean value to define if the original record with the old Tag must be preserved and continue in the pipeline or just be discarded.
111+
If a rule matches, the filter will emit a copy of the record with the new defined tag. The `keep` property takes a Boolean value to determine if the original record with the old tag must be preserved and continue in the pipeline or be discarded.
113112

114-
You can use `true` or `false` to decide the expected behavior. There is no default value and this is a mandatory field in the rule.
113+
You can use `true` or `false` to decide the expected behavior. This field is mandatory and has no default value.
115114

116-
## Configuration Example
115+
## Configuration example
117116

118-
The following configuration example will emit a dummy \(hand-crafted\) record, the filter will rewrite the tag, discard the old record and print the new record to the standard output interface:
117+
The following configuration example will emit a dummy record. The filter will rewrite the tag, discard the old record, and print the new record to the standard output interface:
119118

120119
{% tabs %}
121120
{% tab title="fluent-bit.conf" %}
121+
122122
```python
123123
[SERVICE]
124124
Flush 1
@@ -139,9 +139,11 @@ The following configuration example will emit a dummy \(hand-crafted\) record, t
139139
Name stdout
140140
Match from.*
141141
```
142+
142143
{% endtab %}
143144

144145
{% tab title="fluent-bit.yaml" %}
146+
145147
```yaml
146148
service:
147149
flush: 1
@@ -160,10 +162,10 @@ pipeline:
160162
- name: stdout
161163
match: from.*
162164
```
165+
163166
{% endtab %}
164167
{% endtabs %}
165168
166-
167169
The original tag `test_tag` will be rewritten as `from.test_tag.new.fluent.bit.out`:
168170

169171
```bash
@@ -180,21 +182,19 @@ Fluent Bit v1.x.x
180182

181183
## Monitoring
182184

183-
As described in the [Monitoring](../../administration/monitoring.md) section, every component of the pipeline of Fluent Bit exposes metrics. The basic metrics exposed by this filter are `drop_records` and `add_records`, they summarize the total of dropped records from the incoming data chunk or the new records added.
185+
As described in the [Monitoring](../../administration/monitoring.md) section, every component of the pipeline of Fluent Bit exposes metrics. The basic metrics exposed by this filter are `drop_records` and `add_records`, which summarize the total of dropped records from the incoming data chunk or the new records added.
184186

185-
Since `rewrite_tag` emit new records that goes through the beginning of the pipeline, it exposes an additional metric called `emit_records` that summarize the total number of emitted records.
187+
The `rewrite_tag` filter emits new records that go through the beginning of the pipeline, and exposes an additional metric called `emit_records` that summarize the total number of emitted records.
186188

187-
### Understanding the Metrics
189+
### Understand the metrics
188190

189-
Using the configuration provided above, if we query the metrics exposed in the HTTP interface we will see the following:
190-
191-
Command:
191+
Using the previously provided configuration, when you query the metrics exposed in the HTTP interface:
192192

193193
```bash
194-
$ curl http://127.0.0.1:2020/api/v1/metrics/ | jq
194+
curl http://127.0.0.1:2020/api/v1/metrics/ | jq
195195
```
196196

197-
Metrics output:
197+
You will see metrics output similar to the following:
198198

199199
```javascript
200200
{
@@ -227,13 +227,12 @@ Metrics output:
227227
}
228228
```
229229

230-
The _dummy_ input generated two records, the filter dropped two from the chunks and emitted two new ones under a different Tag.
231-
232-
The records generated are handled by the internal Emitter, so the new records are summarized in the Emitter metrics, take a look at the entry called `emitter_for_rewrite_tag.0`.
230+
The _dummy_ input generated two records, while the filter dropped two from the chunks and emitted two new ones under a different tag.
233231

234-
### What is the Emitter ?
232+
The records generated are handled by the internal emitter, so the new records are summarized in the Emitter metrics. Take a look at the entry called `emitter_for_rewrite_tag.0`.
235233

236-
The Emitter is an internal Fluent Bit plugin that allows other components of the pipeline to emit custom records. On this case `rewrite_tag` creates an Emitter instance to use it exclusively to emit records, on that way we can have a granular control of _who_ is emitting what.
234+
### The Emitter
237235

238-
The Emitter name in the metrics can be changed setting up the `Emitter_Name` configuration property described above.
236+
The _Emitter_ is an internal Fluent Bit plugin that allows other components of the pipeline to emit custom records. On this case `rewrite_tag` creates an emitter instance to use it exclusively to emit records, allowing for granular control of who is emitting what.
239237

238+
Change the Emitter name in the metrics by adding the `Emitter_Name` configuration property described previously.

0 commit comments

Comments
 (0)