Skip to content

Commit 9c1faf0

Browse files
jabbrwckyjabbrwcky
authored
Update Logstash_Key_prefix documentation (#1144)
* Update Logstash_Key_prefix documentation Signed-off-by: Jens Hausherr <[email protected]> * Add link to record_accessor Signed-off-by: Jens Hausherr <[email protected]> --------- Signed-off-by: Jens Hausherr <[email protected]>
1 parent 9695a27 commit 9c1faf0

File tree

2 files changed

+37
-2
lines changed

2 files changed

+37
-2
lines changed

‎pipeline/outputs/elasticsearch.md

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ The **es** output plugin, allows to ingest your records into an [Elasticsearch](
3131
| Type | Type name | \_doc |
3232
| Logstash\_Format | Enable Logstash format compatibility. This option takes a boolean value: True/False, On/Off | Off |
3333
| Logstash\_Prefix | When Logstash\_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash\_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated. | logstash |
34+
| Logstash\_Prefix\_Key | When included: the value of the key in the record will be evaluated as key reference and overrides Logstash\_Prefix for index generation. If the key/value is not found in the record then the Logstash\_Prefix option will act as a fallback. The parameter is expected to be a [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor). | |
3435
| Logstash\_Prefix\_Separator | Set a separator between logstash_prefix and date.| - |
3536
| Logstash\_DateFormat | Time format \(based on [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html)\) to generate the second part of the Index name. | %Y.%m.%d |
3637
| Time\_Key | When Logstash\_Format is enabled, each record will get a new timestamp field. The Time\_Key property defines the name of that field. | @timestamp |
@@ -45,7 +46,7 @@ The **es** output plugin, allows to ingest your records into an [Elasticsearch](
4546
| Trace\_Output | Print all elasticsearch API request payloads to stdout \(for diag only\) | Off |
4647
| Trace\_Error | If elasticsearch return an error, print the elasticsearch API request and response \(for diag only\) | Off |
4748
| Current\_Time\_Index | Use current time for index generation instead of message record | Off |
48-
| Logstash\_Prefix\_Key | When included: the value in the record that belongs to the key will be looked up and over-write the Logstash\_Prefix for index generation. If the key/value is not found in the record then the Logstash\_Prefix option will act as a fallback. Nested keys are not supported \(if desired, you can use the nest filter plugin to remove nesting\) | |
49+
4950
| Suppress\_Type\_Name | When enabled, mapping types is removed and `Type` option is ignored. Types are deprecated in APIs in [v7.0](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html). This options is for v7.0 or later. | Off |
5051
| Workers | Enables dedicated thread(s) for this output. Default value is set since version 1.8.13. For previous versions is 0. | 2 |
5152

@@ -237,3 +238,20 @@ Without this you will see errors like:
237238
```text
238239
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
239240
```
241+
242+
### Logstash_Prefix_Key
243+
244+
The following snippet demonstrates using the namespace name as extracted by the
245+
`kubernetes` filter as logstash prefix:
246+
247+
```text
248+
[OUTPUT]
249+
Name es
250+
Match *
251+
# ...
252+
Logstash_Prefix logstash
253+
Logstash_Prefix_Key $kubernetes["namespace_name"]
254+
# ...
255+
```
256+
257+
For records that do nor have the field `kubernetes.namespace_name`, the default prefix, `logstash` will be used.

‎pipeline/outputs/opensearch.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ The following instructions assumes that you have a fully operational OpenSearch
2828
| Type | Type name. This option is ignored if `Suppress_Type_Name` is enabled. | \_doc |
2929
| Logstash\_Format | Enable Logstash format compatibility. This option takes a boolean value: True/False, On/Off | Off |
3030
| Logstash\_Prefix | When Logstash\_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash\_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated. | logstash |
31+
| Logstash\_Prefix\_Key | When included: the value of the key in the record will be evaluated as key reference and overrides Logstash\_Prefix for index generation. If the key/value is not found in the record then the Logstash\_Prefix option will act as a fallback. The parameter is expected to be a [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor). | |
3132
| Logstash\_Prefix\_Separator | Set a separator between logstash_prefix and date. | - |
3233
| Logstash\_DateFormat | Time format \(based on [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html)\) to generate the second part of the Index name. | %Y.%m.%d |
3334
| Time\_Key | When Logstash\_Format is enabled, each record will get a new timestamp field. The Time\_Key property defines the name of that field. | @timestamp |
@@ -42,7 +43,6 @@ The following instructions assumes that you have a fully operational OpenSearch
4243
| Trace\_Output | When enabled print the OpenSearch API calls to stdout \(for diag only\) | Off |
4344
| Trace\_Error | When enabled print the OpenSearch API calls to stdout when OpenSearch returns an error \(for diag only\) | Off |
4445
| Current\_Time\_Index | Use current time for index generation instead of message record | Off |
45-
| Logstash\_Prefix\_Key | When included: the value in the record that belongs to the key will be looked up and over-write the Logstash\_Prefix for index generation. If the key/value is not found in the record then the Logstash\_Prefix option will act as a fallback. Nested keys are not supported \(if desired, you can use the nest filter plugin to remove nesting\) | |
4646
| Suppress\_Type\_Name | When enabled, mapping types is removed and `Type` option is ignored. | Off |
4747
| Workers | Enables dedicated thread(s) for this output. Default value is set since version 1.8.13. For previous versions is 0. | 2 |
4848
| Compress | Set payload compression mechanism. The only available option is `gzip`. Default = "", which means no compression. | |
@@ -128,6 +128,23 @@ becomes
128128

129129
## FAQ
130130

131+
### Logstash_Prefix_Key
132+
133+
The following snippet demonstrates using the namespace name as extracted by the
134+
`kubernetes` filter as logstash preifix:
135+
136+
```text
137+
[OUTPUT]
138+
Name opensearch
139+
Match *
140+
# ...
141+
Logstash_Prefix logstash
142+
Logstash_Prefix_Key $kubernetes["namespace_name"]
143+
# ...
144+
```
145+
146+
For records that do nor have the field `kubernetes.namespace_name`, the default prefix, `logstash` will be used.
147+
131148
### Fluent Bit + Amazon OpenSearch Service <a id="#aws-es"></a>
132149

133150
The Amazon OpenSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. This plugin supports Amazon OpenSearch Service with IAM Authentication.

0 commit comments

Comments
 (0)