Skip to content

Commit 325075c

Browse files
esmerelesmerel
committed
fluent: docs: elastic search plugin style updates
Signed-off-by: Lynette Miles <[email protected]>
1 parent 43bb34e commit 325075c

File tree

2 files changed

+136
-98
lines changed

2 files changed

+136
-98
lines changed

‎pipeline/outputs/elasticsearch.md

Lines changed: 135 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -12,28 +12,27 @@ operational Elasticsearch service running in your environment.
1212

1313
| Key | Description | Default |
1414
| :--- | :--- | :--- |
15-
| `Host` | IP address or hostname of the target Elasticsearch instance | 127.0.0.1 |
15+
| `Host` | IP address or hostname of the target Elasticsearch instance | `127.0.0.1` |
1616
| `Port` | TCP port of the target Elasticsearch instance | 9200 |
17-
| `Path` | Elasticsearch accepts new data on HTTP query path `/_bulk`. It's also possible to serve Elasticsearch behind a reverse proxy on a sub-path. Define the path by adding a path prefix in the indexing HTTP POST URI. | Empty string |
18-
| `compress` | Set payload compression mechanism. Option available is 'gzip' | |
19-
| `Buffer_Size` | Specify the buffer size used to read the response from the Elasticsearch HTTP service. Useful for debugging purposes where it's required to read full responses. Response size grows depending of the number of records inserted. To set an _unlimited_ amount of memory set this value to **False**, otherwise the value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `512KB` |
20-
| `Pipeline` | Newer versions of Elasticsearch allows to setup filters called pipelines. This option allows to define which pipeline the database should use. For performance reasons is strongly suggested to do parsing and filtering on Fluent Bit side, avoid pipelines. | |
21-
| `AWS_Auth` | Enable AWS Sigv4 Authentication for Amazon OpenSearch Service | Off |
22-
| `AWS_Region` | Specify the AWS region for Amazon OpenSearch Service | |
17+
| `Path` | Elasticsearch accepts new data on HTTP query path `/_bulk`. You can also serve Elasticsearch behind a reverse proxy on a sub-path. Define the path by adding a path prefix in the indexing HTTP POST URI. | Empty string |
18+
| `compress` | Set payload compression mechanism. Option available is `gzip`. | |
19+
| `Buffer_Size` | Specify the buffer size used to read the response from the Elasticsearch HTTP service. Use for debugging purposes where required to read full responses. Response size grows depending of the number of records inserted. To use an unlimited amount of memory, set this value to `False`. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md). | `512KB` |
20+
| `Pipeline` | Define which pipeline the database should use. For performance reasons, it's strongly suggested to do parsing and filtering on Fluent Bit side, and avoid pipelines. | |
21+
| `AWS_Auth` | Enable AWS Sigv4 Authentication for Amazon OpenSearch Service. | `Off` |
22+
| `AWS_Region` | Specify the AWS region for Amazon OpenSearch Service. | |
2323
| `AWS_STS_Endpoint` | Specify the custom STS endpoint to be used with STS API for Amazon OpenSearch Service | |
2424
| `AWS_Role_ARN` | AWS IAM Role to assume to put records to your Amazon cluster | |
2525
| `AWS_External_ID` | External ID for the AWS IAM Role specified with `aws_role_arn` | |
2626
| `AWS_Service_Name` | Service name to use in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See the [FAQ](opensearch.md#faq) section on Amazon OpenSearch Serverless for more information. | `es` |
2727
| `AWS_Profile` | AWS profile name | default |
28-
| `Cloud_ID` | If using Elastic's Elasticsearch Service you can specify the `cloud_id` of the cluster running. The string has the format `<deployment_name>:<base64_info>`. Once decoded, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`.
29-
| |
28+
| `Cloud_ID` | If using Elastic's Elasticsearch Service you can specify the `cloud_id` of the cluster running. The string has the format `<deployment_name>:<base64_info>`. Once decoded, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`. | |
3029
| `Cloud_Auth` | Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud | |
3130
| `HTTP_User` | Optional username credential for Elastic X-Pack access | |
3231
| `HTTP_Passwd` | Password for user defined in `HTTP_User` | |
33-
| `Index` | Index name | fluent-bit |
32+
| `Index` | Index name | `fluent-bit` |
3433
| `Type` | Type name | `_doc` |
3534
| `Logstash_Format` | Enable Logstash format compatibility. This option takes a Boolean value: `True/False`, `On/Off` | `Off` |
36-
| `Logstash_Prefix` | When Logstash\_Format is enabled, the Index name is composed using a prefix and the date, e.g: If `Logstash_Prefix` is equal to `mydata` your index will become `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | `logstash` |
35+
| `Logstash_Prefix` | When `Logstash_Format` is enabled, the Index name is composed using a prefix and the date, e.g: If `Logstash_Prefix` is equal to `mydata` your index will become `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | `logstash` |
3736
| `Logstash_Prefix_Key` | When included: the value of the key in the record will be evaluated as key reference and overrides `Logstash_Prefix` for index generation. If the key/value isn't found in the record then the `Logstash_Prefix` option will act as a fallback. The parameter is expected to be a [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md). | |
3837
| `Logstash_Prefix_Separator` | Set a separator between `Logstash_Prefix` and date.| - |
3938
| `Logstash_DateFormat` | Time format based on [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html) to generate the second part of the Index name. | `%Y.%m.%d` |
@@ -45,17 +44,16 @@ operational Elasticsearch service running in your environment.
4544
| `Generate_ID` | When enabled, generate `_id` for outgoing records. This prevents duplicate records when retrying ES. | `Off` |
4645
| `Id_Key` | If set, `_id` will be the value of the key from incoming record and `Generate_ID` option is ignored. | |
4746
| `Write_Operation` | `Write_operation` can be any of: `create`, `index`, `update`, `upsert`. | `create` |
48-
| `Replace_Dots` | When enabled, replace field name dots with underscore, required by Elasticsearch 2.0-2.3. | `Off` |
49-
| `Trace_Output` | Print all ElasticSearch API request payloads to `stdout` for diagnostics | `Off` |
50-
| `Trace_Error` | If ElasticSearch returns an error, print the ElasticSearch API request and response for diagnostics | `Off` |
51-
| `Current_Time_Index` | Use current time for index generation instead of message record | `Off` |
52-
| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is
53-
ignored. Elasticsearch 8.0.0 or higher [no longer supports mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), and is set to `On`. | `Off` |
47+
| `Replace_Dots` | When enabled, replace field name dots with underscore. Required by Elasticsearch 2.0-2.3. | `Off` |
48+
| `Trace_Output` | Print all ElasticSearch API request payloads to `stdout` for diagnostics. | `Off` |
49+
| `Trace_Error` | If ElasticSearch returns an error, print the ElasticSearch API request and response for diagnostics. | `Off` |
50+
| `Current_Time_Index` | Use current time for index generation instead of message record. | `Off` |
51+
| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is ignored. Elasticsearch 8.0.0 or higher [no longer supports mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), and is set to `On`. | `Off` |
5452
| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` |
5553

56-
The parameters `index` and `type` can be confusing if you are new to Elastic, if you
57-
have used a common relational database before, they can be compared to the `database`
58-
and `table` concepts. Also see [the FAQ](elasticsearch.md#faq)
54+
If you have used a common relational database, the parameters `index` and `type` can
55+
be compared to the `database` and `table` concepts. Also see [the
56+
FAQ](elasticsearch.md#faq)
5957

6058
### TLS / SSL
6159

@@ -66,14 +64,18 @@ available and general configuration, refer to[TLS/SSL](../../administration/tran
6664

6765
The `write_operation` can be any of:
6866

69-
| Operation | Description |
70-
| ------------- | ----------- |
71-
| `create` | Adds new data. If the data already exists (based on its id), the op is skipped.|
67+
| Operation | Description |
68+
| ----------- | ----------- |
69+
| `create` | Adds new data. If the data already exists (based on its id), the op is skipped.|
7270
| `index` | New data is added while existing data (based on its id) is replaced (reindexed).|
73-
| `update` | Updates existing data (based on its id). If no data is found, the op is skipped.|
74-
| `upsert` | Known as merge or insert if the data does not exist, updates if the data exists (based on its id).|
71+
| `update` | Updates existing data (based on its id). If no data is found, the op is skipped. |
72+
| `upsert` | Merge or insert if the data doesn't exist, updates if the data exists (based on its id).|
7573

76-
**Please note, `Id_Key` or `Generate_ID` is required in update, and upsert scenario.**
74+
{% hint style="info" %}
75+
76+
`Id_Key` or `Generate_ID` is required for `update` and `upsert`.
77+
78+
{% endhint %}
7779

7880
## Get started
7981

@@ -82,7 +84,12 @@ command line or through the configuration file:
8284

8385
### Command Line
8486

85-
The **es** plugin can read the parameters from the command line in two ways, through the **-p** argument (property) or setting them directly through the service URI. The URI format is the following:
87+
The **es** plugin can read the parameters from the command line in two ways:
88+
89+
- Through the `-p` argument (property)
90+
- Setting them directly through the service URI.
91+
92+
The URI format is the following:
8693

8794
```text
8895
es://host:port/index/type
@@ -104,7 +111,7 @@ fluent-bit -i cpu -t cpu -o es -p Host=192.168.2.3 -p Port=9200 \
104111

105112
### Configuration File
106113

107-
In your main configuration file append the following _Input_ & _Output_ sections. You can visualize this configuration [here](https://link.calyptia.com/qhq)
114+
In your main configuration file append the following `Input` and `Output` sections. You can visualize this configuration [here](https://link.calyptia.com/qhq)
108115

109116
```python
110117
[INPUT]
@@ -124,7 +131,9 @@ In your main configuration file append the following _Input_ & _Output_ sections
124131

125132
## About Elasticsearch field names
126133

127-
Some input plugins can generate messages where the field names contains dots, since Elasticsearch 2.0 this is not longer allowed, so the current **es** plugin replaces them with an underscore, e.g:
134+
Some input plugins can generate messages where the field names contains dots. For
135+
Elasticsearch 2.0, this isn't allowed. The current **es** plugin replaces
136+
them with an underscore:
128137

129138
```text
130139
{"cpu0.p_cpu"=>17.000000}
@@ -136,62 +145,21 @@ becomes
136145
{"cpu0_p_cpu"=>17.000000}
137146
```
138147

139-
## FAQ
140-
141-
### Elasticsearch rejects requests saying "the final mapping would have more than 1 type" <a id="faq-multiple-types"></a>
142-
143-
Elasticsearch 6.0 can't create multiple types in a single index. This
144-
means that you can't set up your configuration like the following:.
145-
146-
```text
147-
[OUTPUT]
148-
Name es
149-
Match foo.*
150-
Index search
151-
Type type1
152-
153-
[OUTPUT]
154-
Name es
155-
Match bar.*
156-
Index search
157-
Type type2
158-
```
159-
160-
An error message like the following indicats you need to update your configuration to
161-
use a single type on each index.
162-
163-
```text
164-
Rejecting mapping update to [search] as the final mapping would have more than 1 type
165-
```
166-
167-
For details, read [the official blog post on that issue](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html).
168-
169-
### Elasticsearch rejects requests saying "Document mapping type name can't start with '\_'" <a id="faq-underscore"></a>
148+
## Use Fluent Bit ElasticSearch plugin with other services
170149

171-
Fluent Bit v1.5 changed the default mapping type from `flb_type` to `_doc`, which matches the recommendation from Elasticsearch from version 6.2 forwards \([see commit with rationale](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196)\). This doesn't work in Elasticsearch versions 5.6 through 6.1 \([see Elasticsearch discussion and fix](https://discuss.elastic.co/t/cant-use-doc-as-type-despite-it-being-declared-the-preferred-method/113837/9)\). Ensure you set an explicit map \(such as `doc` or `flb_type`\) in the configuration, as seen on the last line:
172-
173-
```text
174-
[OUTPUT]
175-
Name es
176-
Match *
177-
Host vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
178-
Port 443
179-
Index my_index
180-
AWS_Auth On
181-
AWS_Region us-west-2
182-
tls On
183-
Type doc
184-
```
150+
Connect to Amazon OpenSearch or Elastic Cloud with the ElasticSearch plugin.
185151

186-
### Fluent Bit + Amazon OpenSearch Service <a id="#aws-es"></a>
152+
### Amazon OpenSearch Service
187153

188-
The Amazon OpenSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon OpenSearch Service with IAM Authentication.
154+
The Amazon OpenSearch Service adds an extra security layer where HTTP requests must
155+
be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon
156+
OpenSearch Service with IAM Authentication.
189157

190-
See [here](https://github.com/fluent/fluent-bit-docs/tree/43c4fe134611da471e706b0edb2f9acd7cdfdbc3/administration/aws-credentials.md) for details on how AWS credentials are fetched.
158+
See [details](https://github.com/fluent/fluent-bit-docs/tree/43c4fe134611da471e706b0edb2f9acd7cdfdbc3/administration/aws-credentials.md) on how AWS credentials are fetched.
191159

192160
Example configuration:
193161

194-
```text
162+
```text copy
195163
[OUTPUT]
196164
Name es
197165
Match *
@@ -204,16 +172,20 @@ Example configuration:
204172
tls On
205173
```
206174

207-
Notice that the `Port` is set to `443`, `tls` is enabled, and `AWS_Region` is set.
175+
Be aware that the `Port` is set to `443`, `tls` is enabled, and `AWS_Region` is set.
208176

209-
### Fluent Bit + Elastic Cloud
177+
### Use Fluent Bit with Elastic Cloud
210178

211-
Fluent Bit supports connecting to [Elastic Cloud](https://www.elastic.co/guide/en/cloud/current/ec-getting-started.html) providing just the `cloud_id` and the `cloud_auth` settings.
212-
`cloud_auth` uses the `elastic` user and password provided when the cluster was created, for details refer to the [Cloud ID usage page](https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html).
179+
Fluent Bit supports connecting to
180+
[Elastic Cloud](https://www.elastic.co/guide/en/cloud/current/ec-getting-started.html)
181+
by providing the `cloud_id` and the `cloud_auth` settings. `cloud_auth` uses the
182+
`elastic` user and password provided when the cluster was created. For details refer
183+
to the
184+
[Cloud ID usage page](https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html).
213185

214186
Example configuration:
215187

216-
```text
188+
```text copy
217189
[OUTPUT]
218190
Name es
219191
Include_Tag_Key true
@@ -225,35 +197,99 @@ Example configuration:
225197
cloud_auth elastic:2vxxxxxxxxYV
226198
```
227199

228-
### Validation Failed: 1: an id must be provided if version type or value are set
200+
In Elastic Cloud version 8 and great, the type option must be removed by setting
201+
`Suppress_Type_Name On`.
202+
203+
Without this you will see errors like:
204+
205+
```text
206+
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
207+
```
229208

230-
Since v1.8.2, Fluent Bit started using `create` method (instead of `index`) for data submission.
231-
This makes Fluent Bit compatible with Datastream introduced in Elasticsearch 7.9.
209+
## Troubleshooting
232210

233-
If you see `action_request_validation_exception` errors on your pipeline with Fluent Bit >= v1.8.2, you can fix it up by turning on `Generate_ID` as follows:
211+
Use the following information to help resolve errors using the ElasticSearch plugin.
212+
213+
### Using multiple types in a single index
214+
215+
Elasticsearch 6.0 can't create multiple types in a single index. An error message
216+
like the following indicates you need to update your configuration to use a single
217+
type on each index.
218+
219+
```text
220+
Rejecting mapping update to [products] as the final mapping would have more than 1 type:
221+
```
222+
223+
This means that you can't set up your configuration like the following:.
234224

235225
```text
236226
[OUTPUT]
237-
Name es
238-
Match *
239-
Host 192.168.12.1
240-
Generate_ID on
227+
Name es
228+
Match foo.*
229+
Index search
230+
Type type1
231+
232+
[OUTPUT]
233+
Name es
234+
Match bar.*
235+
Index search
236+
Type type2
241237
```
242238

243-
### Action/metadata contains an unknown parameter type
239+
For details, read [the official blog post on that issue](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html).
240+
241+
### Mapping type names can't start with underscores (`_`)
244242

245-
Elastic Cloud is now on version 8 so the type option must be removed by setting `Suppress_Type_Name On` as indicated above.
243+
Fluent Bit v1.5 changed the default mapping type from `flb_type` to `_doc`, matching
244+
the recommendation from Elasticsearch for version 6.2 and greater
245+
([see commit with
246+
rationale](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196)).
246247

247-
Without this you will see errors like:
248+
This doesn't work in Elasticsearch versions 5.6 through 6.1
249+
([discussion and fix](https://discuss.elastic.co/t/cant-use-doc-as-type-despite-it-being-declared-the-preferred-method/113837/9)).
250+
251+
Ensure you set an explicit map such as `doc` or `flb_type` in the configuration,
252+
as seen on the last line:
253+
254+
```text copy
255+
[OUTPUT]
256+
Name es
257+
Match *
258+
Host vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
259+
Port 443
260+
Index my_index
261+
AWS_Auth On
262+
AWS_Region us-west-2
263+
tls On
264+
Type doc
265+
```
266+
267+
### Validation failures
268+
269+
In Fluent Bit v1.8.2 and greater, Fluent Bit started using `create` method (instead
270+
of `index`) for data submission. This makes Fluent Bit compatible with Datastream,
271+
introduced in Elasticsearch 7.9. You might see errors like:
248272

249273
```text
250-
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
274+
Validation Failed: 1: an id must be provided if version type or value are set
275+
```
276+
277+
If you see `action_request_validation_exception` errors on your pipeline with
278+
Fluent Bit versions greater than v1.8.2, correct them by turning on `Generate_ID`
279+
as follows:
280+
281+
```text copy
282+
[OUTPUT]
283+
Name es
284+
Match *
285+
Host 192.168.12.1
286+
Generate_ID on
251287
```
252288

253-
### Logstash_Prefix_Key
289+
### `Logstash_Prefix_Key`
254290

255291
The following snippet demonstrates using the namespace name as extracted by the
256-
`kubernetes` filter as logstash prefix:
292+
`kubernetes` filter as `logstash` prefix:
257293

258294
```text
259295
[OUTPUT]
@@ -265,4 +301,5 @@ The following snippet demonstrates using the namespace name as extracted by the
265301
# ...
266302
```
267303

268-
For records that do nor have the field `kubernetes.namespace_name`, the default prefix, `logstash` will be used.
304+
For records that don't have the field `kubernetes.namespace_name`, the default prefix
305+
`logstash` will be used.

‎vale-styles/FluentBit/Spelling-exceptions.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ Datadog
2525
Datagen
2626
datapoint
2727
datapoints
28+
Datastream
2829
declaratively
2930
deduplicate
3031
Deployer

0 commit comments

Comments
 (0)