Skip to content

Commit 4ea5e13

Browse files
authored
Adding YAML examples to elastic search output plugin docs. Fixes #1723. (#1750)
Signed-off-by: Eric D. Schabell <[email protected]>
1 parent cbd5ef1 commit 4ea5e13

File tree

1 file changed

+183
-8
lines changed

1 file changed

+183
-8
lines changed

pipeline/outputs/elasticsearch.md

Lines changed: 183 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,29 @@ fluent-bit -i cpu -t cpu -o es -p Host=192.168.2.3 -p Port=9200 \
112112

113113
In your main configuration file append the following `Input` and `Output` sections.
114114

115-
```python
115+
% tabs %}
116+
{% tab title="fluent-bit.yaml" %}
117+
118+
```yaml
119+
pipeline:
120+
inputs:
121+
- name: cpu
122+
tag: cpu
123+
124+
outputs:
125+
- name: es
126+
match: '*'
127+
host: 192.168.2.3
128+
port: 9200
129+
index: my_index
130+
type: my_type
131+
```
132+
133+
{% endtab %}
134+
135+
{% tab title="fluent-bit.conf" %}
136+
137+
```text
116138
[INPUT]
117139
Name cpu
118140
Tag cpu
@@ -126,7 +148,8 @@ In your main configuration file append the following `Input` and `Output` sectio
126148
Type my_type
127149
```
128150

129-
![example configuration visualization from Calyptia](../../.gitbook/assets/image%20%282%29.png)
151+
{% endtab %}
152+
{% endtabs %}
130153

131154
## About Elasticsearch field names
132155

@@ -158,7 +181,31 @@ See [details](https://github.com/fluent/fluent-bit-docs/tree/43c4fe134611da471e7
158181

159182
Example configuration:
160183

161-
```text copy
184+
% tabs %}
185+
{% tab title="fluent-bit.yaml" %}
186+
187+
```yaml
188+
pipeline:
189+
inputs:
190+
...
191+
192+
outputs:
193+
- name: es
194+
match: '*'
195+
host: vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
196+
port: 443
197+
index: my_index
198+
type: my_type
199+
aws_auth: on
200+
aws_region: us-west-2
201+
tls: on
202+
```
203+
204+
{% endtab %}
205+
206+
{% tab title="fluent-bit.conf" %}
207+
208+
```text
162209
[OUTPUT]
163210
Name es
164211
Match *
@@ -171,6 +218,9 @@ Example configuration:
171218
tls On
172219
```
173220

221+
{% endtab %}
222+
{% endtabs %}
223+
174224
Be aware that the `Port` is set to `443`, `tls` is enabled, and `AWS_Region` is set.
175225

176226
### Use Fluent Bit with Elastic Cloud
@@ -184,7 +234,29 @@ to the
184234

185235
Example configuration:
186236

187-
```text copy
237+
% tabs %}
238+
{% tab title="fluent-bit.yaml" %}
239+
240+
```yaml
241+
pipeline:
242+
inputs:
243+
...
244+
245+
outputs:
246+
- name: es
247+
include_tag_key: true
248+
tag_key: tags
249+
tls: on
250+
tls.verify: off
251+
cloud_id: 'elastic-obs-deployment:ZXVybxxxxxxxxxxxg=='
252+
cloud_auth: 'elastic:2vxxxxxxxxYV'
253+
```
254+
255+
{% endtab %}
256+
257+
{% tab title="fluent-bit.conf" %}
258+
259+
```text
188260
[OUTPUT]
189261
Name es
190262
Include_Tag_Key true
@@ -196,6 +268,9 @@ Example configuration:
196268
cloud_auth elastic:2vxxxxxxxxYV
197269
```
198270

271+
{% endtab %}
272+
{% endtabs %}
273+
199274
In Elastic Cloud version 8 and great, the type option must be removed by setting
200275
`Suppress_Type_Name On`.
201276

@@ -221,6 +296,30 @@ Rejecting mapping update to [products] as the final mapping would have more than
221296

222297
This means that you can't set up your configuration like the following:.
223298

299+
% tabs %}
300+
{% tab title="fluent-bit.yaml" %}
301+
302+
```yaml
303+
pipeline:
304+
inputs:
305+
...
306+
307+
outputs:
308+
- name: es
309+
match: 'foo.*'
310+
index: search
311+
type: type1
312+
313+
- name: es
314+
match: 'bar.*'
315+
index: search
316+
type: type2
317+
```
318+
319+
{% endtab %}
320+
321+
{% tab title="fluent-bit.conf" %}
322+
224323
```text
225324
[OUTPUT]
226325
Name es
@@ -235,6 +334,9 @@ This means that you can't set up your configuration like the following:.
235334
Type type2
236335
```
237336

337+
{% endtab %}
338+
{% endtabs %}
339+
238340
For details, read [the official blog post on that issue](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html).
239341

240342
### Mapping type names can't start with underscores (`_`)
@@ -250,7 +352,31 @@ This doesn't work in Elasticsearch versions 5.6 through 6.1
250352
Ensure you set an explicit map such as `doc` or `flb_type` in the configuration,
251353
as seen on the last line:
252354

253-
```text copy
355+
% tabs %}
356+
{% tab title="fluent-bit.yaml" %}
357+
358+
```yaml
359+
pipeline:
360+
inputs:
361+
...
362+
363+
outputs:
364+
- name: es
365+
match: '*'
366+
host: vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
367+
port: 443
368+
index: my_index
369+
aws_auth: on
370+
aws_region: us-west-2
371+
tls: on
372+
type: doc
373+
```
374+
375+
{% endtab %}
376+
377+
{% tab title="fluent-bit.conf" %}
378+
379+
```text
254380
[OUTPUT]
255381
Name es
256382
Match *
@@ -263,10 +389,13 @@ as seen on the last line:
263389
Type doc
264390
```
265391

392+
{% endtab %}
393+
{% endtabs %}
394+
266395
### Validation failures
267396

268397
In Fluent Bit v1.8.2 and greater, Fluent Bit started using `create` method (instead
269-
of `index`) for data submission. This makes Fluent Bit compatible with Datastream,
398+
of `index`) for data submission. This makes Fluent Bit compatible with `Datastream`,
270399
introduced in Elasticsearch 7.9. You might see errors like:
271400

272401
```text
@@ -277,19 +406,62 @@ If you see `action_request_validation_exception` errors on your pipeline with
277406
Fluent Bit versions greater than v1.8.2, correct them by turning on `Generate_ID`
278407
as follows:
279408

280-
```text copy
409+
% tabs %}
410+
{% tab title="fluent-bit.yaml" %}
411+
412+
```yaml
413+
pipeline:
414+
inputs:
415+
...
416+
417+
outputs:
418+
- name: es
419+
match: '*'
420+
host: 192.168.12.1
421+
generate_id: on
422+
```
423+
424+
{% endtab %}
425+
426+
{% tab title="fluent-bit.conf" %}
427+
428+
```text
281429
[OUTPUT]
282430
Name es
283431
Match *
284432
Host 192.168.12.1
285433
Generate_ID on
286434
```
287435

436+
{% endtab %}
437+
{% endtabs %}
438+
288439
### `Logstash_Prefix_Key`
289440

290441
The following snippet demonstrates using the namespace name as extracted by the
291442
`kubernetes` filter as `logstash` prefix:
292443

444+
% tabs %}
445+
{% tab title="fluent-bit.yaml" %}
446+
447+
```yaml
448+
pipeline:
449+
inputs:
450+
...
451+
452+
outputs:
453+
- name: es
454+
match: '*'
455+
# ...
456+
logstash_prefix: logstash
457+
logstash_prefix_key: $kubernetes['namespace_name']
458+
# ...
459+
```
460+
461+
{% endtab %}
462+
463+
{% tab title="fluent-bit.conf" %}
464+
293465
```text
294466
[OUTPUT]
295467
Name es
@@ -300,5 +472,8 @@ The following snippet demonstrates using the namespace name as extracted by the
300472
# ...
301473
```
302474

475+
{% endtab %}
476+
{% endtabs %}
477+
303478
For records that don't have the field `kubernetes.namespace_name`, the default prefix
304-
`logstash` will be used.
479+
`logstash` will be used.

0 commit comments

Comments
 (0)