Skip to content

Commit 0595551

Browse files
authored
Adding YAML examples and fixing shell layouts for kafka input plugin docs. (#1812)
Signed-off-by: Eric D. Schabell <[email protected]>
1 parent deb6f99 commit 0595551

File tree

1 file changed

+57
-4
lines changed

1 file changed

+57
-4
lines changed

pipeline/inputs/kafka.md

Lines changed: 57 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,19 +20,38 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill
2020

2121
## Get started
2222

23-
To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file:
23+
To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file as shown below.
2424

2525
### Command line
2626

2727
The Kafka plugin can read parameters through the `-p` argument (property):
2828

2929
```shell
30-
fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
30+
$ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
3131
```
3232

3333
### Configuration file
3434

35-
In your main configuration file append the following `Input` and `Output` sections:
35+
In your main configuration file append the following:
36+
37+
{% tabs %}
38+
{% tab title="fluent-bit.yaml" %}
39+
40+
```yaml
41+
pipeline:
42+
inputs:
43+
- name: kafka
44+
brokers: 192.168.1.3:9092
45+
topics: some-topic
46+
poll_ms: 100
47+
48+
outputs:
49+
- name: stdout
50+
match: '*'
51+
```
52+
53+
{% endtab %}
54+
{% tab title="fluent-bit.conf" %}
3655
3756
```text
3857
[INPUT]
@@ -43,12 +62,43 @@ In your main configuration file append the following `Input` and `Output` sectio
4362

4463
[OUTPUT]
4564
Name stdout
65+
Match *
4666
```
4767

68+
{% endtab %}
69+
{% endtabs %}
70+
4871
#### Example of using Kafka input and output plugins
4972

5073
The Fluent Bit source repository contains a full example of using Fluent Bit to process Kafka records:
5174

75+
{% tabs %}
76+
{% tab title="fluent-bit.yaml" %}
77+
78+
```yaml
79+
pipeline:
80+
inputs:
81+
- name: kafka
82+
brokers: kafka-broker:9092
83+
topics: fb-source
84+
poll_ms: 100
85+
format: json
86+
87+
filters:
88+
- name: lua
89+
match: '*'
90+
script: kafka.lua
91+
call: modify_kafka_message
92+
93+
outputs:
94+
- name: kafka
95+
brokers: kafka-broker:9092
96+
topics: fb-sink
97+
```
98+
99+
{% endtab %}
100+
{% tab title="fluent-bit.conf" %}
101+
52102
```text
53103
[INPUT]
54104
Name kafka
@@ -69,10 +119,13 @@ The Fluent Bit source repository contains a full example of using Fluent Bit to
69119
topics fb-sink
70120
```
71121

122+
{% endtab %}
123+
{% endtabs %}
124+
72125
The previous example will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds.
73126

74127
Since the payload will be in JSON format, the plugin is configured to parse the payload with `format json`.
75128

76129
Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
77130

78-
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
131+
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).

0 commit comments

Comments
 (0)