Skip to content

Commit 801d74b

Browse files
lucabelluccinikilfoyle
lucabelluccini
and
kilfoyle
authored
[Fleet/EA] Logstash & Kafka Outputs refresh (#1306)
* Update output-logstash.asciidoc * Update output-kafka.asciidoc * Update fleet-settings-output-kafka.asciidoc * Update fleet-settings-output-logstash.asciidoc * Update output-logstash.asciidoc * Update docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc Co-authored-by: David Kilfoyle <[email protected]> * Update docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc Co-authored-by: David Kilfoyle <[email protected]> * Update docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc Co-authored-by: David Kilfoyle <[email protected]> * Update docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc Co-authored-by: David Kilfoyle <[email protected]> * Update docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc Co-authored-by: David Kilfoyle <[email protected]> * Update docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc Co-authored-by: David Kilfoyle <[email protected]> --------- Co-authored-by: David Kilfoyle <[email protected]>
1 parent 86de30a commit 801d74b

File tree

4 files changed

+100
-6
lines changed

4 files changed

+100
-6
lines changed

‎docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,29 @@ outputs:
4444
verification_mode: full
4545
----
4646

47+
== Kafka output and using {ls} to index data to {es}
48+
49+
If you are considering using {ls} to ship the data from `kafka` to {es}, please
50+
be aware Elastic is not currently testing this kind of setup.
51+
52+
The structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
53+
We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec.
54+
55+
Refer to <<logstash-output,{ls} output for {agent}>> documentation for more details.
56+
57+
[source,yaml]
58+
----
59+
inputs {
60+
kafka {
61+
...
62+
ecs_compatibility => "disabled"
63+
codec => json { ecs_compatibility => "disabled" }
64+
...
65+
}
66+
}
67+
...
68+
----
69+
4770
== Kafka output configuration settings
4871

4972
The `kafka` output supports the following settings, grouped by category.
@@ -502,4 +525,4 @@ Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silentl
502525

503526
// =============================================================================
504527

505-
|===
528+
|===

‎docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,28 +32,38 @@ To receive the events in {ls}, you also need to create a {ls} configuration pipe
3232
The {ls} configuration pipeline listens for incoming {agent} connections,
3333
processes received events, and then sends the events to {es}.
3434

35-
The following example configures a {ls} pipeline that listens on port `5044` for
36-
incoming {agent} connections and routes received events to {es}:
35+
The following {ls} pipeline definition example configures a pipeline that listens on port `5044` for
36+
incoming {agent} connections and routes received events to {es}.
37+
3738

3839
[source,yaml]
3940
----
4041
input {
4142
elastic_agent {
4243
port => 5044
4344
enrich => none # don't modify the events' schema at all
44-
# or minimal change, add only ssl and source metadata
45-
# enrich => [ssl_peer_metadata, source_metadata]
45+
ssl => true
46+
ssl_certificate_authorities => ["<ca_path>"]
47+
ssl_certificate => "<server_cert_path>"
48+
ssl_key => "<server_cert_key_in_pkcs8>"
49+
ssl_verify_mode => "force_peer"
4650
}
4751
}
4852
4953
output {
5054
elasticsearch {
5155
hosts => ["http://localhost:9200"] <1>
56+
# cloud_id => "..."
5257
data_stream => "true"
58+
api_key => "<api_key>" <2>
59+
data_stream => true
60+
ssl => true
61+
# cacert => "<elasticsearch_ca_path>"
5362
}
5463
}
5564
----
5665
<1> The {es} server and the port (`9200`) where {es} is running.
66+
<2> The API Key used by {ls} to ship data to the destination data streams.
5767

5868
For more information about configuring {ls}, refer to
5969
{logstash-ref}/configuration.html[Configuring {ls}] and

‎docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,29 @@
55

66
Specify these settings to send data over a secure connection to Kafka. In the {fleet} <<output-settings,Output settings>>, make sure that the Kafka output type is selected.
77

8+
== Kafka output and using {ls} to index data to {es}
9+
10+
If you are considering using {ls} to ship the data from `kafka` to {es}, please
11+
be aware Elastic is not currently testing this kind of setup.
12+
13+
The structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
14+
We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec.
15+
16+
Refer to the <<ls-output-settings,{ls} output for {agent}>> documentation for more details.
17+
18+
[source,yaml]
19+
----
20+
inputs {
21+
kafka {
22+
...
23+
ecs_compatibility => "disabled"
24+
codec => json { ecs_compatibility => "disabled" }
25+
...
26+
}
27+
}
28+
...
29+
----
30+
831
[discrete]
932
== General settings
1033

‎docs/en/ingest-management/fleet/fleet-settings-output-logstash.asciidoc‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,44 @@ Before using the {ls} output, you need to make sure that for any integrations th
1313

1414
To learn how to generate certificates, refer to <<secure-logstash-connections>>.
1515

16+
To receive the events in {ls}, you also need to create a {ls} configuration pipeline.
17+
The {ls} configuration pipeline listens for incoming {agent} connections,
18+
processes received events, and then sends the events to {es}.
19+
20+
The following example configures a {ls} pipeline that listens on port `5044` for
21+
incoming {agent} connections and routes received events to {es}.
22+
23+
The {ls} pipeline definition below is an example. Please refer to the `Additional Logstash
24+
configuration required` steps when creating the {ls} output in the Fleet outputs page.
25+
26+
[source,yaml]
27+
----
28+
input {
29+
elastic_agent {
30+
port => 5044
31+
enrich => none # don't modify the events' schema at all
32+
ssl => true
33+
ssl_certificate_authorities => ["<ca_path>"]
34+
ssl_certificate => "<server_cert_path>"
35+
ssl_key => "<server_cert_key_in_pkcs8>"
36+
ssl_verify_mode => "force_peer"
37+
}
38+
}
39+
output {
40+
elasticsearch {
41+
hosts => ["http://localhost:9200"] <1>
42+
# cloud_id => "..."
43+
data_stream => "true"
44+
api_key => "<api_key>" <2>
45+
data_stream => true
46+
ssl => true
47+
# cacert => "<elasticsearch_ca_path>"
48+
}
49+
}
50+
----
51+
<1> The {es} server and the port (`9200`) where {es} is running.
52+
<2> The API Key obtained from the {ls} output creation steps in Fleet.
53+
1654
[cols="2*<a"]
1755
|===
1856
|
@@ -196,4 +234,4 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[
196234

197235
|===
198236

199-
:type!:
237+
:type!:

0 commit comments

Comments
 (0)