You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
= Configuring the Flow Collector resource with Kafka
8
-
You can configure the `FlowCollector` resource to use Kafka. A Kafka instance needs to be running, and a Kafka topic dedicated to {product-title} Network Observability must be created in that instance. For more information, refer to your Kafka documentation, such as link:https://access.redhat.com/documentation/en-us/red_hat_amq/7.7/html/using_amq_streams_on_openshift/using-the-topic-operator-str[Kafka documentation with AMQ Streams].
8
+
You can configure the `FlowCollector` resource to use Kafka for high-throughput and low-latency data feeds. A Kafka instance needs to be running, and a Kafka topic dedicated to {product-title} Network Observability must be created in that instance. For more information, see link:https://access.redhat.com/documentation/en-us/red_hat_amq/7.7/html/using_amq_streams_on_openshift/using-the-topic-operator-str[Kafka documentation with AMQ Streams].
9
9
10
-
The following example shows how to modify the `FlowCollector` resource for {product-title} Network Observability operator to use Kafka:
10
+
.Prerequisites
11
+
* Kafka is installed. Red Hat supports Kafka with AMQ Streams Operator.
12
+
13
+
.Procedure
14
+
. In the web console, navigate to *Operators* → *Installed Operators*.
15
+
16
+
. Under the *Provided APIs* heading for the Network Observability Operator, select *Flow Collector*.
17
+
18
+
. Select the cluster and then click the *YAML* tab.
19
+
20
+
. Modify the `FlowCollector` resource for {product-title} Network Observability Operator to use Kafka, as shown in the following sample YAML:
11
21
12
22
.Sample Kafka configuration in `FlowCollector` resource
The Kafka Operator is supported for large scale environments. You can install the Kafka Operator as link:https://access.redhat.com/documentation/en-us/red_hat_amq_streams/2.2[Red Hat AMQ Streams] from the Operator Hub, just as the Loki Operator and Network Observability Operator were installed.
8
+
The Kafka Operator is supported for large scale environments. Kafka provides high-throughput and low-latency data feeds for forwarding network flow data in a more resilient, scalable way. You can install the Kafka Operator as link:https://access.redhat.com/documentation/en-us/red_hat_amq_streams/2.2[Red Hat AMQ Streams] from the Operator Hub, just as the Loki Operator and Network Observability Operator were installed. Refer to "Configuring the FlowCollector resource with Kafka" to configure Kafka as a storage option.
Copy file name to clipboardExpand all lines: modules/network-observability-operator-install.adoc
+17-19Lines changed: 17 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,6 +7,11 @@
7
7
= Installing the Network Observability Operator
8
8
You can install the Network Observability Operator using the {product-title} web console Operator Hub. When you install the Operator, it provides the `FlowCollector` custom resource definition (CRD). You can set specifications in the web console when you create the `FlowCollector`.
9
9
10
+
[IMPORTANT]
11
+
====
12
+
The actual memory consumption of the Operator depends on your cluster size and the number of resources deployed. Memory consumption might need to be adjusted accordingly. For more information refer to "Network Observability controller manager pod runs out of memory" in the "Important Flow Collector configuration considerations" section.
13
+
====
14
+
10
15
.Prerequisites
11
16
12
17
* If you choose to use Loki, install the link:https://catalog.redhat.com/software/containers/openshift-logging/loki-rhel8-operator/622b46bcae289285d6fcda39[Loki Operator version 5.7+].
@@ -26,26 +31,19 @@ This documentation assumes that your `LokiStack` instance name is `loki`. Using
26
31
. Choose *Network Observability Operator* from the list of available Operators in the *OperatorHub*, and click *Install*.
27
32
. Select the checkbox `Enable Operator recommended cluster monitoring on this Namespace`.
28
33
. Navigate to *Operators*->*Installed Operators*. Under Provided APIs for Network Observability, select the *Flow Collector* link.
29
-
.. Navigate to the *Flow Collector* tab, and click *Create FlowCollector*. Make the following selections in the form view:
30
-
+
31
-
* *spec.agent.ebpf.Sampling* : Specify a sampling size for flows. Lower sampling sizes will have higher impact on resource utilization. For more information, see the `FlowCollector` API reference, under spec.agent.ebpf.
32
-
* *spec.deploymentModel*: If you are using Kafka, verify Kafka is selected.
33
-
* *spec.exporters*: If you are using Kafka, you can optionally send network flows to Kafka, so that they can be consumed by any processor or storage that supports Kafka input, such as Splunk, Elasticsearch, or Fluentd. To do this, set the following specifications:
34
-
** Set the *type* to `KAFKA`.
35
-
** Set the *address* as `kafka-cluster-kafka-bootstrap.netobserv`.
36
-
** Set the *topic* as `netobserv-flows-export`. The Operator exports all flows to the configured Kafka topic.
37
-
** Set the following *tls* specifications:
38
-
*** *certFile*: `service-ca.crt`, *name*: `kafka-gateway-ca-bundle`, and *type*: `configmap`.
39
-
+
40
-
You can also configure this option at a later time by directly editing the YAML. For more information, see _Export enriched network flow data_.
41
-
* *loki.enable*: Set to `true`.
42
-
* *loki.url*: Since authentication is specified separately, this URL needs to be updated to `https://loki-gateway-http.netobserv.svc:8080/api/logs/v1/network`. The first part of the URL, "loki", should match the name of your LokiStack.
43
-
* *loki.statusUrl*: Set this to `https://loki-query-frontend-http.netobserv.svc:3100/`. The first part of the URL, "loki", should match the name of your LokiStack.
44
-
* *loki.authToken*: Select the `FORWARD` value.
45
-
* *tls.enable*: Verify that the box is checked so it is enabled.
46
-
* *statusTls*: The `enable` value is false by default.
34
+
. Navigate to the *Flow Collector* tab, and click *Create FlowCollector*. Make the following selections in the form view:
35
+
.. *spec.agent.ebpf.Sampling*: Specify a sampling size for flows. Lower sampling sizes will have higher impact on resource utilization. For more information, see the "FlowCollector API reference", `spec.agent.ebpf`.
36
+
.. If you are using Loki, set the following specifications:
37
+
...*spec.loki.enable*: Select the check box to enable storing flows in Loki.
38
+
...*spec.loki.url*: Since authentication is specified separately, this URL needs to be updated to `https://loki-gateway-http.netobserv.svc:8080/api/logs/v1/network`. The first part of the URL, "loki", must match the name of your `LokiStack`.
39
+
...*spec.loki.authToken*: Select the `FORWARD` value.
40
+
...*spec.loki.statusUrl*: Set this to `https://loki-query-frontend-http.netobserv.svc:3100/`. The first part of the URL, "loki", must match the name of your `LokiStack`.
41
+
...*spec.loki.tls.enable*: Select the checkbox to enable TLS.
42
+
...*spec.loki.statusTls*: The `enable` value is false by default.
47
43
+
48
-
For the first part of the certificate reference names: `loki-gateway-ca-bundle`, `loki-ca-bundle`, and `loki-query-frontend-http`,`loki`, should match the name of your `LokiStack`.
44
+
For the first part of the certificate reference names: `loki-gateway-ca-bundle`, `loki-ca-bundle`, and `loki-query-frontend-http`,`loki`, must match the name of your `LokiStack`.
45
+
.. Optional: If you are in a large-scale environment, consider configuring the `FlowCollector` with Kafka for forwarding data in a more resilient, scalable way. See "Configuring the Flow Collector resource with Kafka storage" in the "Important Flow Collector configuration considerations" section.
46
+
.. Optional: Configure other optional settings before the next step of creating the `FlowCollector`. For example, if you choose not to use Loki, then you can configure exporting flows to Kafka or IPFIX. See "Export enriched network flow data to Kafka and IPFIX" and more in the "Important Flow Collector configuration considerations" section.
Copy file name to clipboardExpand all lines: modules/network-observability-quickfilter.adoc
+4-5Lines changed: 4 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,12 +27,11 @@ The default values in *Quick filters* drop-down menu are defined in the `FlowCol
27
27
Advanced filters::
28
28
You can set the advanced filters, *Common*, *Source*, or *Destination*, by selecting the parameter to be filtered from the dropdown list. The flow data is filtered based on the selection. To enable or disable the applied filter, you can click on the applied filter listed below the filter options.
29
29
30
+
You can toggle between image:arrow-up-long-solid.png[,10] *One way* and image:arrow-up-long-solid.png[,10] image:arrow-down-long-solid.png[,10] *Back and forth* filtering. The image:arrow-up-long-solid.png[,10] *One way* filter shows only *Source* and *Destination* traffic according to your filter selections. You can use *Swap* to change the directional view of the *Source* and *Destination* traffic. The image:arrow-up-long-solid.png[,10] image:arrow-down-long-solid.png[,10] *Back and forth* filter includes return traffic with the *Source* and *Destination* filters. The directional flow of network traffic is shown in the *Direction* column in the Traffic flows table as `Ingress`or `Egress` for inter-node traffic and `Inner`for traffic inside a single node.
31
+
32
+
You can click *Reset defaults* to remove the existing filters, and apply the filter defined in `FlowCollector` configuration.
33
+
30
34
[NOTE]
31
35
====
32
36
To understand the rules of specifying the text value, click *Learn More*.
33
37
====
34
-
35
-
You can toggle between image:arrow-up-long-solid.png[,10] *One way* and image:arrow-up-long-solid.png[,10] image:arrow-down-long-solid.png[,10] *Back and forth* filtering. The image:arrow-up-long-solid.png[,10] *One way* filter shows only *Source* and *Destination* traffic according to your filter selections. You can use *Swap* to change the directional view of the *Source* and *Destination* traffic. The image:arrow-up-long-solid.png[,10] image:arrow-down-long-solid.png[,10] *Back and forth* filter includes return traffic with the *Source* and *Destination* filters.
36
-
37
-
You can click *Reset defaults* to remove the existing filters, and apply the filter defined in `FlowCollector` configuration.
You can use Network Observability without Loki by not performing the Loki installation steps in the following section and instead using exporters, such as Kafka or IPFIX. The following table compares available features with and without Loki:
7
+
You can use Network Observability without Loki by not performing the Loki installation steps and skipping directly to "Installing the Network Observability Operator". If you only want to export flows to a Kafka consumer or IPFIX collector, or you only need dashboard metrics, then you do not need to install Loki or provide storage for Loki. Without Loki, there won't be a Network Traffic panel under Observe, which means there is no overview charts, flow table, or topology. The following table compares available features with and without Loki:
8
8
9
9
.Comparison of feature availability with and without Loki
* For more information about the option to use different namespaces for the separate components, see the `spec.loki.tls.caCert.namespace` specification in the xref:../network_observability/flowcollector-api.adoc#network-observability-flowcollector-api-specifications_network_observability[Flow Collector API Reference] and callout number 5 in the xref:../network_observability/configuring-operator.adoc#network-observability-flowcollector-view_network_observability[Flow Collector sample resource].
27
+
* For more information about the option to use different namespaces for the separate components, see the `spec.loki.tls.caCert.namespace` specification in the xref:../../networking/network_observability/flowcollector-api.adoc#network-observability-flowcollector-api-specifications_network_observability[Flow Collector API Reference] and callout number 5 in the xref:../../networking/network_observability/configuring-operator.adoc#network-observability-flowcollector-view_network_observability[Flow Collector sample resource].
== Important Flow Collector configuration considerations
39
+
Once you create the `FlowCollector` instance, you can reconfigure it, but the pods are terminated and recreated again, which can be disruptive. Therefore, you can consider configuring the following options when creating the `FlowCollector` for the first time:
40
+
41
+
* xref:../../networking/network_observability/configuring-operator.adoc#network-observability-flowcollector-kafka-config_network_observability[Configuring the Flow Collector resource with Kafka]
42
+
* xref:../../networking/network_observability/configuring-operator.adoc#network-observability-enriched-flows_network_observability[Export enriched network flow data to Kafka or IPFIX]
43
+
* xref:../../networking/network_observability/configuring-operator.adoc#network-observability-SR-IOV-config_network_observability[Configuring monitoring for SR-IOV interface traffic]
44
+
* xref:../../networking/network_observability/observing-network-traffic.adoc#network-observability-working-with-conversations_nw-observe-network-traffic[Working with conversation tracking]
45
+
* xref:../../networking/network_observability/observing-network-traffic.adoc#network-observability-dns-tracking_nw-observe-network-traffic[Working with DNS tracking]
46
+
* xref:../../networking/network_observability/observing-network-traffic.adoc#network-observability-packet-drops_nw-observe-network-traffic[Working with packet drops]
47
+
37
48
[role="_additional-resources"]
38
49
.Additional resources
39
-
* For more information about Flow Collector specifications, see the xref:../../networking/network_observability/flowcollector-api.adoc#network-observability-flowcollector-api-specifications_network_observability[Flow Collector API Reference] and the xref:../../networking/network_observability/configuring-operator.adoc#network-observability-flowcollector-view_network_observability[Flow Collector sample resource].
50
+
For more general information about Flow Collector specifications and the Network Observability Operator architecture and resource use, see the following resources:
40
51
41
-
* For more information about exporting flow data to Kafka or IPFIX for third party processing consumption, see xref:../../networking/network_observability/configuring-operator.adoc#network-observability-enriched-flows_network_observability[Export enriched network flow data].
52
+
* xref:../../networking/network_observability/flowcollector-api.adoc#network-observability-flowcollector-api-specifications_network_observability[Flow Collector API Reference]
* xref:../../networking/network_observability/troubleshooting-network-observability.adoc#controller-manager-pod-runs-out-of-memory_network-observability-troubleshooting[Troubleshooting Network Observability controller manager pod runs out of memory]
xref:../../networking/network_observability/configuring-operator.adoc#network-observability-flowcollector-kafka-config_network_observability[Configuring the FlowCollector resource with Kafka].
0 commit comments