Merge pull request #56104 from skrthomas/OSDOCS-5286

Author: skrthomas

modules/network-observability-flowcollector-view.adoc

@@ -44,7 +44,10 @@ spec:
         cpu: 100m
       limits:
         memory: 800Mi
-  loki:                                       <3>
+    conversationEndTimeout: 10s
+    logTypes: FLOW                            <3>
+    conversationHeartbeatInterval: 30s
+  loki:                                       <4>
     url: 'http://loki-gateway-http.netobserv.svc:8080/api/logs/v1/network'
     statusUrl: 'https://loki-query-frontend-http.netobserv.svc:3100/'
     authToken: FORWARD
@@ -61,7 +64,7 @@ spec:
       enable: true
       portNames:
         "3100": loki
-    quickFilters:                             <4>
+    quickFilters:                             <5>
     - name: Applications
       filter:
         src_namespace!: 'openshift-,netobserv'
@@ -82,5 +85,6 @@ spec:
 ----
 <1> The Agent specification, `spec.agent.type`, must be `EBPF`. eBPF is the only {product-title} supported option.
 <2> You can set the Sampling specification, `spec.agent.ebpf.sampling`, to manage resources. Lower sampling values might consume a large amount of computational, memory and storage resources. You can mitigate this by specifying a sampling ratio value. A value of 100 means 1 flow every 100 is sampled. A value of 0 or 1 means all flows are captured. The lower the value, the increase in returned flows and the accuracy of derived metrics. By default, eBPF sampling is set to a value of 50, so 1 flow every 50 is sampled. Note that more sampled flows also means more storage needed. It is recommend to start with default values and refine empirically, to determine which setting your cluster can manage.
-<3> The Loki specification, `spec.loki`, specifies the Loki client. The default values match the Loki install paths mentioned in the Installing the Loki Operator section. If you used another installation method for Loki, specify the appropriate client information for your install.
-<4> The `spec.quickFilters` specification defines filters that show up in the web console. The `Application` filter keys,`src_namespace` and `dst_namespace`, are negated (`!`), so the `Application` filter shows all traffic that _does not_ originate from, or have a destination to, any `openshift-` or `netobserv` namespaces. For more information, see Configuring quick filters below.
+<3> The optional specifications `spec.processor.logTypes`, `spec.processor.conversationHeartbeatInterval`, and `spec.processor.conversationEndTimeout` can be set to enable conversation tracking. When enabled, conversation events are queryable in the web console. The values for `spec.processor.logTypes` are as follows: `FLOWS` `CONVERSATIONS`, `ENDED_CONVERSATIONS`, or `ALL`. Storage requirements are highest for `ALL` and lowest for `ENDED_CONVERSATIONS`.
+<4> The Loki specification, `spec.loki`, specifies the Loki client. The default values match the Loki install paths mentioned in the Installing the Loki Operator section. If you used another installation method for Loki, specify the appropriate client information for your install.
+<5> The `spec.quickFilters` specification defines filters that show up in the web console. The `Application` filter keys,`src_namespace` and `dst_namespace`, are negated (`!`), so the `Application` filter shows all traffic that _does not_ originate from, or have a destination to, any `openshift-` or `netobserv` namespaces. For more information, see Configuring quick filters below.

modules/network-observability-quickfilter.adoc

@@ -7,6 +7,13 @@
 = Filtering the network traffic
 By default, the Network Traffic page displays the traffic flow data in the cluster based on the default filters configured in the `FlowCollector` instance. You can use the filter options to observe the required data by changing the preset filter.
 
+Query Options::
+You can use *Query Options* to optimize the search results, as listed below:
+
+** *Log Type*: The available options *Conversation* and *Flows* provide the ability to query flows by log type, such as flow log, new conversation, completed conversation, and a heartbeat, which is a periodic record with updates for long conversations. A conversation is an aggregation of flows between the same peers.  
+** *Reporter Node*: Every flow can be reported from both source and destination nodes. For cluster ingress, the flow is reported from the destination node and for cluster egress, the flow is reported from the source node. You can select either *Source* or *Destination*. The option *Both* is disabled for the *Overview* and *Topology* view. The default selected value is *Destination*.
+** *Match filters*: You can determine the relation between different filter parameters selected in the advanced filter. The available options are *Match all* and *Match any*. *Match all*  provides results that match all the values, and *Match any* provides results that match any of the values entered. The default value is *Match all*.
+** *Limit*: The data limit for internal backend queries. Depending upon the matching and the filter settings, the number of traffic flow data is displayed within the specified limit.
 
 Quick filters::
 The default values in *Quick filters* drop-down menu are defined in the `FlowCollector` configuration. You can modify the options from console. 
@@ -19,11 +26,4 @@ You can set the advanced filters by providing the parameter to be filtered and i
 To understand the rules of specifying the text value, click *Learn More*.
 ====
 
-Query Options::
-You can use *Query Options* to optimize the search results, as listed below:
-
-** *Reporter Node*: Every flow can be reported from both source and destination nodes. For cluster ingress, the flow is reported from the destination node and for cluster egress, the flow is reported from the source node. You can select either *Source* or *Destination*. The option *Both* is disabled for the *Overview* and *Topology* view. The default selected value is *Destination*.
-** *Match filters*: You can determine the relation between different filter parameters selected in the advanced filter. The available options are *Match all* and *Match any*. *Match all*  provides results that match all the values, and *Match any* provides results that match any of the values entered. The default value is *Match all*.
-** *Limit*: The data limit for internal backend queries. Depending upon the matching and the filter settings, the number of traffic flow data is displayed within the specified limit.
-
 You can click *Reset default filter* to remove the existing filters, and apply the filter defined in `FlowCollector` configuration.

modules/network-observability-working-with-conversations.adoc

@@ -0,0 +1,47 @@
+// Module included in the following assemblies:
+//
+// network_observability/observing-network-traffic.adoc
+
+:_content-type: PROCEDURE
+[id="network-observability-working-with-conversations_{context}"]
+= Working with conversation tracking
+As an administrator, you can you can group network flows that are part of the same conversation. A conversation is defined as a grouping of peers that are identified by their IP addresses, ports, and protocols, resulting in an unique *Conversation Id*. You can query conversation events in the web console. These events are represented in the web console as follows:
+
+** *Conversation start*: This event happens when a connection is starting or TCP flag intercepted
+** *Conversation tick*: This event happens at each specified interval defined in the `FlowCollector` `spec.processor.conversationHeartbeatInterval` parameter while the connection is active.
+** *Conversation end*: This event happens when the `FlowCollector` `spec.processor.conversationEndTimeout` parameter is reached or  the TCP flag is intercepted.
+** *Flow*: This is the network traffic flow that occurs within the specified interval.
+
+
+.Procedure
+. In the web console, navigate to *Operators* -> *Installed Operators*.
+. Under the *Provided APIs* heading for the *NetObserv Operator*, select *Flow Collector*. 
+. Select *cluster* then select the *YAML* tab.
+. Configure the `FlowCollector` custom resource so that `spec.processor.logTypes`, `conversationEndTimeout`, and `conversationHeartbeatInterval` parameters are set according to your observation needs. A sample configuration is as follows:
++
+[id="network-observability-flowcollector-configuring-conversations_{context}"]
+.Configure `FlowCollector` for conversation tracking
+[source, yaml]
+----
+apiVersion: flows.netobserv.io/v1alpha1
+kind: FlowCollector
+metadata:
+  name: cluster
+spec:
+ processor:
+  conversationEndTimeout: 10s                  <1>
+  logTypes: FLOWS                              <2>
+  conversationHeartbeatInterval: 30s           <3>
+----
+<1> The *Conversation end* event represents the point when the `conversationEndTimeout` is reached or the TCP flag is intercepted.
+<2> When `logTypes` is set to `FLOWS`, only the *Flow* event is exported. If you set the value to `ALL`, both conversation and flow events are exported and visible in the *Network Traffic* page. To focus only on conversation events, you can specify `CONVERSATIONS` which exports the *Conversation start*, *Conversation tick* and *Conversation end* events; or `ENDED_CONVERSATIONS` exports only the *Conversation end* events. Storage requirements are highest for `ALL` and lowest for `ENDED_CONVERSATIONS`.
+<3> The *Conversation tick* event represents each specified interval defined in the `FlowCollector` `conversationHeartbeatInterval` parameter while the network connection is active.
++
+[NOTE]
+====
+If you update the `logType` option, the flows from the previous selection do not clear from the console plugin. For example, if you initially set `logType` to `CONVERSATIONS` for a span of time until 10 AM and then move to `ENDED_CONVERSATIONS`, the console plugin shows all conversation events before 10 AM and only ended conversations after 10 AM. 
+====
+. Refresh the *Network Traffic* page on the *Traffic flows* tab. Notice there are two new columns, *Event/Type* and *Conversation Id*. All the *Event/Type* fields are `Flow` when *Flow* is the selected query option.
+. Select *Query Options* and choose the *Log Type*, *Conversation*. Now the *Event/Type* shows all of the desired conversation events.
+. Next you can filter on a specific conversation ID or switch between the *Conversation* and *Flow* log type options from the side panel.
+

networking/network_observability/configuring-operator.adoc

@@ -9,6 +9,11 @@ You can update the Flow Collector API resource to configure the Network Observab
 
 
 include::modules/network-observability-flowcollector-view.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+For more information about conversation tracking, see xref:../../networking/network_observability/observing-network-traffic.adoc#network-observability-working-with-conversations_nw-observe-network-traffic[Working with conversations].
+
 include::modules/network-observability-flowcollector-kafka-config.adoc[leveloffset=+1]
 include::modules/network-observability-configuring-FLP-sampling.adoc[leveloffset=+1]
 include::modules/network-observability-configuring-quickfilters-flowcollector.adoc[leveloffset=+1]

networking/network_observability/observing-network-traffic.adoc

@@ -17,6 +17,7 @@ include::modules/network-observability-configuring-options-overview.adoc[levelof
 include::modules/network-observability-trafficflow.adoc[leveloffset=+1]
 include::modules/network-observability-working-with-trafficflow.adoc[leveloffset=+2]
 include::modules/network-observability-configuring-options-trafficflow.adoc[leveloffset=+2]
+include::modules/network-observability-working-with-conversations.adoc[leveloffset=+2]
 
 //Topology
 include::modules/network-observability-topology.adoc[leveloffset=+1]