Skip to content

Commit bba7c5b

Browse files
snarayan-redhatsnarayan-redhat
authored
Merge pull request #53713 from xenolinux/troubleshooting-no
OSDOCS#4604: Document troubleshooting Network Observability
2 parents a550d15 + b3d9805 commit bba7c5b

6 files changed

+212
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1329,6 +1329,8 @@ Topics:
13291329
File: observing-network-traffic
13301330
- Name: API reference
13311331
File: flowcollector-api
1332+
- Name: Troubleshooting Network Observability
1333+
File: troubleshooting-network-observability
13321334
---
13331335
Name: Storage
13341336
Dir: storage

‎modules/troubleshooting-network-observability-after-installation.adoc

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/network_observability/troubleshooting-network-observability.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="configure-network-traffic-console_{context}"]
7+
= Configuring network traffic menu entry in the {product-title} console
8+
9+
Manually configure the network traffic menu entry in the {product-title} console when the network traffic menu entry is not listed in *Observe* menu in the {product-title} console.
10+
11+
.Prerequisites
12+
13+
* You have installed {product-title} version 4.10 or newer.
14+
15+
.Procedure
16+
17+
. Check if the `spec.consolePlugin.register` field is set to `true` by running the following command:
18+
+
19+
[source,terminal]
20+
----
21+
$ oc -n netobserv get flowcollector cluster -o yaml
22+
----
23+
+
24+
.Example output
25+
----
26+
apiVersion: flows.netobserv.io/v1alpha1
27+
kind: FlowCollector
28+
metadata:
29+
name: cluster
30+
spec:
31+
consolePlugin:
32+
register: false
33+
----
34+
35+
. Optional: Add the `netobserv-plugin` plugin by manually editing the Console Operator config:
36+
+
37+
[source,terminal]
38+
----
39+
$ oc edit console.operator.openshift.io cluster
40+
----
41+
+
42+
.Example output
43+
----
44+
...
45+
spec:
46+
plugins:
47+
- netobserv-plugin
48+
...
49+
----
50+
51+
. Optional: Set the `spec.consolePlugin.register` field to `true` by running the following command:
52+
+
53+
[source,terminal]
54+
----
55+
$ oc -n netobserv edit flowcollector cluster -o yaml
56+
----
57+
+
58+
.Example output
59+
----
60+
apiVersion: flows.netobserv.io/v1alpha1
61+
kind: FlowCollector
62+
metadata:
63+
name: cluster
64+
spec:
65+
consolePlugin:
66+
register: true
67+
----
68+
69+
. Ensure the status of console pods is `running` by running the following command:
70+
+
71+
[source,terminal]
72+
----
73+
$ oc get pods -n openshift-console -l app=console
74+
----
75+
76+
. Restart the console pods by running the following command:
77+
+
78+
[source,terminal]
79+
----
80+
$ oc delete pods -n openshift-console -l app=console
81+
----
82+
83+
. Clear your browser cache and history.
84+
85+
. Check the status of Network Observability plugin pods by running the following command:
86+
+
87+
[source,terminal]
88+
----
89+
$ oc get pods -n netobserv -l app=netobserv-plugin
90+
----
91+
+
92+
.Example output
93+
----
94+
NAME READY STATUS RESTARTS AGE
95+
netobserv-plugin-68c7bbb9bb-b69q6 1/1 Running 0 21s
96+
----
97+
98+
. Check the logs of the Network Observability plugin pods by running the following command:
99+
+
100+
[source,terminal]
101+
----
102+
$ oc logs -n netobserv -l app=netobserv-plugin
103+
----
104+
+
105+
.Example output
106+
[source,terminal]
107+
----
108+
time="2022-12-13T12:06:49Z" level=info msg="Starting netobserv-console-plugin [build version: , build date: 2022-10-21 15:15] at log level info" module=main
109+
time="2022-12-13T12:06:49Z" level=info msg="listening on https://:9001" module=server
110+
----

‎modules/troubleshooting-network-observability-controller-manager-pod-out-of-memory.adoc

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/network_observability/troubleshooting-network-observability.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="controller-manager-pod-runs-out-of-memory_{context}"]
7+
= Network Observability controller manager pod runs out of memory
8+
9+
You can increase memory limits for the Network Observability operator by patching the Cluster Service Version (CSV), where Network Observability controller manager pod runs out of memory.
10+
11+
.Procedure
12+
13+
. Run the following command to patch the CSV:
14+
+
15+
[source,terminal]
16+
----
17+
$ oc -n netobserv patch csv network-observability-operator.v1.0.0 --type='json' -p='[{"op": "replace", "path":"/spec/install/spec/deployments/0/spec/template/spec/containers/0/resources/limits/memory", value: "1Gi"}]'
18+
----
19+
+
20+
.Example output
21+
----
22+
clusterserviceversion.operators.coreos.com/network-observability-operator.v1.0.0 patched
23+
----
24+
25+
. Run the following command to view the updated CSV:
26+
+
27+
[source,terminal]
28+
----
29+
$ oc -n netobserv get csv network-observability-operator.v1.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].resources.limits.memory}'
30+
1Gi
31+
----

‎modules/troubleshooting-network-observability-flowlogs-pipeline-kafka.adoc

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/network_observability/troubleshooting-network-observability.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="configure-network-traffic-flowlogs-pipeline-kafka_{context}"]
7+
= Flowlogs-Pipeline does not consume network flows after installing Kafka
8+
9+
If you deployed the flow collector first with `deploymentModel: KAFKA` and then deployed Kafka, the flow collector might not connect correctly to Kafka. Manually restart the flow-pipeline pods where Flowlogs-pipeline does not consume network flows from Kafka.
10+
11+
.Procedure
12+
13+
. Delete the flow-pipeline pods to restart them by running the following command:
14+
+
15+
[source,terminal]
16+
----
17+
$ oc delete pods -n netobserv -l app=flowlogs-pipeline-transformer
18+
----

‎modules/troubleshooting-network-observability-network-flow.adoc

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/network_observability/troubleshooting-network-observability.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="configure-network-traffic-interfaces_{context}"]
7+
= Failing to see network flows from both `br-int` and `br-ex` interfaces
8+
9+
br-ex` and `br-int` are virtual bridge devices operated at OSI layer 2. The eBPF agent works at the IP and TCP levels, layers 3 and 4 respectively. You can expect that the eBPF agent captures the network traffic passing through `br-ex` and `br-int`, when the network traffic is processed by other interfaces such as physical host or virtual pod interfaces. If you restrict the eBPF agent network interfaces to attach only to `br-ex` and `br-int`, you do not see any network flow.
10+
11+
Manually remove the part in the `interfaces` or `excludeInterfaces` that restricts the network interfaces to `br-int` and `br-ex`.
12+
13+
.Procedure
14+
15+
. Remove the `interfaces: [ 'br-int', 'br-ex' ]` field. This allows the agent to fetch information from all the interfaces. Alternatively, you can specify the Layer-3 interface for example, `eth0`. Run the following command:
16+
+
17+
[source,terminal]
18+
----
19+
$ oc edit -n netobserv flowcollector.yaml -o yaml
20+
----
21+
+
22+
.Example output
23+
----
24+
apiVersion: flows.netobserv.io/v1alpha1
25+
kind: FlowCollector
26+
metadata:
27+
name: cluster
28+
spec:
29+
agent:
30+
type: EBPF
31+
ebpf:
32+
interfaces: [ 'br-int', 'br-ex' ] <1>
33+
----
34+
<1> Specifies the network interfaces.

‎networking/network_observability/troubleshooting-network-observability.adoc

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
:_content-type: ASSEMBLY
2+
[id="installing-troubleshooting"]
3+
= Troubleshooting Network Observability
4+
include::_attributes/common-attributes.adoc[]
5+
:context: network-observability-troubleshooting
6+
7+
toc::[]
8+
9+
To assist in troubleshooting Network Observability issues, you can perform some troubleshooting actions.
10+
11+
include::modules/troubleshooting-network-observability-after-installation.adoc[leveloffset=+1]
12+
13+
include::modules/troubleshooting-network-observability-flowlogs-pipeline-kafka.adoc[leveloffset=+1]
14+
15+
include::modules/troubleshooting-network-observability-network-flow.adoc[leveloffset=+1]
16+
17+
include::modules/troubleshooting-network-observability-controller-manager-pod-out-of-memory.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)