Merge pull request #39615 from mikemckiernan/feat-must-gath-tcpdump

OSDOCS-2171: run tcpdump from must gather

Author: mikemckiernan

modules/support-collecting-host-network-trace.adoc

@@ -0,0 +1,73 @@
+// Module included in the following assemblies:
+//
+// * support/gathering-cluster-data.adoc
+
+[id="support-collecting-host-network-trace_{context}"]
+= Collecting a host network trace
+
+Sometimes, troubleshooting a network-related issue is simplified by tracing network communication and capturing packets on multiple nodes at the same time.
+
+You can use a combination of the `oc adm must-gather` command and the `registry.redhat.io/openshift4/network-tools-rhel8` container image to gather packet captures from nodes.
+Analyzing packet captures can help you troubleshoot network communication issues.
+
+The `oc adm must-gather` command is used to run the `tcpdump` command in pods on specific nodes.
+The `tcpdump` command records the packet captures in the pods.
+When the `tcpdump` command exits, the `oc adm must-gather` command transfers the files with the packet captures from the pods to your client machine.
+
+[TIP]
+====
+The sample command in the following procedure demonstrates performing a packet capture with the `tcpdump` command.
+However, you can run any command in the container image that is specified in the `--image` argument to gather troubleshooting information from multiple nodes at the same time.
+====
+
+.Prerequisites
+
+* You have access to the cluster as a user with the `cluster-admin` role.
+
+* You have installed the OpenShift CLI (`oc`).
+
+.Procedure
+
+. Run a packet capture from the host network on some nodes by running the following command:
++
+[source,terminal]
+----
+$ oc adm must-gather \
+    --dest-dir /tmp/captures \  <.>
+    --source-dir '/tmp/tcpdump/' \  <.>
+    --image registry.redhat.io/openshift4/network-tools-rhel8:latest \  <.>
+    --node-selector 'node-role.kubernetes.io/worker' \  <.>
+    --host-network=true \  <.>
+    --timeout 30s \  <.>
+    -- \
+    tcpdump -i any \  <.>
+    -w /tmp/tcpdump/%Y-%m-%dT%H:%M:%S.pcap -W 1 -G 300
+----
+<.> The `--dest-dir` argument specifies that `oc adm must-gather` stores the packet captures in directories that are relative to `/tmp/captures` on the client machine. You can specify any writable directory.
+<.> When `tcpdump` is run in the debug pod that `oc adm must-gather` starts, the `--source-dir` argument specifies that the packet captures are temporarily stored in the `/tmp/tcpdump` directory on the pod.
+<.> The `--image` argument specifies a container image that includes the `tcpdump` command.
+<.> The `--node-selector` argument and example value specifies to perform the packet captures on the worker nodes. As an alternative, you can specify the `--node-name` argument instead to run the packet capture on a single node. If you omit both the `--node-selector` and the `--node-name` argument, the packet captures are performed on all nodes.
+<.> The `--host-network=true` argument is required so that the packet captures are performed on the network interfaces of the node.
+<.> The `--timeout` argument and value specify to run the debug pod for 30 seconds. If you do not specify the `--timeout` argument and a duration, the debug pod runs for 10 minutes.
+<.> The `-i any` argument for the `tcpdump` command specifies to capture packets on all network interfaces. As an alternative, you can specify a network interface name.
+
+. Perform the action, such as accessing a web application, that triggers the network communication issue while the network trace captures packets.
+
+. Review the packet capture files that `oc adm must-gather` transferred from the pods to your client machine:
++
+[source,text]
+----
+tmp/captures
+├── event-filter.html
+├── ip-10-0-192-217-ec2-internal  <1>
+│   └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca...
+│       └── 2022-01-13T19:31:31.pcap
+├── ip-10-0-201-178-ec2-internal  <1>
+│   └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca...
+│       └── 2022-01-13T19:31:30.pcap
+├── ip-...
+└── timestamp
+----
++
+<1> The packet captures are stored in directories that identify the hostname, container, and file name.
+If you did not specify the `--node-selector` argument, then the directory level for the hostname is not present.

modules/support-network-trace-methods.adoc

@@ -0,0 +1,34 @@
+// Module included in the following assemblies:
+//
+// * support/gathering-cluster-data.adoc
+
+[id="support-network-trace-methods_{context}"]
+= Network trace methods
+
+Collecting network traces, in the form of packet capture records, can assist Red Hat Support with troubleshooting network issues.
+
+{product-title} supports two ways of performing a network trace.
+Review the following table and choose the method that meets your needs.
+
+.Supported methods of collecting a network trace
+[cols="1,4a",options="header"]
+|===
+
+|Method
+|Benefits and capabilities
+
+|Collecting a host network trace
+|You perform a packet capture for a duration that you specify on one or more nodes at the same time.
+The packet capture files are transferred from nodes to the client machine when the specified duration is met.
+
+You can troubleshoot why a specific action triggers network communication issues. Run the packet capture, perform the action that triggers the issue, and use the logs to diagnose the issue.
+
+|Collecting a network trace from an {product-title} node or container
+|You perform a packet capture on one node or one container.
+You run the `tcpdump` command interactively, so you can control the duration of the packet capture.
+
+You can start the packet capture manually, trigger the network communication issue, and then stop the packet capture manually.
+
+This method uses the `cat` command and shell redirection to copy the packet capture data from the node or container to the client machine.
+
+|===

support/gathering-cluster-data.adoc

@@ -41,6 +41,12 @@ include::modules/querying-bootstrap-node-journal-logs.adoc[leveloffset=+1]
 // Querying cluster node journal logs
 include::modules/querying-cluster-node-journal-logs.adoc[leveloffset=+1]
 
+// Network trace methods
+include::modules/support-network-trace-methods.adoc[leveloffset=+1]
+
+// Collecting a host network trace
+include::modules/support-collecting-host-network-trace.adoc[leveloffset=+1]
+
 // Collecting a network trace from an {product-title} node or container
 include::modules/support-collecting-network-trace.adoc[leveloffset=+1]