[ZT] Tunnel diag logs

src/content/changelogs/tunnel.yaml

@@ -5,6 +5,10 @@ productLink: "/cloudflare-one/connections/connect-networks/"
 productArea: Cloudflare One
 productAreaLink: /cloudflare-one/changelog/
 entries:
+  - publish_date: "2024-12-19"
+    title: Tunnel diagnostic logs
+    description: |-
+      The latest `cloudflared` build [2024.12.2](https://github.com/cloudflare/cloudflared/releases/tag/2024.12.2) introduces the ability to collect all the diagnostic logs needed to troubleshoot a `cloudflared` instance. For more information, refer to [Diagnostic logs](/cloudflare-one/connections/connect-networks/troubleshoot-tunnels/diag-logs/).
   - publish_date: "2024-10-17"
     title: Simplifed WARP Connector deployment
     description: |-

src/content/docs/cloudflare-one/connections/connect-devices/warp/troubleshooting/warp-logs.mdx

@@ -1,8 +1,9 @@
 ---
 pcx_content_type: reference
-title: Diagnostic logs
+title: WARP diagnostic logs
 sidebar:
   order: 7
+	label: Diagnostic logs
 ---
 
 import { TabItem, Tabs } from "~/components";

src/content/docs/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters.mdx

@@ -75,15 +75,15 @@ Saves application log to this file. Mainly useful for reporting issues. For more
 | ---------------------------------------------------------- | ------- | -------------------- |
 | `cloudflared tunnel --loglevel <VALUE> run <UUID or NAME>` | `info`  | `TUNNEL_LOGLEVEL`    |
 
-Specifies the verbosity of logging. The default `info` level does not produce much output, but you may wish to use the `warn` level in production. Available values are: `debug`, `info`, `warn`, `error`, `fatal`.
+Specifies the verbosity of logging for the local `cloudflared` instance. Available values are `debug`, `info` (default), `warn`, `error`, and `fatal`. At the `debug` level, `cloudflared` will log and display the request URL, method, protocol, content length, as well as all request and response headers. However, note that this can expose sensitive information in your logs.
 
 ## `metrics`
 
 | Syntax                                                      | Default      | Environment Variable |
 | ----------------------------------------------------------- | ------------ | -------------------- |
-| `cloudflared tunnel --metrics <IP:PORT> run <UUID or NAME>` | `localhost:` | `TUNNEL_METRICS`     |
+| `cloudflared tunnel --metrics <IP:PORT> run <UUID or NAME>` | Refer to [Tunnel metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/)| `TUNNEL_METRICS`     |
 
-Exposes a Prometheus endpoint on the specified IP address/port, which you can then query for [usage metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/).
+Exposes a Prometheus endpoint on the specified IP address and port, which you can then query for [usage metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/).
 
 ## `no-autoupdate`
 

src/content/docs/cloudflare-one/connections/connect-networks/monitor-tunnels/logs.mdx

@@ -1,58 +1,42 @@
 ---
 pcx_content_type: reference
-title: Logs
+title: Log streams
 sidebar:
   order: 1
 head:
   - tag: title
-    content: Tunnel logs
+    content: Tunnel log streams
 ---
 
 Tunnel logs record all activity between a `cloudflared` instance and Cloudflare's global network, as well as all activity between `cloudflared` and your origin server. These logs allow you to investigate connectivity or performance issues with a Cloudflare Tunnel. You can configure your server to store persistent logs, or you can stream real-time logs from any client machine.
 
 ## View logs on the server
 
-If you have access to the origin server, you can enable logging when you start the tunnel:
+If you have access to the origin server, you can use the [`--loglevel` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#loglevel) to enable logging when you start the tunnel. By default, `cloudflared` prints logs to stdout and does not store logs on the server. You can optionally use the [`--logfile` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#logfile) to write your logs to a file.
 
-```sh
-cloudflared tunnel --loglevel debug run <UUID>
-```
-
-The [`--loglevel` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#loglevel) indicates the logging level for the local `cloudflared` instance, which can be one of {`debug`, `info`, `warn`, `error`, `fatal`} (default: `info`). At the `debug` level, `cloudflared` will log and display the request URL, method, protocol, content length, as well as all request and response headers. However, note that this can expose sensitive information in your logs.
-
-### Write logs to file
-
-By default, `cloudflared` prints logs to stdout and does not store logs on the server. You can use the [`--logfile` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#logfile) to save your logs to a file:
+To enable logs for a locally-managed tunnel:
 
 ```sh
-cloudflared tunnel --logfile mytunnel.log run <UUID>
+cloudflared tunnel --loglevel debug --logfile cloudflared.log run <UUID>
 ```
 
+To enable logs for a remotely-managed tunnel, add `--loglevel debug` and `--logfile <PATH>` to your system service as shown in [Add tunnel run parameters](/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#add-tunnel-run-parameters).
+
 ## View logs on your local machine
 
-You can view real-time logs for a Cloudflare Tunnel via the dashboard or from any machine that has `cloudflared` installed. With remote log streams, you do not need to SSH into the server that is running the tunnel.
+You can view real-time logs for a Cloudflare Tunnel via the dashboard or from any machine that has `cloudflared` installed. With remote log streams, you do not need to SSH into the server that is running the tunnel. To get remote logs, the tunnel must be active and able to receive requests.
 
 ### Dashboard
 
-#### Prerequisites
+Dashboard log streams are only available for remotely-managed tunnels. To view logs from the dashboard:
 
-- `cloudflared` version 2023.5.1 or higher is installed on the origin server.
-- The tunnel is active and able to receive requests.
-
-#### View logs
-
-1. In [Zero Trust](https://one.dash.cloudflare.com/), go to **Networks** > **Tunnels** and select your tunnel.
+1. In [Zero Trust](https://one.dash.cloudflare.com/), go to **Networks** > **Tunnels** and select a remotely-managed tunnel.
 2. In the sidebar, select the **Connector ID** for the `cloudflared` instance you want to view.
 3. Select **Begin log stream**.
 
 ### CLI
 
-The `cloudflared` daemon can stream logs from any tunnel in your account to the local command line.
-
-#### Prerequisites
-
-- `cloudflared` version 2023.5.1 or higher is installed on both your local machine and the origin server.
-- The tunnel is active and able to receive requests.
+The `cloudflared` daemon can stream logs from any tunnel in your account to the local command line. `cloudflared` must be installed on both your local machine and the origin server.
 
 #### View logs
 

src/content/docs/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics.mdx

@@ -6,27 +6,35 @@ sidebar:
 head:
   - tag: title
     content: Tunnel metrics
-tableOfContents: false
 ---
 
-Tunnel metrics show a Cloudflare Tunnel's throughput and resource usage over time. When you run a tunnel, you can configure `cloudflared` to spin up a Prometheus metrics endpoint — an HTTP server that exposes metrics in [Prometheus](https://prometheus.io/docs/introduction/overview/) format. You can then use the Prometheus toolkit on a remote machine to scrape metrics data from the `cloudflared` server.
+Tunnel metrics show a Cloudflare Tunnel's throughput and resource usage over time. When you run a tunnel, `cloudflared` will spin up a Prometheus metrics endpoint — an HTTP server that exposes metrics in [Prometheus](https://prometheus.io/docs/introduction/overview/) format. You can use the Prometheus toolkit on a remote machine to scrape metrics data from the `cloudflared` server.
 
-## Start the metrics server
+## Default metrics server address
 
-Perform these steps on the `cloudflared` server.
+In non-containerized environments, `cloudflared` starts the metrics server on `127.0.0.1:<PORT>/metrics`, where `<PORT>` is the first available port in the range `20241` to `20245`. In case of all ports being unavailable then the fallback is to bind to a random port. In containerized environments such as Docker and Kubernetes, the default address is `0.0.0.0:<PORT>/metrics`.
 
-1. Use the [--metrics](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#metrics) flag to create a metrics endpoint at the specified IP address and port. Here is an example command for a locally-managed tunnel:
+To determine the default port being used by a `cloudflared` instance, you can check your [Tunnel logs](/cloudflare-one/connections/connect-networks/monitor-tunnels/logs/) around the time when the tunnel started. For example:
+
+```text
+2024-12-19T21:17:58Z INF Starting metrics server on 127.0.0.1:20241/metrics
+```
+
+## Configure the metrics server address
+
+To serve metrics on a custom IP address and port, perform these steps on the `cloudflared` host:
+
+1. Run the tunnel using the [--metrics](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#metrics) flag. Here is an example command for a locally-managed tunnel:
 
    ```sh
    cloudflared tunnel --metrics 127.0.0.1:60123 run my-tunnel
    ```
 
    To learn how to add the `--metrics` flag to a remotely-managed tunnel, refer to [Configure a remotely-managed tunnel](/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#add-tunnel-run-parameters).
 
-:::note
-
-If you plan to fetch metrics from another machine on the local network, replace `127.0.0.1` with the internal IP of the `cloudflared` server (for example, `198.168.x.x`). To serve metrics on all available network interfaces, use `0.0.0.0`.
-:::
+		:::note
+		If you plan to fetch metrics from another machine on the local network, replace `127.0.0.1` with the internal IP of the `cloudflared` server (for example, `198.168.x.x`). To serve metrics on all available network interfaces, use `0.0.0.0`.
+		:::
 
 2. Verify that the metrics server is running by going to `http://localhost:60123/metrics`. This will only work if you configured a localhost IP (`127.0.0.1` or `0.0.0.0`).
 

src/content/docs/cloudflare-one/connections/connect-networks/troubleshoot-tunnels/diag-logs.mdx

@@ -0,0 +1,150 @@
+---
+pcx_content_type: how-to
+title: Tunnel diagnostic logs
+sidebar:
+  order: 1
+  label: Diagnostic logs
+---
+
+import {Details} from "~/components";
+
+Cloudflare Tunnel generates a set of diagnostic logs that can be used to troubleshoot issues with `cloudflared`. A diagnostic report collects data from a single instance of `cloudflared` running on the local machine.
+
+## Get diagnostic logs
+
+The steps for getting diagnostic logs depend on your `cloudflared` deployment environment.
+
+### Prerequisites
+
+- `cloudflared` version 2024.12.2 or later installed on the host
+
+### Host environment
+
+These instructions apply to remotely-managed and locally-managed tunnels running directly on the host machine.
+
+1. (Linux only) To include network diagnostics in the logs, allow the `cloudflared` user to create RAW and PACKET sockets without root permissions:
+
+		```sh
+		sudo setcap cap_net_raw+ep /usr/bin/traceroute && sudo setcap cap_net_raw+ep /usr/bin/traceroute
+		```
+
+		If you do not set `cap_net_raw`, then traceroute data will be unavailable.
+
+2. Get diagnostic logs:
+
+		```sh
+		cloudflared tunnel diag
+		```
+
+		If multiple instances of `cloudflared` are running on the same host, specify the [metrics server IP and port](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#check-the-metrics-server-address) for the instance you want to diagnose. For example:
+
+		```sh
+		cloudflared tunnel diag --metrics 127.0.0.1:20241
+		```
+
+This command will output the status of each diagnostic task and place a `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` file in your working directory.
+
+### Docker
+
+`cloudflared` reads diagnostic data from the [tunnel metrics server](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/). To get diagnostic logs, the metrics server must be exposed from the Docker container and reachable from the host machine.
+
+1. Determine the [metrics server port](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#default-metrics-server-address) for the `cloudflared` instance running in Docker.
+
+2. Ensure the container is deployed with port forwarding enabled. The diagnostic feature will request information from the Docker instance using local port `20241`, therefore you should forward port `20241` to the container port obtained in Step 1:
+
+		```sh
+		docker run -d -p 20241:<metrics_port> docker.io/cloudflare/cloudflared tunnel ...
+		```
+
+3. Verify that you can reach the metrics server address from the Docker host environment:
+
+		```curl
+		curl localhost:20241/diag/tunnel
+		```
+
+		This command should return a JSON:
+		```json
+		{
+			"tunnelID": "ef96b330-a7f5-4bce-a00e-827ce5be077f",
+			"connectorID": "d236670a-9f74-422f-adf1-030f5c5f0523",
+			"connections": [
+				{ "isConnected": true, "protocol": 1, "edgeAddress": "198.41.192.167"},
+				{"isConnected": true, "protocol": 1, "edgeAddress": "198.41.200.113", "index": 1},
+				{"isConnected": true, "protocol": 1, "edgeAddress": "198.41.192.47", "index": 2},
+				{"isConnected": true, "protocol": 1, "edgeAddress": "198.41.200.73", "index": 3}
+			],
+			"icmp_sources": ["192.168.1.243", "fe80::c59:bd4a:e815:ed6"]
+		}
+		```
+
+4. Run the diagnostic using the Docker container ID:
+
+		```sh
+		cloudflared tunnel diag --diag-container-id=<containerID>
+		```
+
+		Alternatively, you can specify the container's name instead of its ID:
+		```sh
+		cloudflared tunnel diag --diag-container-id=<containerName>
+		```
+
+		Running the diagnostic command with the container ID allows `cloudflared` to collect information from the Docker environment such as logs and container details.
+
+This command will output the status of each diagnostic task and place a `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` file in your working directory.
+
+### Kubernetes
+
+The diagnostic feature will request data from the [tunnel metrics server](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/) using ports `20241` to `20245`. You will need to use port forwarding to allow the local `cloudflared` instance to connect to the metrics server on one of these ports.
+
+1. Determine the tunnel's [metrics server port](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#default-metrics-server-address).
+
+2. Enable port forwarding:
+
+		```sh
+		kubectl port-forward <pod> <diagnostic_port>:<metrics_port>
+		```
+
+		- `<pod>`: Name of the pod where the tunnel is running
+		- `<diagnostic_port>` is any local port in the range `20241` to `20245`.
+		- `<metrics_port>` is the Kubernetes pod port for the `cloudflared` instance you want to diagnose (obtained in Step 1).
+
+		For example, if you set the metrics server address to `0.0.0.0:12345`:
+
+		```sh
+		kubectl port-forward cloudflared-6d4897585b-r8kfz 20244:12345
+		```
+		Connections made to local port `20244` are forwarded to port `12345` of the pod that is running the tunnel.
+
+3. Run the diagnostic:
+
+		```sh
+		cloudflared tunnel diag --diag-pod-id=<podID>
+		```
+
+		If the pod has multiple applications/services running and `cloudflared` is not the first in the pod, you must specify either the container ID or name:
+
+		```sh
+		cloudflared tunnel diag --diag-pod-id=<podID> --diag-container-id=<containerName>
+		```
+
+This command will output the status of each diagnostic task and place a `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` file in your working directory.
+
+## cloudflared-diag files
+
+The `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` archive contains the files listed below. The data in a file either applies to the `cloudflared` instance being diagnosed (`diagnosee`) or the instance that triggered the diagnosis (`diagnoser`). For example, if your tunnel is running in a Docker container, the diagnosee is the Docker instance and the diagnoser is the host instance.
+
+| File name  | Description | Instance |
+| -| - | - |
+| `cli-configuration.json`| [Tunnel run parameters](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/) used when starting the tunnel | diagnosee|
+| `cloudflared_logs.txt` | [Tunnel log file](/cloudflare-one/connections/connect-networks/monitor-tunnels/logs/)[^1]  | diagnosee|
+| `configuration.json` | Tunnel configuration parameters | diagnosee|
+| `goroutine.pprof` | goroutine profile made available by `pprof` | diagnosee|
+| `heap.pprof` | 	heap profile made available by `pprof`| diagnosee|
+| `metrics.txt` | Snapshot of [Tunnel metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#available-metrics) at the time of diagnosis | diagnosee|
+| `network.txt` | JSON traceroutes to Cloudflare's global network using IPv4 and IPv6 | diagnoser|
+| `raw-network.txt` | Raw traceroutes to Cloudflare's global network using IPv4 and IPv6 | diagnoser|
+| `systeminformation.json` | Operating system information and resource usage | diagnosee|
+| `task-result.json` | Result of each diagnostic task | diagnoser |
+| `tunnelstate.json` | Tunnel connections at the time of diagnosis| diagnosee|
+
+[^1]: If the log file is blank, you may need to [set `--loglevel` to `debug`](/cloudflare-one/connections/connect-networks/monitor-tunnels/logs/#view-logs-on-the-server) when you start the tunnel. The `--loglevel` parameter is only required if you ran the tunnel from the CLI using a `cloudflared tunnel run` command. It is not necessary if the tunnel runs as a Linux/macOS service or runs in Docker/Kubernetes.