Skip to content

Commit a986ea8

Browse files
ranbelDevinCarr
ranbel
and
DevinCarr
authored
[ZT] Tunnel diag logs (#18704)
* update existing logs page * add metrics info * update title * instructions for getting logs * add changelog entry * clarify port forwarding * edit changelog * log files * update Contact Support * tweak wording * update default address * update Docker instructions * Apply suggestions from code review Co-authored-by: Devin <[email protected]> * apply review deedback * update titles * cap_net_raw optional --------- Co-authored-by: Devin <[email protected]>
1 parent 881d0cf commit a986ea8

File tree

7 files changed

+192
-63
lines changed

7 files changed

+192
-63
lines changed

‎src/content/changelogs/tunnel.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ productLink: "/cloudflare-one/connections/connect-networks/"
55
productArea: Cloudflare One
66
productAreaLink: /cloudflare-one/changelog/
77
entries:
8+
- publish_date: "2024-12-19"
9+
title: Tunnel diagnostic logs
10+
description: |-
11+
The latest `cloudflared` build [2024.12.2](https://github.com/cloudflare/cloudflared/releases/tag/2024.12.2) introduces the ability to collect all the diagnostic logs needed to troubleshoot a `cloudflared` instance. For more information, refer to [Diagnostic logs](/cloudflare-one/connections/connect-networks/troubleshoot-tunnels/diag-logs/).
812
- publish_date: "2024-10-17"
913
title: Simplifed WARP Connector deployment
1014
description: |-

‎src/content/docs/cloudflare-one/connections/connect-devices/warp/troubleshooting/warp-logs.mdx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
22
pcx_content_type: reference
3-
title: Diagnostic logs
3+
title: WARP diagnostic logs
44
sidebar:
55
order: 7
6+
label: Diagnostic logs
67
---
78

89
import { TabItem, Tabs } from "~/components";

‎src/content/docs/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters.mdx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,15 +75,15 @@ Saves application log to this file. Mainly useful for reporting issues. For more
7575
| ---------------------------------------------------------- | ------- | -------------------- |
7676
| `cloudflared tunnel --loglevel <VALUE> run <UUID or NAME>` | `info` | `TUNNEL_LOGLEVEL` |
7777

78-
Specifies the verbosity of logging. The default `info` level does not produce much output, but you may wish to use the `warn` level in production. Available values are: `debug`, `info`, `warn`, `error`, `fatal`.
78+
Specifies the verbosity of logging for the local `cloudflared` instance. Available values are `debug`, `info` (default), `warn`, `error`, and `fatal`. At the `debug` level, `cloudflared` will log and display the request URL, method, protocol, content length, as well as all request and response headers. However, note that this can expose sensitive information in your logs.
7979

8080
## `metrics`
8181

8282
| Syntax | Default | Environment Variable |
8383
| ----------------------------------------------------------- | ------------ | -------------------- |
84-
| `cloudflared tunnel --metrics <IP:PORT> run <UUID or NAME>` | `localhost:` | `TUNNEL_METRICS` |
84+
| `cloudflared tunnel --metrics <IP:PORT> run <UUID or NAME>` | Refer to [Tunnel metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/)| `TUNNEL_METRICS` |
8585

86-
Exposes a Prometheus endpoint on the specified IP address/port, which you can then query for [usage metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/).
86+
Exposes a Prometheus endpoint on the specified IP address and port, which you can then query for [usage metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/).
8787

8888
## `no-autoupdate`
8989

‎src/content/docs/cloudflare-one/connections/connect-networks/monitor-tunnels/logs.mdx‎

Lines changed: 11 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,42 @@
11
---
22
pcx_content_type: reference
3-
title: Logs
3+
title: Log streams
44
sidebar:
55
order: 1
66
head:
77
- tag: title
8-
content: Tunnel logs
8+
content: Tunnel log streams
99
---
1010

1111
Tunnel logs record all activity between a `cloudflared` instance and Cloudflare's global network, as well as all activity between `cloudflared` and your origin server. These logs allow you to investigate connectivity or performance issues with a Cloudflare Tunnel. You can configure your server to store persistent logs, or you can stream real-time logs from any client machine.
1212

1313
## View logs on the server
1414

15-
If you have access to the origin server, you can enable logging when you start the tunnel:
15+
If you have access to the origin server, you can use the [`--loglevel` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#loglevel) to enable logging when you start the tunnel. By default, `cloudflared` prints logs to stdout and does not store logs on the server. You can optionally use the [`--logfile` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#logfile) to write your logs to a file.
1616

17-
```sh
18-
cloudflared tunnel --loglevel debug run <UUID>
19-
```
20-
21-
The [`--loglevel` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#loglevel) indicates the logging level for the local `cloudflared` instance, which can be one of {`debug`, `info`, `warn`, `error`, `fatal`} (default: `info`). At the `debug` level, `cloudflared` will log and display the request URL, method, protocol, content length, as well as all request and response headers. However, note that this can expose sensitive information in your logs.
22-
23-
### Write logs to file
24-
25-
By default, `cloudflared` prints logs to stdout and does not store logs on the server. You can use the [`--logfile` flag](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#logfile) to save your logs to a file:
17+
To enable logs for a locally-managed tunnel:
2618

2719
```sh
28-
cloudflared tunnel --logfile mytunnel.log run <UUID>
20+
cloudflared tunnel --loglevel debug --logfile cloudflared.log run <UUID>
2921
```
3022

23+
To enable logs for a remotely-managed tunnel, add `--loglevel debug` and `--logfile <PATH>` to your system service as shown in [Add tunnel run parameters](/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#add-tunnel-run-parameters).
24+
3125
## View logs on your local machine
3226

33-
You can view real-time logs for a Cloudflare Tunnel via the dashboard or from any machine that has `cloudflared` installed. With remote log streams, you do not need to SSH into the server that is running the tunnel.
27+
You can view real-time logs for a Cloudflare Tunnel via the dashboard or from any machine that has `cloudflared` installed. With remote log streams, you do not need to SSH into the server that is running the tunnel. To get remote logs, the tunnel must be active and able to receive requests.
3428

3529
### Dashboard
3630

37-
#### Prerequisites
31+
Dashboard log streams are only available for remotely-managed tunnels. To view logs from the dashboard:
3832

39-
- `cloudflared` version 2023.5.1 or higher is installed on the origin server.
40-
- The tunnel is active and able to receive requests.
41-
42-
#### View logs
43-
44-
1. In [Zero Trust](https://one.dash.cloudflare.com/), go to **Networks** > **Tunnels** and select your tunnel.
33+
1. In [Zero Trust](https://one.dash.cloudflare.com/), go to **Networks** > **Tunnels** and select a remotely-managed tunnel.
4534
2. In the sidebar, select the **Connector ID** for the `cloudflared` instance you want to view.
4635
3. Select **Begin log stream**.
4736

4837
### CLI
4938

50-
The `cloudflared` daemon can stream logs from any tunnel in your account to the local command line.
51-
52-
#### Prerequisites
53-
54-
- `cloudflared` version 2023.5.1 or higher is installed on both your local machine and the origin server.
55-
- The tunnel is active and able to receive requests.
39+
The `cloudflared` daemon can stream logs from any tunnel in your account to the local command line. `cloudflared` must be installed on both your local machine and the origin server.
5640

5741
#### View logs
5842

‎src/content/docs/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics.mdx‎

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,27 +6,35 @@ sidebar:
66
head:
77
- tag: title
88
content: Tunnel metrics
9-
tableOfContents: false
109
---
1110

12-
Tunnel metrics show a Cloudflare Tunnel's throughput and resource usage over time. When you run a tunnel, you can configure `cloudflared` to spin up a Prometheus metrics endpoint — an HTTP server that exposes metrics in [Prometheus](https://prometheus.io/docs/introduction/overview/) format. You can then use the Prometheus toolkit on a remote machine to scrape metrics data from the `cloudflared` server.
11+
Tunnel metrics show a Cloudflare Tunnel's throughput and resource usage over time. When you run a tunnel, `cloudflared` will spin up a Prometheus metrics endpoint — an HTTP server that exposes metrics in [Prometheus](https://prometheus.io/docs/introduction/overview/) format. You can use the Prometheus toolkit on a remote machine to scrape metrics data from the `cloudflared` server.
1312

14-
## Start the metrics server
13+
## Default metrics server address
1514

16-
Perform these steps on the `cloudflared` server.
15+
In non-containerized environments, `cloudflared` starts the metrics server on `127.0.0.1:<PORT>/metrics`, where `<PORT>` is the first available port in the range `20241` to `20245`. In case of all ports being unavailable then the fallback is to bind to a random port. In containerized environments such as Docker and Kubernetes, the default address is `0.0.0.0:<PORT>/metrics`.
1716

18-
1. Use the [--metrics](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#metrics) flag to create a metrics endpoint at the specified IP address and port. Here is an example command for a locally-managed tunnel:
17+
To determine the default port being used by a `cloudflared` instance, you can check your [Tunnel logs](/cloudflare-one/connections/connect-networks/monitor-tunnels/logs/) around the time when the tunnel started. For example:
18+
19+
```text
20+
2024-12-19T21:17:58Z INF Starting metrics server on 127.0.0.1:20241/metrics
21+
```
22+
23+
## Configure the metrics server address
24+
25+
To serve metrics on a custom IP address and port, perform these steps on the `cloudflared` host:
26+
27+
1. Run the tunnel using the [--metrics](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/#metrics) flag. Here is an example command for a locally-managed tunnel:
1928

2029
```sh
2130
cloudflared tunnel --metrics 127.0.0.1:60123 run my-tunnel
2231
```
2332

2433
To learn how to add the `--metrics` flag to a remotely-managed tunnel, refer to [Configure a remotely-managed tunnel](/cloudflare-one/connections/connect-networks/configure-tunnels/remote-management/#add-tunnel-run-parameters).
2534

26-
:::note
27-
28-
If you plan to fetch metrics from another machine on the local network, replace `127.0.0.1` with the internal IP of the `cloudflared` server (for example, `198.168.x.x`). To serve metrics on all available network interfaces, use `0.0.0.0`.
29-
:::
35+
:::note
36+
If you plan to fetch metrics from another machine on the local network, replace `127.0.0.1` with the internal IP of the `cloudflared` server (for example, `198.168.x.x`). To serve metrics on all available network interfaces, use `0.0.0.0`.
37+
:::
3038

3139
2. Verify that the metrics server is running by going to `http://localhost:60123/metrics`. This will only work if you configured a localhost IP (`127.0.0.1` or `0.0.0.0`).
3240

‎src/content/docs/cloudflare-one/connections/connect-networks/troubleshoot-tunnels/diag-logs.mdx‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
---
2+
pcx_content_type: how-to
3+
title: Tunnel diagnostic logs
4+
sidebar:
5+
order: 1
6+
label: Diagnostic logs
7+
---
8+
9+
import {Details} from "~/components";
10+
11+
Cloudflare Tunnel generates a set of diagnostic logs that can be used to troubleshoot issues with `cloudflared`. A diagnostic report collects data from a single instance of `cloudflared` running on the local machine.
12+
13+
## Get diagnostic logs
14+
15+
The steps for getting diagnostic logs depend on your `cloudflared` deployment environment.
16+
17+
### Prerequisites
18+
19+
- `cloudflared` version 2024.12.2 or later installed on the host
20+
21+
### Host environment
22+
23+
These instructions apply to remotely-managed and locally-managed tunnels running directly on the host machine.
24+
25+
1. (Linux only) To include network diagnostics in the logs, allow the `cloudflared` user to create RAW and PACKET sockets without root permissions:
26+
27+
```sh
28+
sudo setcap cap_net_raw+ep /usr/bin/traceroute && sudo setcap cap_net_raw+ep /usr/bin/traceroute
29+
```
30+
31+
If you do not set `cap_net_raw`, then traceroute data will be unavailable.
32+
33+
2. Get diagnostic logs:
34+
35+
```sh
36+
cloudflared tunnel diag
37+
```
38+
39+
If multiple instances of `cloudflared` are running on the same host, specify the [metrics server IP and port](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#check-the-metrics-server-address) for the instance you want to diagnose. For example:
40+
41+
```sh
42+
cloudflared tunnel diag --metrics 127.0.0.1:20241
43+
```
44+
45+
This command will output the status of each diagnostic task and place a `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` file in your working directory.
46+
47+
### Docker
48+
49+
`cloudflared` reads diagnostic data from the [tunnel metrics server](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/). To get diagnostic logs, the metrics server must be exposed from the Docker container and reachable from the host machine.
50+
51+
1. Determine the [metrics server port](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#default-metrics-server-address) for the `cloudflared` instance running in Docker.
52+
53+
2. Ensure the container is deployed with port forwarding enabled. The diagnostic feature will request information from the Docker instance using local port `20241`, therefore you should forward port `20241` to the container port obtained in Step 1:
54+
55+
```sh
56+
docker run -d -p 20241:<metrics_port> docker.io/cloudflare/cloudflared tunnel ...
57+
```
58+
59+
3. Verify that you can reach the metrics server address from the Docker host environment:
60+
61+
```curl
62+
curl localhost:20241/diag/tunnel
63+
```
64+
65+
This command should return a JSON:
66+
```json
67+
{
68+
"tunnelID": "ef96b330-a7f5-4bce-a00e-827ce5be077f",
69+
"connectorID": "d236670a-9f74-422f-adf1-030f5c5f0523",
70+
"connections": [
71+
{ "isConnected": true, "protocol": 1, "edgeAddress": "198.41.192.167"},
72+
{"isConnected": true, "protocol": 1, "edgeAddress": "198.41.200.113", "index": 1},
73+
{"isConnected": true, "protocol": 1, "edgeAddress": "198.41.192.47", "index": 2},
74+
{"isConnected": true, "protocol": 1, "edgeAddress": "198.41.200.73", "index": 3}
75+
],
76+
"icmp_sources": ["192.168.1.243", "fe80::c59:bd4a:e815:ed6"]
77+
}
78+
```
79+
80+
4. Run the diagnostic using the Docker container ID:
81+
82+
```sh
83+
cloudflared tunnel diag --diag-container-id=<containerID>
84+
```
85+
86+
Alternatively, you can specify the container's name instead of its ID:
87+
```sh
88+
cloudflared tunnel diag --diag-container-id=<containerName>
89+
```
90+
91+
Running the diagnostic command with the container ID allows `cloudflared` to collect information from the Docker environment such as logs and container details.
92+
93+
This command will output the status of each diagnostic task and place a `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` file in your working directory.
94+
95+
### Kubernetes
96+
97+
The diagnostic feature will request data from the [tunnel metrics server](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/) using ports `20241` to `20245`. You will need to use port forwarding to allow the local `cloudflared` instance to connect to the metrics server on one of these ports.
98+
99+
1. Determine the tunnel's [metrics server port](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#default-metrics-server-address).
100+
101+
2. Enable port forwarding:
102+
103+
```sh
104+
kubectl port-forward <pod> <diagnostic_port>:<metrics_port>
105+
```
106+
107+
- `<pod>`: Name of the pod where the tunnel is running
108+
- `<diagnostic_port>` is any local port in the range `20241` to `20245`.
109+
- `<metrics_port>` is the Kubernetes pod port for the `cloudflared` instance you want to diagnose (obtained in Step 1).
110+
111+
For example, if you set the metrics server address to `0.0.0.0:12345`:
112+
113+
```sh
114+
kubectl port-forward cloudflared-6d4897585b-r8kfz 20244:12345
115+
```
116+
Connections made to local port `20244` are forwarded to port `12345` of the pod that is running the tunnel.
117+
118+
3. Run the diagnostic:
119+
120+
```sh
121+
cloudflared tunnel diag --diag-pod-id=<podID>
122+
```
123+
124+
If the pod has multiple applications/services running and `cloudflared` is not the first in the pod, you must specify either the container ID or name:
125+
126+
```sh
127+
cloudflared tunnel diag --diag-pod-id=<podID> --diag-container-id=<containerName>
128+
```
129+
130+
This command will output the status of each diagnostic task and place a `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` file in your working directory.
131+
132+
## cloudflared-diag files
133+
134+
The `cloudflared-diag-YYYY-MM-DDThh-mm-ss.zip` archive contains the files listed below. The data in a file either applies to the `cloudflared` instance being diagnosed (`diagnosee`) or the instance that triggered the diagnosis (`diagnoser`). For example, if your tunnel is running in a Docker container, the diagnosee is the Docker instance and the diagnoser is the host instance.
135+
136+
| File name | Description | Instance |
137+
| -| - | - |
138+
| `cli-configuration.json`| [Tunnel run parameters](/cloudflare-one/connections/connect-networks/configure-tunnels/tunnel-run-parameters/) used when starting the tunnel | diagnosee|
139+
| `cloudflared_logs.txt` | [Tunnel log file](/cloudflare-one/connections/connect-networks/monitor-tunnels/logs/)[^1] | diagnosee|
140+
| `configuration.json` | Tunnel configuration parameters | diagnosee|
141+
| `goroutine.pprof` | goroutine profile made available by `pprof` | diagnosee|
142+
| `heap.pprof` | heap profile made available by `pprof`| diagnosee|
143+
| `metrics.txt` | Snapshot of [Tunnel metrics](/cloudflare-one/connections/connect-networks/monitor-tunnels/metrics/#available-metrics) at the time of diagnosis | diagnosee|
144+
| `network.txt` | JSON traceroutes to Cloudflare's global network using IPv4 and IPv6 | diagnoser|
145+
| `raw-network.txt` | Raw traceroutes to Cloudflare's global network using IPv4 and IPv6 | diagnoser|
146+
| `systeminformation.json` | Operating system information and resource usage | diagnosee|
147+
| `task-result.json` | Result of each diagnostic task | diagnoser |
148+
| `tunnelstate.json` | Tunnel connections at the time of diagnosis| diagnosee|
149+
150+
[^1]: If the log file is blank, you may need to [set `--loglevel` to `debug`](/cloudflare-one/connections/connect-networks/monitor-tunnels/logs/#view-logs-on-the-server) when you start the tunnel. The `--loglevel` parameter is only required if you ran the tunnel from the CLI using a `cloudflared tunnel run` command. It is not necessary if the tunnel runs as a Linux/macOS service or runs in Docker/Kubernetes.

0 commit comments

Comments
 (0)