Skip to content

Commit ecbbedf

Browse files
nevikashahmikenomitchAbhiPrasadjmorrell-cloudflare
authored
[DRAFT] Workers Tracing / Exporting OTel (#24713)
* WIP PR for tracing * Various tweaks to tracing docs * Update known limitations and spans documentation for tracing - Corrected links in the known limitations section to point to the appropriate security model documentation. - Updated span and attribute references to reflect changes in the Durable Object Storage API, including renaming and correcting links for various methods. * Update links in tracing documentation for accuracy - Corrected the link for the Durable Object Storage SQL API to point to the correct SQLite storage API documentation. - Ensured consistency in the references to OpenTelemetry and observability platforms. * add sentry instructions (#25996) * Enhance OpenTelemetry configuration documentation - Added `head_sampling_rate` and `persist` options for traces and logs in the OpenTelemetry configuration example. - Improved clarity in the tracing documentation regarding default sampling rates and configuration options. * Update Datadog link and improve comments in OpenTelemetry configuration documentation - Updated the Datadog entry to include a direct link to the official documentation. - Enhanced comments for clarity regarding the optional disabling of traces and logs in the Cloudflare dashboard. * Update spans-and-attributes.mdx (#26036) * Update spans-and-attributes.mdx Add explanation for sequence number and remove attributes for durable_object_subrequest that we do not have yet * Update spans-and-attributes.mdx Also remove cold start since we do not have that info yet * Update Sentry and spans-and-attributes documentation for clarity - Clarified the format for the Sentry authentication header in the OpenTelemetry data exporting instructions. - Updated attribute names in spans-and-attributes documentation to include the correct units for CPU and wall time. * Clarify Sentry authentication header format in OpenTelemetry documentation - Updated the formatting of the Sentry authentication header description to ensure proper usage of curly braces around the public key placeholder in the OpenTelemetry data exporting instructions. * Remove spans / attributes we haven't landed yet (#26099) --------- Co-authored-by: Mike Nomitch <[email protected]> Co-authored-by: Abhijeet Prasad <[email protected]> Co-authored-by: Jeremy Morrell (Cloudflare) <[email protected]>
1 parent 60470d1 commit ecbbedf

File tree

17 files changed

+1130
-6
lines changed

17 files changed

+1130
-6
lines changed

‎src/assets/images/workers-observability/axiom-example.png‎

336 KB
Loading

‎src/assets/images/workers-observability/destination-setup.png‎

201 KB
Loading

‎src/assets/images/workers-observability/destinations.png‎

131 KB
Loading

‎src/assets/images/workers-observability/grafana-traces.png‎

145 KB
Loading

‎src/assets/images/workers-observability/honeycomb-example.png‎

156 KB
Loading

‎src/assets/images/workers-observability/sentry-example.png‎

241 KB
Loading

‎src/assets/images/workers-observability/trace-waterfall-example.png‎

132 KB
Loading

‎src/content/docs/workers/observability/exporting-opentelemetry-data/axiom.mdx‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
pcx_content_type: how-to
3+
title: Export to Axiom
4+
sidebar:
5+
order: 3
6+
---
7+
8+
import {WranglerConfig} from "~/components";
9+
10+
Axiom is a serverless log analytics platform that helps you store, search, and analyze massive amounts of data. By exporting your Cloudflare Workers application telemetry to Axiom, you can:
11+
- Store and query logs and traces at scale
12+
- Create dashboards and alerts to monitor your Workers
13+
14+
![Trace view with timing information displayed on a timeline](~/assets/images/workers-observability/axiom-example.png)
15+
16+
This guide will walk you through exporting OpenTelemetry-compliant traces and logs to Axiom from your Cloudflare Worker application
17+
18+
## Prerequisites
19+
20+
Before you begin, ensure you have:
21+
22+
- An active [Axiom account](https://app.axiom.co/register) (free tier available)
23+
- A deployed Worker that you want to monitor
24+
- An Axiom dataset to send data to
25+
26+
## Step 1: Create a dataset
27+
28+
If you don't already have a dataset to send data to:
29+
1. Log in to your [Axiom account](https://app.axiom.co/)
30+
2. Navigate to **Datasets** in the left sidebar
31+
3. Click **New Dataset**
32+
4. Enter a name (e.g. `cloudflare-workers-otel`)
33+
5. Click **Create Dataset**
34+
35+
## Step 2: Get your Axiom API token and dataset
36+
37+
1. Navigate to **Settings** in the left sidebar
38+
2. Click on **API Tokens**
39+
3. Click **Create API Token**
40+
4. Configure your API token:
41+
- **Name**: Enter a descriptive name (e.g., `cloudflare-workers-otel`)
42+
- **Permissions**: Select **Ingest** permission (required for sending telemetry data)
43+
- **Datasets**: Choose which datasets this token can write to, or select **All Datasets**
44+
5. Click **Create**
45+
6. **Important**: Copy the API token immediately and store it securely - you won't be able to see it again
46+
47+
The API token will look something like: `xaat-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`
48+
49+
## Step 3: Configure Cloudflare destinations
50+
51+
Now you'll create destinations in the Cloudflare dashboard that point to Axiom.
52+
53+
### Axiom OTLP endpoints
54+
55+
Axiom provides separate OTLP endpoints for traces and logs:
56+
- **Traces**: `https://api.axiom.co/v1/traces`
57+
- **Logs**: `https://api.axiom.co/v1/logs`
58+
59+
### Configure trace or logs destination
60+
61+
1. Navigate to your Cloudflare account's [Workers Observability](https://dash.cloudflare.com/?to=/:account/workers-and-pages/observability/pipelines) section
62+
2. Click **Add destination**
63+
3. Configure your trace destination:
64+
- **Destination Name**: `axiom-traces` (or any descriptive name)
65+
- **Destination Type**: Select **Traces**
66+
- **OTLP Endpoint**: `https://api.axiom.co/v1/traces` (or `/v1/logs`)
67+
- **Custom Headers**: Add two required headers:
68+
- Authentication header
69+
- Header name: `Authorization`
70+
- Header value: `Bearer <your-api-token>`
71+
- Dataset header:
72+
- Header name: `X-Axiom-Dataset`
73+
- Header value: Your dataset name (e.g., `cloudflare-workers-otel`)
74+
4. Click **Save**
75+
76+
## Step 3: Configure your Worker
77+
78+
With your destinations created in the Cloudflare dashboard, update your Worker's configuration to enable telemetry export.
79+
80+
<WranglerConfig>
81+
82+
```json
83+
{
84+
"observability": {
85+
"traces": {
86+
"enabled": true,
87+
// Must match the destination name in the dashboard
88+
"destinations": ["axiom-traces"]
89+
},
90+
"logs": {
91+
"enabled": true,
92+
// Must match the destination name in the dashboard
93+
"destinations": ["axiom-logs"]
94+
}
95+
}
96+
}
97+
```
98+
99+
</WranglerConfig>
100+
101+
After updating your configuration, deploy your Worker for the changes to take effect.
102+
103+
:::note
104+
It may take a few minutes after deployment for data to appear in Axiom.
105+
:::
106+

‎src/content/docs/workers/observability/exporting-opentelemetry-data/grafana-cloud.mdx‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
pcx_content_type: how-to
3+
title: Export to Grafana Cloud
4+
sidebar:
5+
order: 2
6+
---
7+
import {WranglerConfig} from "~/components";
8+
9+
Grafana Cloud is a fully managed observability platform that provides visualization, alerting, and analytics for your telemetry data. By exporting your Cloudflare Workers telemetry to Grafana Cloud, you can:
10+
- Visualize distributed traces in **Grafana Tempo** to understand request flows and performance bottlenecks
11+
- Query and analyze logs in **Grafana Loki** alongside your traces
12+
13+
This guide will walk you through configuring Cloudflare Workers to export OpenTelemetry-compliant traces and logs to your Grafana Cloud stack.
14+
15+
![Grafana Tempo trace view showing a distributed trace for a service with multiple spans including fetch requests, durable object subrequests, and queue operations, with timing information displayed on a timeline](~/assets/images/workers-observability/grafana-traces.png)
16+
17+
## Prerequisites
18+
19+
Before you begin, ensure you have:
20+
21+
- An active [Grafana Cloud account](https://grafana.com/auth/sign-up/create-user) (free tier available)
22+
- A deployed Worker that you want to monitor
23+
24+
## Step 1: Access the OpenTelemetry setup guide
25+
26+
1. Log in to your [Grafana Cloud portal](https://grafana.com/)
27+
2. From your organization's home page, navigate to **Connections** → **Add new connection**
28+
3. Search for "OpenTelemetry" and select **OpenTelemetry (OTLP)**
29+
4. Select **Quickstart** then select **JavaScript**
30+
6. Click **Create a new token**
31+
7. Enter a name for your token (e.g., `cloudflare-workers-otel`) and click **create token**
32+
8. Click on **Close** without copying the token
33+
9. Copy and Save the value for `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` in the `Environment variables` code block as the OTel endpoint and as the Auth header value respectively
34+
35+
36+
## Step 2: Set up destination
37+
1. Navigate to your Cloudflare account's [Workers Observability](https://dash.cloudflare.com/?to=/:account/workers-and-pages/observability/pipelines) section
38+
2. Click **Add destination** and configure a destination name (e.g. `grafana-tracing`)
39+
3. From Grafana, copy your Otel endpoint, auth header, and auth value
40+
* Your OTEL endpoint will look like `https://otlp-gateway-prod-us-east-2.grafana.net/otlp` (append `/v1/traces` for traces and `/v1/logs` for logs)
41+
* Your custom header should include:
42+
* Your auth header name `Authorization`
43+
* Your auth header value `Basic MTMxxx...`
44+
45+
## Step 3: Configure your Worker
46+
47+
With your destination created in the Cloudflare dashboard, update your Worker's configuration to enable telemetry export.
48+
49+
<WranglerConfig>
50+
51+
```json
52+
{
53+
"observability": {
54+
"traces": {
55+
"enabled": true,
56+
// Must match the destination name in the dashboard
57+
"destinations": ["grafana-traces"]
58+
},
59+
"logs": {
60+
"enabled": true,
61+
// Must match the destination name in the dashboard
62+
"destinations": ["grafana-logs"]
63+
}
64+
}
65+
}
66+
```
67+
68+
</WranglerConfig>
69+
70+
After updating your configuration, deploy your Worker for the changes to take effect.
71+
72+
:::note
73+
It may take a few minutes after deployment for data to appear in Grafana Cloud.
74+
:::

‎src/content/docs/workers/observability/exporting-opentelemetry-data/honeycomb.mdx‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
pcx_content_type: how-to
3+
title: Export to Honeycomb
4+
sidebar:
5+
order: 1
6+
---
7+
8+
import {WranglerConfig} from "~/components";
9+
10+
Honeycomb is an observability platform built for high-cardinality data that helps you understand and debug your applications. By exporting your Cloudflare Workers application telemetry to Honeycomb, you can:
11+
- Visualize traces to understand request flows and identify performance bottlenecks
12+
- Query and analyze logs with unlimited dimensionality across any attribute
13+
- Create custom queries and dashboards to monitor your Workers
14+
15+
![Trace view including POST request, fetch operations, durable object subrequest, and queue send, with timing information displayed on a timeline](~/assets/images/workers-observability/honeycomb-example.png)
16+
17+
This guide will walk you through configuring your Cloudflare Worker application to export OpenTelemetry-compliant traces and logs to Honeycomb.
18+
19+
## Prerequisites
20+
21+
Before you begin, ensure you have:
22+
23+
- An active [Honeycomb account](https://ui.honeycomb.io/signup) (free tier available)
24+
- A deployed Worker that you want to monitor
25+
26+
## Step 1: Get your Honeycomb API key
27+
28+
1. Log in to your [Honeycomb account](https://ui.honeycomb.io/)
29+
2. Navigate to your account settings by clicking on your profile icon in the top right
30+
3. Select **Team Settings**
31+
4. In the left sidebar, click **Environments** and click the gear icon
32+
5. Find your environment (e.g., `production`, `test`) or create a new one
33+
6. Under **API Keys**, click **Create Ingest API Key**
34+
7. Configure your API key:
35+
- **Name**: Enter a descriptive name (e.g., `cloudflare-workers-otel`)
36+
- **Permissions**: Select **Can create services/datasets** (required for OTLP ingestion)
37+
8. Click **Create**
38+
9. **Important**: Copy the API key immediately and store it securely - you won't be able to see it again
39+
40+
The API key will look something like: `hcaik_01hq...`
41+
42+
## Step 2: Configure Cloudflare destinations
43+
44+
Now you'll create destinations in the Cloudflare dashboard that point to Honeycomb.
45+
46+
### Honeycomb OTLP endpoints
47+
48+
Honeycomb provides separate OTLP endpoints for traces and logs:
49+
- **Traces**: `https://api.honeycomb.io/v1/traces`
50+
- **Logs**: `https://api.honeycomb.io/v1/logs`
51+
52+
### Configure trace destination
53+
54+
1. Navigate to your Cloudflare account's [Workers Observability](https://dash.cloudflare.com/?to=/:account/workers-and-pages/observability/pipelines) section
55+
2. Click **Add destination**
56+
3. Configure your trace destination:
57+
- **Destination Name**: `honeycomb-traces` (or any descriptive name)
58+
- **Destination Type**: Select **Traces**
59+
- **OTLP Endpoint**: `https://api.honeycomb.io/v1/traces`
60+
- **Custom Headers**: Add the authentication header:
61+
- Header name: `x-honeycomb-team`
62+
- Header value: Your Honeycomb API key (e.g., `hcaik_01hq...`)
63+
4. Click **Save**
64+
65+
### Configure logs destination
66+
67+
Repeat the process for logs:
68+
69+
1. Click **Add destination** again
70+
2. Configure your logs destination:
71+
- **Destination Name**: `honeycomb-logs` (or any descriptive name)
72+
- **Destination Type**: Select **Logs**
73+
- **OTLP Endpoint**: `https://api.honeycomb.io/v1/logs`
74+
- **Custom Headers**: Add the authentication header:
75+
- Header name: `x-honeycomb-team`
76+
- Header value: Your Honeycomb API key (same as above)
77+
3. Click **Save**
78+
79+
## Step 3: Configure your Worker
80+
81+
With your destinations created in the Cloudflare dashboard, update your Worker's configuration to enable telemetry export.
82+
83+
<WranglerConfig>
84+
85+
```json
86+
{
87+
"observability": {
88+
"traces": {
89+
"enabled": true,
90+
// Must match the destination name in the dashboard
91+
"destinations": ["honeycomb-traces"]
92+
},
93+
"logs": {
94+
"enabled": true,
95+
// Must match the destination name in the dashboard
96+
"destinations": ["honeycomb-logs"]
97+
}
98+
}
99+
}
100+
```
101+
102+
</WranglerConfig>
103+
104+
After updating your configuration, deploy your Worker for the changes to take effect.
105+
106+
:::note
107+
It may take a few minutes after deployment for data to appear in Honeycomb.
108+
:::

0 commit comments

Comments
 (0)