Skip to content

Commit 54c000b

Browse files
brett0000FFjade-guiton-ddhestonhoffman
authored
OTel Correlation Guides (#30354)
* Initial draft. * Flesh out content into second draft. * Tweak examples. * Remove vague statement. * Create new Data Correlation parent page with UST. * Update correlate logs and traces page. * Add separate pages for metrics and dbm. * Add new pages to left-nav. * Clean up. * Add example to DBM page. * Restructure into two distinct steps with a few log pipelines. * Remove repetition of correlate in nav. * Apply suggestions from code review Co-authored-by: Jade Guiton <[email protected]> * Remove redundant alert. * Remove misleading description of attributes. * Clarify how trace context is injected using bridges. * Remove level param from LoggingHandler example. * Revise 'Scrape logs from files' approach. * Edits. * Apply final feedback and clean up. * Fix spacing. * Address final nits. * Remove DBM content; move to new branch. * Update content/en/opentelemetry/correlate/_index.md Co-authored-by: Heston Hoffman <[email protected]> --------- Co-authored-by: Jade Guiton <[email protected]> Co-authored-by: Heston Hoffman <[email protected]>
1 parent 2d54921 commit 54c000b

File tree

5 files changed

+343
-273
lines changed

5 files changed

+343
-273
lines changed

‎config/_default/menus/main.en.yaml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -780,16 +780,21 @@ menu:
780780
identifier: otel_explore
781781
parent: opentelemetry_top_level
782782
weight: 7
783-
- name: Correlate Logs and Traces
783+
- name: Logs and Traces
784784
url: /opentelemetry/correlate/logs_and_traces/
785785
identifier: otel_logs
786786
parent: otel_explore
787787
weight: 701
788-
- name: Correlate RUM and Traces
788+
- name: Metrics and Traces
789+
url: /opentelemetry/correlate/metrics_and_traces/
790+
identifier: otel_metrics_traces
791+
parent: otel_explore
792+
weight: 702
793+
- name: RUM and Traces
789794
url: /opentelemetry/correlate/rum_and_traces/
790795
identifier: otel_rum
791796
parent: otel_explore
792-
weight: 702
797+
weight: 703
793798
- name: Integrations
794799
url: opentelemetry/integrations/
795800
identifier: otel_integrations

‎content/en/opentelemetry/correlate/_index.md‎

Lines changed: 84 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
---
2-
title: Correlate Data
2+
title: Correlate OpenTelemetry Data
3+
description: Learn how to correlate your OpenTelemetry traces, metrics, logs, and other telemetry in Datadog to get a unified view of your application's performance.
34
aliases:
45
- /opentelemetry/otel_logs/
56
further_reading:
@@ -10,13 +11,89 @@ further_reading:
1011

1112
## Overview
1213

13-
Link your telemetry data for full-stack observability:
14+
Getting a unified view of your application's performance requires connecting its traces, metrics, logs, user interactions, and more. By correlating your OpenTelemetry data in Datadog, you can navigate between all related telemetry in a single view, allowing you to diagnose and resolve issues faster.
1415

15-
{{< whatsnext desc=" " >}}
16-
{{< nextlink href="/opentelemetry/correlate/logs_and_traces/" >}}Connect Logs and Traces{{< /nextlink >}}
17-
{{< nextlink href="/opentelemetry/correlate/rum_and_traces/" >}}Connect RUM and Traces{{< /nextlink >}}
18-
{{< /whatsnext >}}
16+
17+
## Prerequisite: Unified service tagging
18+
19+
Datadog uses three standard tags to link telemetry together: `env`, `service`, and `version`.
20+
21+
To ensure your OpenTelemetry data is properly correlated, you must configure your application or system to use these tags by setting a standard set of OpenTelemetry resource attributes. Datadog automatically maps these attributes to the correct tags.
22+
23+
| OpenTelemetry Resource Attribute | Datadog Tag | Notes |
24+
|----------------------------------|-------------|---------------------------------------------------------------------------------------------------------|
25+
| `deployment.environment.name` | `env` | **Recommended**. Supported in Agent v7.58.0+ and Collector Exporter v0.110.0+. |
26+
| `deployment.environment` | `env` | Use instead of `deployment.environment.name` if you are running an Agent version older than v7.58.0 or a Collector Exporter older than v0.110.0. |
27+
| `service.name` | `service` | |
28+
| `service.version` | `version` | |
29+
30+
You can set these attributes in your application's environment variables, SDK, or in the OpenTelemetry Collector.
31+
32+
{{< tabs >}}
33+
{{% tab "Environment Variables" %}}
34+
35+
Set the `OTEL_RESOURCE_ATTRIBUTES` environment variable with your service's information:
36+
37+
```sh
38+
export OTEL_SERVICE_NAME="my-service"
39+
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.version=1.2.3"
40+
```
41+
42+
{{% /tab %}}
43+
{{% tab "SDK" %}}
44+
45+
Create a Resource with the required attributes and associate it with your TracerProvider in your application code.
46+
47+
Here's an example using the OpenTelemetry SDK for Python:
48+
49+
```python
50+
from opentelemetry.sdk.resources import Resource
51+
from opentelemetry.sdk.trace import TracerProvider
52+
53+
resource = Resource(attributes={
54+
"service.name": "<SERVICE>",
55+
"deployment.environment.name": "<ENV>",
56+
"service.version": "<VERSION>"
57+
})
58+
tracer_provider = TracerProvider(resource=resource)
59+
```
60+
61+
{{% /tab %}}
62+
{{% tab "Collector" %}}
63+
64+
Use the `resource` processor in your Collector configuration to set the resource attributes on your telemetry data:
65+
66+
```yaml
67+
processors:
68+
resource:
69+
attributes:
70+
- key: service.name
71+
value: "my-service"
72+
action: upsert
73+
- key: deployment.environment.name
74+
value: "production"
75+
action: upsert
76+
- key: service.version
77+
value: "1.2.3"
78+
action: upsert
79+
...
80+
```
81+
82+
{{% /tab %}}
83+
{{< /tabs >}}
84+
85+
## Correlate telemetry
86+
87+
After unified service tagging is configured, you can connect your various telemetry streams. Select a guide below for platform-specific instructions.
88+
89+
- [Correlate logs and traces][1]
90+
- [Correlate metrics and traces][2]
91+
- [Correlate RUM and traces][3]
1992

2093
## Further reading
2194

22-
{{< partial name="whats-next/whats-next.html" >}}
95+
{{< partial name="whats-next/whats-next.html" >}}
96+
97+
[1]: /opentelemetry/correlate/logs_and_traces
98+
[2]: /opentelemetry/correlate/metrics_and_traces
99+
[3]: /opentelemetry/correlate/rum_and_traces

‎content/en/opentelemetry/correlate/metrics_and_traces.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
---
2+
title: Correlate OpenTelemetry Traces and Metrics
3+
further_reading:
4+
- link: "/opentelemetry/otel_tracing/"
5+
tag: "Documentation"
6+
text: "Send OpenTelemetry Traces to Datadog"
7+
---
8+
9+
## Overview
10+
11+
Correlating traces with host metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host or container it ran on. This helps you determine if resource contention was the root cause of a performance issue.
12+
13+
Correlation between traces and metrics relies on the following resource attributes:
14+
15+
- `host.name`: For correlating with host metrics (CPU, memory, disk).
16+
- `container.id`: For correlating with container metrics.
17+
18+
## Requirements
19+
20+
Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog.
21+
22+
## Setup
23+
24+
To correlate traces and metrics, you must:
25+
26+
1. **Collect Host Metrics**: You must have the OpenTelemetry Collector configured to collect and send host metrics to Datadog.
27+
28+
2. **Ensure Consistent Tagging**: Your traces and metrics must share a consistent `host.name` (for hosts) or `container.id` (for containers) attribute for Datadog to link them.
29+
30+
31+
### 1. Collect host metrics
32+
33+
To collect system-level metrics from your infrastructure, enable the `hostmetrics` receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage.
34+
35+
Add the `hostmetrics` receiver to the `receivers` section of your Collector configuration and enable it in your `metrics` pipeline:
36+
37+
38+
```yaml
39+
receivers:
40+
hostmetrics:
41+
collection_interval: 10s
42+
scrapers:
43+
cpu:
44+
memory:
45+
disk:
46+
...
47+
48+
service:
49+
pipelines:
50+
metrics:
51+
receivers: [hostmetrics, ...]
52+
processors: [...]
53+
exporters: [...]
54+
```
55+
56+
For the complete, working configuration, including Kubernetes-specific setup, see the [Host Metrics][2] documentation.
57+
58+
### 2. Ensure consistent host and container tagging
59+
60+
For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver.
61+
62+
## View correlated data in Datadog
63+
64+
After your application is sending traces and the Collector is sending host metrics, you can see the correlation in the APM Trace View.
65+
66+
1. Navigate to [**APM** > **Traces**][3].
67+
2. Find and click on a trace from your instrumented service.
68+
3. In the trace's flame graph, select a span that ran on the instrumented host.
69+
4. In the details panel, click the **Infrastructure** tab. You should see the host metrics, like CPU and memory utilization, from the host that executed that part of the request.
70+
71+
This allows you to immediately determine if a spike in host metrics corresponds with the performance of a specific request.
72+
73+
## Further reading
74+
75+
{{< partial name="whats-next/whats-next.html" >}}
76+
77+
[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging
78+
[2]: /opentelemetry/integrations/host_metrics
79+
[3]: https://app.datadoghq.com/apm/traces

‎content/en/opentelemetry/integrations/host_metrics.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ The metrics, mapped to Datadog metrics, are used in the following views:
9999
- [Host default dashboards][8]
100100
- [APM Trace view Host info][9]
101101

102-
**Note**: To correlate trace and host metrics, configure [Universal Service Monitoring attributes][10] for each service, and set the `host.name` resource attribute to the corresponding underlying host for both service and collector instances.
102+
**Note**: To correlate trace and host metrics, configure [Unified Service Tagging attributes][10] for each service, and set the `host.name` resource attribute to the corresponding underlying host for both service and collector instances.
103103

104104
The following table shows which Datadog host metric names are associated with corresponding OpenTelemetry host metric names, and, if applicable, what math is applied to the OTel host metric to transform it to Datadog units during the mapping.
105105

@@ -168,6 +168,6 @@ Value: 1153183744
168168
[7]: https://app.datadoghq.com/infrastructure
169169
[8]: /opentelemetry/collector_exporter/#out-of-the-box-dashboards
170170
[9]: /tracing/trace_explorer/trace_view/?tab=hostinfo
171-
[10]: /universal_service_monitoring/setup/
171+
[10]: /opentelemetry/correlate/#prerequisite-unified-service-tagging
172172

173173

0 commit comments

Comments
 (0)