Skip to content

OTel Correlation: DBM #30825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 36 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
67dfe22
Initial draft.
brett0000FF Jul 2, 2025
ffb9f0b
Flesh out content into second draft.
brett0000FF Jul 8, 2025
c06a960
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF Jul 8, 2025
3ee6847
Tweak examples.
brett0000FF Jul 9, 2025
409da0e
Remove vague statement.
brett0000FF Jul 9, 2025
69dc93e
Create new Data Correlation parent page with UST.
brett0000FF Jul 14, 2025
9dabb1a
Update correlate logs and traces page.
brett0000FF Jul 14, 2025
d1251d3
Add separate pages for metrics and dbm.
brett0000FF Jul 14, 2025
75e126c
Add new pages to left-nav.
brett0000FF Jul 14, 2025
ffeaaac
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF Jul 14, 2025
e0dd27d
Clean up.
brett0000FF Jul 16, 2025
4d3d48d
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF Jul 16, 2025
dd49a7b
Add example to DBM page.
brett0000FF Jul 16, 2025
e974e04
Restructure into two distinct steps with a few log pipelines.
brett0000FF Jul 17, 2025
aac4af7
Remove repetition of correlate in nav.
brett0000FF Jul 17, 2025
25dd8f0
Merge branch 'master' into brett.blue/otel-correlation
brett0000FF Jul 21, 2025
780c393
Apply suggestions from code review
brett0000FF Jul 22, 2025
52bee79
Remove redundant alert.
brett0000FF Jul 22, 2025
5ebe932
Remove misleading description of attributes.
brett0000FF Jul 22, 2025
aefb382
Clarify how trace context is injected using bridges.
brett0000FF Jul 22, 2025
fe0405e
Remove level param from LoggingHandler example.
brett0000FF Jul 22, 2025
0e504fa
Revise 'Scrape logs from files' approach.
brett0000FF Jul 22, 2025
0f3e55a
Edits.
brett0000FF Jul 22, 2025
2f31c00
Apply final feedback and clean up.
brett0000FF Jul 22, 2025
a522cc8
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF Jul 22, 2025
b5de195
Fix spacing.
brett0000FF Jul 22, 2025
e9bae95
Address final nits.
brett0000FF Jul 28, 2025
f8cdc9f
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF Jul 28, 2025
8e7da6b
Remove DBM content; move to new branch.
brett0000FF Jul 31, 2025
3f1d1c7
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF Jul 31, 2025
5a21149
Restore DBM content to new branch.
brett0000FF Jul 31, 2025
0e71ec5
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF Aug 7, 2025
f1da80c
Address feedback from review.
brett0000FF Aug 15, 2025
54ecfb8
Fix menu file.
brett0000FF Aug 15, 2025
776d91b
Merge branch 'master' into brett.blue/otel-correlate-dbm
brett0000FF Aug 15, 2025
c51d3bb
Fix landing page.
brett0000FF Aug 15, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -750,6 +750,11 @@ menu:
identifier: otel_rum
parent: otel_explore
weight: 703
name: DBM and Traces
url: /opentelemetry/correlate/dbm_and_traces/
identifier: otel_dbm
parent: otel_explore
weight: 704
- name: Integrations
url: opentelemetry/integrations/
identifier: otel_integrations
Expand Down
4 changes: 3 additions & 1 deletion content/en/opentelemetry/correlate/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,11 +89,13 @@ After unified service tagging is configured, you can connect your various teleme
- [Correlate logs and traces][1]
- [Correlate metrics and traces][2]
- [Correlate RUM and traces][3]
- [Correlate DBM and traces][4]

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /opentelemetry/correlate/logs_and_traces
[2]: /opentelemetry/correlate/metrics_and_traces
[3]: /opentelemetry/correlate/rum_and_traces
[3]: /opentelemetry/correlate/rum_and_traces
[4]: /opentelemetry/correlate/dbm_and_traces
120 changes: 120 additions & 0 deletions content/en/opentelemetry/correlate/dbm_and_traces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
---
title: Correlate OpenTelemetry Traces and DBM
further_reading:
- link: "/opentelemetry/otel_tracing/"
tag: "Documentation"
text: "Send OpenTelemetry Traces to Datadog"
---

## Overview

Correlate backend traces to detailed database performance data in Datadog Database Monitoring (DBM). This allows you to link spans from your OpenTelemetry-instrumented application to related query metrics and execution plans to identify the exact queries that are slowing down your application.

## Requirements

Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog.

## Setup

To correlate traces and metrics, you must:

1. **Instrument database spans**: Add specific OpenTelemetry attributes to your database spans to enable correlation with DBM.

2. **Configure trace ingestion path**: Enable the correct feature gate on your Collector or Agent to ensure database spans are properly processed for DBM.

### Step 1: Instrument your database spans

For DBM correlation to work, your database spans must include the following attributes.

| Attribute | Description | Example |
|----------------|-----------------------------------------------------------------------------------------------------|------------------------------------|
| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` |
| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` |
| `db.name` | The logical database or schema name being queried. | `user_accounts` |
| `span.type` | **Required (Datadog-specific).** The type of span such as `sql`,`postgres`, `mysql`, or `sql.query` | `sql` |

#### Example

The method for adding these attributes depends on your setup. If you are using an OpenTelemetry auto-instrumentation library for your database client, see its documentation for configuration options. If you are manually creating spans with the OpenTelemetry SDK, you can set the attributes directly in your code. For more information, see the [OpenTelemetry documentation][4].

The following is a conceptual example of manual instrumentation using Python's OpenTelemetry SDK:

```python
from opentelemetry import trace

tracer = trace.get_tracer("my-app.instrumentation")

# When making a database call, create a span and set attributes
with tracer.start_as_current_span("postgres.query") as span:
# Set attributes required for DBM correlation
span.set_attribute("span.type", "sql")
span.set_attribute("db.system", "postgres")
span.set_attribute("db.statement", "SELECT * FROM users WHERE id = ?")
span.set_attribute("db.name", "user_accounts")

# Your actual database call would go here
# db_cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))
```

### Step 2: Configure your ingest path

Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly.

{{< tabs >}}
{{% tab "Datadog Agent (DDOT Collector)" %}}


If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`:

```yaml
datadog:
otelCollector:
featureGates: datadog.EnableOperationAndResourceNameV2
```

{{% /tab %}}
{{% tab "OTel Collector" %}}

When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later.

```sh
otelcontribcol --config=config.yaml \
--feature-gates=datadog.EnableOperationAndResourceNameV2
```

{{% /tab %}}

{{% tab "Datadog Agent (OTLP Ingest)" %}}

In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`.

{{% /tab %}}

{{< /tabs >}}

### View correlated data in Datadog

After your application is sending traces, you can see the correlation in the APM Trace View:

1. Navigate to [**APM** > **Traces**][3].
2. Find and click on a trace from your instrumented service.
3. In the trace's flame graph, select a database span (for example, a span with `span.type: sql`)
4. In the details panel, click the **SQL Queries** tab. You should see performance metrics and execution plans for the query.

## Troubleshooting

If you don't see the expected correlation between your APM traces and DBM, it's typically due to a missing or incorrect configuration. Check the following common causes:

- **All required attributes (`db.system`, `db.statement`, `span.type`) must be present** on the database span.
- **The SQL query may not be parsable**: The correlation relies on Datadog's ability to parse the SQL query from the `db.statement` attribute. If the query uses non-standard or highly complex syntax, parsing may fail. If you suspect this is the case, [contact Datadog support][5] for assistance.
- **The correct feature gates must be enabled** for your specific trace ingestion path as described in the setup steps.

## Further reading
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to have a list of some known gotchas at this point? The correlation here is best-effort and not guaranteed, so I want to make sure people are aware of that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Added a troubleshooting section at the bottom that we can add to for common scenarios.


{{< partial name="whats-next/whats-next.html" >}}

[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging
[2]: /opentelemetry/integrations/host_metrics
[3]: https://app.datadoghq.com/apm/traces
[4]: https://opentelemetry.io/docs/languages/
[5]: /help
Loading