-
Notifications
You must be signed in to change notification settings - Fork 1.2k
OTel Correlation: DBM #30825
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
brett0000FF
wants to merge
36
commits into
master
Choose a base branch
from
brett.blue/otel-correlate-dbm
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
OTel Correlation: DBM #30825
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
67dfe22
Initial draft.
brett0000FF ffb9f0b
Flesh out content into second draft.
brett0000FF c06a960
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF 3ee6847
Tweak examples.
brett0000FF 409da0e
Remove vague statement.
brett0000FF 69dc93e
Create new Data Correlation parent page with UST.
brett0000FF 9dabb1a
Update correlate logs and traces page.
brett0000FF d1251d3
Add separate pages for metrics and dbm.
brett0000FF 75e126c
Add new pages to left-nav.
brett0000FF ffeaaac
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF e0dd27d
Clean up.
brett0000FF 4d3d48d
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF dd49a7b
Add example to DBM page.
brett0000FF e974e04
Restructure into two distinct steps with a few log pipelines.
brett0000FF aac4af7
Remove repetition of correlate in nav.
brett0000FF 25dd8f0
Merge branch 'master' into brett.blue/otel-correlation
brett0000FF 780c393
Apply suggestions from code review
brett0000FF 52bee79
Remove redundant alert.
brett0000FF 5ebe932
Remove misleading description of attributes.
brett0000FF aefb382
Clarify how trace context is injected using bridges.
brett0000FF fe0405e
Remove level param from LoggingHandler example.
brett0000FF 0e504fa
Revise 'Scrape logs from files' approach.
brett0000FF 0f3e55a
Edits.
brett0000FF 2f31c00
Apply final feedback and clean up.
brett0000FF a522cc8
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF b5de195
Fix spacing.
brett0000FF e9bae95
Address final nits.
brett0000FF f8cdc9f
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF 8e7da6b
Remove DBM content; move to new branch.
brett0000FF 3f1d1c7
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF 5a21149
Restore DBM content to new branch.
brett0000FF 0e71ec5
Merge branch 'master' of github.com:DataDog/documentation into brett.…
brett0000FF f1da80c
Address feedback from review.
brett0000FF 54ecfb8
Fix menu file.
brett0000FF 776d91b
Merge branch 'master' into brett.blue/otel-correlate-dbm
brett0000FF c51d3bb
Fix landing page.
brett0000FF File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,120 @@ | ||
--- | ||
title: Correlate OpenTelemetry Traces and DBM | ||
further_reading: | ||
- link: "/opentelemetry/otel_tracing/" | ||
tag: "Documentation" | ||
text: "Send OpenTelemetry Traces to Datadog" | ||
--- | ||
|
||
## Overview | ||
|
||
Correlate backend traces to detailed database performance data in Datadog Database Monitoring (DBM). This allows you to link spans from your OpenTelemetry-instrumented application to related query metrics and execution plans to identify the exact queries that are slowing down your application. | ||
|
||
## Requirements | ||
|
||
Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog. | ||
|
||
## Setup | ||
|
||
To correlate traces and metrics, you must: | ||
|
||
1. **Instrument database spans**: Add specific OpenTelemetry attributes to your database spans to enable correlation with DBM. | ||
|
||
2. **Configure trace ingestion path**: Enable the correct feature gate on your Collector or Agent to ensure database spans are properly processed for DBM. | ||
|
||
### Step 1: Instrument your database spans | ||
|
||
For DBM correlation to work, your database spans must include the following attributes. | ||
|
||
| Attribute | Description | Example | | ||
|----------------|-----------------------------------------------------------------------------------------------------|------------------------------------| | ||
| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | | ||
| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | | ||
| `db.name` | The logical database or schema name being queried. | `user_accounts` | | ||
| `span.type` | **Required (Datadog-specific).** The type of span such as `sql`,`postgres`, `mysql`, or `sql.query` | `sql` | | ||
|
||
#### Example | ||
|
||
The method for adding these attributes depends on your setup. If you are using an OpenTelemetry auto-instrumentation library for your database client, see its documentation for configuration options. If you are manually creating spans with the OpenTelemetry SDK, you can set the attributes directly in your code. For more information, see the [OpenTelemetry documentation][4]. | ||
|
||
The following is a conceptual example of manual instrumentation using Python's OpenTelemetry SDK: | ||
|
||
```python | ||
from opentelemetry import trace | ||
|
||
tracer = trace.get_tracer("my-app.instrumentation") | ||
|
||
# When making a database call, create a span and set attributes | ||
with tracer.start_as_current_span("postgres.query") as span: | ||
# Set attributes required for DBM correlation | ||
span.set_attribute("span.type", "sql") | ||
span.set_attribute("db.system", "postgres") | ||
span.set_attribute("db.statement", "SELECT * FROM users WHERE id = ?") | ||
span.set_attribute("db.name", "user_accounts") | ||
|
||
# Your actual database call would go here | ||
# db_cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) | ||
``` | ||
|
||
### Step 2: Configure your ingest path | ||
|
||
Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. | ||
|
||
{{< tabs >}} | ||
{{% tab "Datadog Agent (DDOT Collector)" %}} | ||
|
||
|
||
If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: | ||
|
||
```yaml | ||
datadog: | ||
otelCollector: | ||
featureGates: datadog.EnableOperationAndResourceNameV2 | ||
``` | ||
|
||
{{% /tab %}} | ||
{{% tab "OTel Collector" %}} | ||
|
||
When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. | ||
|
||
```sh | ||
otelcontribcol --config=config.yaml \ | ||
--feature-gates=datadog.EnableOperationAndResourceNameV2 | ||
``` | ||
|
||
{{% /tab %}} | ||
|
||
{{% tab "Datadog Agent (OTLP Ingest)" %}} | ||
|
||
In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. | ||
|
||
{{% /tab %}} | ||
|
||
{{< /tabs >}} | ||
|
||
### View correlated data in Datadog | ||
|
||
After your application is sending traces, you can see the correlation in the APM Trace View: | ||
|
||
1. Navigate to [**APM** > **Traces**][3]. | ||
2. Find and click on a trace from your instrumented service. | ||
3. In the trace's flame graph, select a database span (for example, a span with `span.type: sql`) | ||
4. In the details panel, click the **SQL Queries** tab. You should see performance metrics and execution plans for the query. | ||
|
||
## Troubleshooting | ||
|
||
If you don't see the expected correlation between your APM traces and DBM, it's typically due to a missing or incorrect configuration. Check the following common causes: | ||
|
||
- **All required attributes (`db.system`, `db.statement`, `span.type`) must be present** on the database span. | ||
- **The SQL query may not be parsable**: The correlation relies on Datadog's ability to parse the SQL query from the `db.statement` attribute. If the query uses non-standard or highly complex syntax, parsing may fail. If you suspect this is the case, [contact Datadog support][5] for assistance. | ||
- **The correct feature gates must be enabled** for your specific trace ingestion path as described in the setup steps. | ||
|
||
## Further reading | ||
|
||
{{< partial name="whats-next/whats-next.html" >}} | ||
|
||
[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging | ||
[2]: /opentelemetry/integrations/host_metrics | ||
[3]: https://app.datadoghq.com/apm/traces | ||
[4]: https://opentelemetry.io/docs/languages/ | ||
[5]: /help |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to have a list of some known gotchas at this point? The correlation here is best-effort and not guaranteed, so I want to make sure people are aware of that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. Added a troubleshooting section at the bottom that we can add to for common scenarios.