|
| 1 | +--- |
| 2 | +title: Correlate OpenTelemetry Traces and DBM |
| 3 | +further_reading: |
| 4 | +- link: "/opentelemetry/otel_tracing/" |
| 5 | + tag: "Documentation" |
| 6 | + text: "Send OpenTelemetry Traces to Datadog" |
| 7 | +--- |
| 8 | + |
| 9 | +## Overview |
| 10 | + |
| 11 | +Datadog Database Monitoring (DBM) correlates backend traces from your OpenTelemetry-instrumented application with detailed database performance data. This allows you to link spans from your application to related query metrics and execution plans, helping you identify the exact queries that are slowing down your services. |
| 12 | + |
| 13 | +## Requirements |
| 14 | + |
| 15 | +Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog. |
| 16 | + |
| 17 | +## Setup |
| 18 | + |
| 19 | +To correlate traces and metrics, you must: |
| 20 | + |
| 21 | +1. **Instrument database spans**: Add specific OpenTelemetry attributes to your database spans to enable correlation with DBM. |
| 22 | + |
| 23 | +2. **Configure trace ingestion path**: Enable the correct feature gate on your Collector or Agent to ensure database spans are properly processed for DBM. |
| 24 | + |
| 25 | +### Step 1: Instrument your database spans |
| 26 | + |
| 27 | +For DBM correlation to work, your database spans must include the following attributes. |
| 28 | + |
| 29 | + |
| 30 | +| Attribute | Required? | Description | Example | |
| 31 | +|----------------|-----------|:------------------------------------------------------------------------------------------------------------------|-----------------------------------------| |
| 32 | +| `db.system` | Yes | The database technology. | `postgres`, `mysql`, `sqlserver` | |
| 33 | +| `db.statement` | Yes | The raw SQL query text. Datadog uses this to set the span's resource name after obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | |
| 34 | +| `span.type` | Yes | **(Datadog-specific)** Identifies and processes database spans. Usually derived automatically by the OpenTelemetry SDK or Datadog Agent. Only set manually when spans are created directly with the SDK. | `sql`, `postgres`, `mysql`, `sql.query` | |
| 35 | +| `db.name` | No | The logical database or schema name being queried. | `user_accounts` | |
| 36 | + |
| 37 | +<div class="alert alert-info"> |
| 38 | +The <code>span.type</code> attribute is a Datadog-specific convention for identifying and processing database spans. |
| 39 | +When using OpenTelemetry auto-instrumentation or the Datadog Agent, this attribute is set automatically. |
| 40 | +Only add it manually if you are instrumenting spans directly with the SDK. |
| 41 | +</div> |
| 42 | + |
| 43 | +#### Using auto instrumentation |
| 44 | + |
| 45 | +To get started, instrument your application using the appropriate OpenTelemetry auto-instrumentation library for your language. For setup instructions, see the official [OpenTelemetry instrumentation documentation][4]. |
| 46 | + |
| 47 | +These libraries automatically add the required `db.system` and `db.statement` attributes. The Datadog Agent or SDK then derives `span.type` automatically, so no manual attribute configuration is needed. |
| 48 | + |
| 49 | +{{% collapse-content title="Set attributes manually (advanced)" level="h4" %}} |
| 50 | +If your environment involves a custom database client or spans not recognized by the library, you can enrich them using the OpenTelemetry Collector’s `attributes` processor. |
| 51 | + |
| 52 | +For example, you can add `span.type: sql` to any span that has the `db.system` attribute: |
| 53 | + |
| 54 | +```yaml |
| 55 | +processors: |
| 56 | + attributes/add_span_type: |
| 57 | + actions: |
| 58 | + - key: span.type |
| 59 | + value: "sql" |
| 60 | + action: insert |
| 61 | + # Apply this action only to spans that have the db.system attribute |
| 62 | + from_context: span |
| 63 | + when: |
| 64 | + - span.attributes["db.system"] != nil |
| 65 | + |
| 66 | +service: |
| 67 | + pipelines: |
| 68 | + traces: |
| 69 | + # Add the processor to your traces pipeline |
| 70 | + processors: [..., attributes/add_span_type, ...] |
| 71 | +``` |
| 72 | +{{% /collapse-content %}} |
| 73 | +
|
| 74 | +#### Using manual instrumentation |
| 75 | +
|
| 76 | +If you are manually creating spans with the OpenTelemetry SDK, you can set the attributes directly in your code. For more information, see the [OpenTelemetry documentation][4]. |
| 77 | +
|
| 78 | +The following is a conceptual example of manual instrumentation using Python's OpenTelemetry SDK: |
| 79 | +
|
| 80 | +```python |
| 81 | +from opentelemetry import trace |
| 82 | + |
| 83 | +tracer = trace.get_tracer("my-app.instrumentation") |
| 84 | + |
| 85 | +# When making a database call, create a span and set attributes |
| 86 | +with tracer.start_as_current_span("postgres.query") as span: |
| 87 | + # Set attributes required for DBM correlation |
| 88 | + span.set_attribute("span.type", "sql") |
| 89 | + span.set_attribute("db.system", "postgres") |
| 90 | + span.set_attribute("db.statement", "SELECT * FROM users WHERE id = ?") |
| 91 | + span.set_attribute("db.name", "user_accounts") |
| 92 | + |
| 93 | + # Your actual database call would go here |
| 94 | + # db_cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) |
| 95 | +``` |
| 96 | + |
| 97 | +### Step 2: Configure your ingest path |
| 98 | + |
| 99 | +Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. |
| 100 | + |
| 101 | +{{< tabs >}} |
| 102 | +{{% tab "Datadog Agent (DDOT Collector)" %}} |
| 103 | + |
| 104 | + |
| 105 | +If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: |
| 106 | + |
| 107 | +```yaml |
| 108 | +datadog: |
| 109 | + otelCollector: |
| 110 | + featureGates: "datadog.EnableReceiveResourceSpansV2,datadog.EnableOperationAndResourceNameV2" |
| 111 | +``` |
| 112 | +
|
| 113 | +{{% /tab %}} |
| 114 | +{{% tab "OTel Collector" %}} |
| 115 | +
|
| 116 | +When starting the Collector, you must enable the correct feature gate for your version. |
| 117 | +
|
| 118 | +#### Collector v0.124.0 and later |
| 119 | +
|
| 120 | +For recent versions of the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate: |
| 121 | + |
| 122 | +```sh |
| 123 | +otelcontribcol --config=config.yaml \ |
| 124 | +--feature-gates=datadog.EnableOperationAndResourceNameV2 |
| 125 | +``` |
| 126 | +#### Collector v0.118.0 - v0.123.0 |
| 127 | + |
| 128 | +For older versions of the Collector, both of the following feature gates are required: |
| 129 | + |
| 130 | +```sh |
| 131 | +otelcontribcol --config=config.yaml \ |
| 132 | +--feature-gates=datadog.EnableReceiveResourceSpansV2,datadog.EnableOperationAndResourceNameV2 |
| 133 | +``` |
| 134 | + |
| 135 | +{{% /tab %}} |
| 136 | + |
| 137 | +{{% tab "Datadog Agent (OTLP Ingest)" %}} |
| 138 | + |
| 139 | +In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. |
| 140 | + |
| 141 | +{{% /tab %}} |
| 142 | + |
| 143 | +{{< /tabs >}} |
| 144 | + |
| 145 | +### View correlated data in Datadog |
| 146 | + |
| 147 | +After your application is sending traces, you can see the correlation in the APM Trace View: |
| 148 | + |
| 149 | +1. Navigate to [**APM** > **Traces**][3]. |
| 150 | +2. Find and click on a trace from your instrumented service. |
| 151 | +3. In the trace's flame graph, select a database span (for example, a span with `span.type: sql`) |
| 152 | +4. In the details panel, click the **SQL Queries** tab. You should see performance metrics and execution plans for the query. |
| 153 | + |
| 154 | +## Troubleshooting |
| 155 | + |
| 156 | +If you don't see the expected correlation between your APM traces and DBM, it's typically due to a missing or incorrect configuration. Check the following common causes: |
| 157 | + |
| 158 | +- **Missing attributes**: The database span must contain `db.system` and `db.statement`. The `span.type` attribute is also required but is typically derived automatically by Datadog. |
| 159 | +- **Incorrect unified service tagging**: The `service` tag on your traces must match the `service` tag on your database host metrics. Verify that [unified service tagging][1] is configured correctly. |
| 160 | +- **The SQL query may not be parsable**: The correlation relies on Datadog's ability to parse the SQL query from the `db.statement` attribute. If the query uses non-standard or complex syntax, parsing may fail. If you suspect this is the case, [contact Datadog support][5] for assistance. |
| 161 | +- **The correct feature gates must be enabled** for your specific trace ingestion path as described in the setup steps. |
| 162 | + |
| 163 | +## Further reading |
| 164 | + |
| 165 | +{{< partial name="whats-next/whats-next.html" >}} |
| 166 | + |
| 167 | +[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging |
| 168 | +[2]: /opentelemetry/integrations/host_metrics |
| 169 | +[3]: https://app.datadoghq.com/apm/traces |
| 170 | +[4]: https://opentelemetry.io/docs/languages/ |
| 171 | +[5]: /help |
0 commit comments