Skip to content

[db] Add docs for context info propagation #2236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 24 commits into from
Closed

[db] Add docs for context info propagation #2236

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
1529979
[db] Add docs for context info propagation
May 6, 2025
ab6e386
Update docs/database/database-spans.md
sincejune May 7, 2025
08b02a8
Update docs/database/sql-server.md
sincejune May 7, 2025
2c26648
Merge branch 'main' into db-propagate-context-info
May 7, 2025
0ccfda5
feedback
May 7, 2025
a6e65f4
feedback
May 7, 2025
886aa37
feedback
May 7, 2025
f633fa2
feedback
May 7, 2025
5b835f8
Update docs/database/database-spans.md
sincejune May 7, 2025
e021083
Update docs/database/database-spans.md
sincejune May 7, 2025
1bcbf0d
feedback
May 7, 2025
e6803f5
Update docs/database/database-spans.md
sincejune May 8, 2025
f986ee2
add single quotes
May 8, 2025
8794c1d
Add co-author
XSAM May 13, 2025
ec00e3c
feedback
May 19, 2025
fa6cf6d
fix format
May 19, 2025
2c0e01a
update baggage.service.name example
May 19, 2025
430957b
Keep only `service.name` for general databases
XSAM May 21, 2025
7f05f5e
Update sql-server doc
XSAM May 21, 2025
5534272
Adjust font format
XSAM May 21, 2025
9876582
Add an example for sql server
XSAM May 21, 2025
89a4baf
Merge remote-tracking branch 'upstream/main' into db-propagate-contex…
XSAM May 21, 2025
0e7ff53
Fix CI
XSAM May 21, 2025
8effebc
Merge remote-tracking branch 'upstream/main' into db-propagate-contex…
XSAM May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .chloggen/db-propagate-context-info.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Use this changelog template to create an entry for release notes.
#
# If your change doesn't affect end users you should instead start
# your pull request title with [chore] or use the "Skip Changelog" label.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the area of concern in the attributes-registry, (e.g. http, cloud, db)
component: db

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add doc for context information propagation

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
# The values here must be integers.
issues: [2162]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:
32 changes: 32 additions & 0 deletions docs/database/database-spans.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ linkTitle: Spans
- [Notes and well-known identifiers for `db.system.name`](#notes-and-well-known-identifiers-for-dbsystemname)
- [Sanitization of `db.query.text`](#sanitization-of-dbquerytext)
- [Generating a summary of the query](#generating-a-summary-of-the-query)
- [Propagating context to databases](#propagating-context-to-databases)
- [Semantic conventions for specific database technologies](#semantic-conventions-for-specific-database-technologies)

<!-- tocstop -->
Expand Down Expand Up @@ -478,6 +479,37 @@ Semantic conventions for individual database systems or specialized instrumentat
MAY specify a different `db.query.summary` format as long as produced summary remains
relatively short and its cardinality remains low comparing to the `db.query.text`.

## Propagating context to databases

**Status**: [Development][DocumentStatus]

Instrumentations SHOULD propagate the context information to the SQL queries following [sqlcommenter](https://google.github.io/sqlcommenter/spec/).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to share a learning from my org. We've implemented a sqlcommenter-like approach for our instrumentations and we found it more beneficial to prepend the sql comment rather than append to the query text. This runs counter to the sqlcommenter spec, but we found that most databases truncate the query in views like pg_stat_activity, so prepending the attributes offered a better guarantee that they wouldn't be truncated

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zacharycmontoya Thank you for pointing this out!

I think it makes sense to prepend the sql comment instead. I'll update this PR accordingly if there are no objections. cc @open-telemetry/semconv-db-approvers @XSAM

Copy link
Member

@XSAM XSAM May 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using prepend conflicts the spec of sql commenter. We might need to change the spec of sql commenter if prepend approach is superior than append approach.

@zacharycmontoya Could you elaborate more about this? How serious and often the truncate will happen to certain databases if we use append.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Postgres limits query text to 1024 characters by default, but you can increase it.

Copy link
Member

@nenadnoveljic nenadnoveljic May 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepended comments break pg_hint_plan functionality:

EXPLAIN /*+ IndexScan(t) */  /* a */ SELECT * FROM t WHERE n = 1;
                         QUERY PLAN                          
-------------------------------------------------------------
 Index Scan using i on t  (cost=0.15..36.38 rows=13 width=4)
   Index Cond: (n = 1)
(2 rows)

EXPLAIN /* a */ /*+ IndexScan(t) */  SELECT * FROM t WHERE n = 1;
                           QUERY PLAN                            
-----------------------------------------------------------------
 Bitmap Heap Scan on t  (cost=4.26..14.95 rows=13 width=4)
   Recheck Cond: (n = 1)
   ->  Bitmap Index Scan on i  (cost=0.00..4.25 rows=13 width=0)
         Index Cond: (n = 1)
(4 rows)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another idea is to make it flexible by allowing the implementation of propagation for a specific database to choose between append or prepend.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified this PR to incorporate the idea.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned about pushing this as a standard in light of google/sqlcommenter#284

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to use sqlcommenter only with attributes that do not change frequently. In the SQL Server documentation, I have added the section below. @trask Does this plan seem good to you?

Instrumentations SHOULD make use of SET CONTEXT_INFO to add high-cardinality information(e.g. traceparent) as adding sql comments to queries changes their unique identifier in SQL Server.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this PR is still about the context propagation, right? trace-id + span-id are pretty much guaranteed to be unique

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lmolkova The traceparent format is more widely used, so I prefer adopting that. What are your thoughts?


| Attribute | Type | Description | Require level | Stability |
|----------------|--------|---------------------------------------|-------------------|----------------------------------------------------------------|
| `service.name` | string | Logical name of the service [1] | `Required` | ![Development](https://img.shields.io/badge/-development-blue) |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Copy link
Member

@XSAM XSAM May 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is because of the limitation of actual implementation.

When propagating traceparent via sqlcommenter is not available for certain databases due to the performance impact (google/sqlcommenter#284) and propagating traceparent via SET CONTEXT_INFO is also not available due to the additional network round trip delay, only pushing a service.name (or peer.service) via sqlcommenter become a solution to achieve limited correlation for knowing which service is making the query request.

Some questions here:

  • What is the best name to use here? service.name or peer.service.
    • I slightly prefer peer.service.
  • Should we mark this field as Optional or Recommended?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@XSAM, it is reasonable to me to have this attribute when traceid cannot be injected due to performance reasons.
What do you think about making it conditionally required, where there is not information about span?
If you have traceparent, is it IMO useless.

Copy link
Member

@pellared pellared May 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. In such case service.name feels more like a baggage item that you want to propagate.
Then it could encoded something like

SELECT * FROM songs /* baggage: service.name=music-player:play */

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated this PR and still keep the sheet to represent the minimal data that should be propagated. Please let me know if you feel it's unnecessary.

Copy link
Member

@pellared pellared May 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but I'm not sure if we should be referencing the baggage concept or name for this attribute name.

We somehow need to propagate this information. I was thinking of reusing some existing propagator, instead of creating a new one. The alternative would be probably creating a new "service name propagator".

However, it feels that a baggage propagator can be used for this. From https://www.w3.org/TR/baggage/#example-use-case:

For example, if you need to annotate logs with some request-specific information, you could propagate a property using the baggage header.

I just noticed that here is a nice doc for context propagation for AWS: https://github.com/open-telemetry/semantic-conventions/blob/45e91aedff7d347955ba09269b871f8fd9049ce3/docs/non-normative/compatibility/aws.md#context-propagation

@pellared what do you think of going back to service.name or something slightly different like resource.service.name?

service.name as I originally proposed. Especially that it is already a "reserved" key:

semantic-conventions/docs/resource/README.md

Line 58 in 45e91ae

- [`service.name`](#service)

Example:

/*baggage='service.name%3Dbar'*/

@pellared, do you know how to express an attribute in baggage in the semantic convention doc?

I think that we just need to write it down as I do not think we have any semantic conventions for baggage key-values so far.

CC @reyang, @dyladan, @yurishkuro, @arminru

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the best name to use here? service.name or peer.service.
I slightly prefer peer.service.

service.name. peer.service applies to the remote (downstream service): link

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pellared Do you recommend using the format /*baggage='service.name%3Dbar'*/ instead of /*baggage.service.name='bar'*/? I'm not sure if there is an existing propagator we can use. Just wanted to double-check.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest exploring /*baggage='service.name%3Dbar'* and use of the baggage propagator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pellared Updated the examples accordingly.

| `traceparent` | string | The trace context of current span [2] | `Recommended` [3] | ![Development](https://img.shields.io/badge/-development-blue) |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question, not a blocker for the PR, can be considered separately.
Should we support any text-propagators here by using format/names knows from http headers?
Ref: B3/B3multi as an example: https://github.com/openzipkin/b3-propagation?tab=readme-ov-file#single-header


**[1] `service.name`:** MUST be the same for all instances of horizontally scaled services. If the value was not specified, SDKs MUST fall back to `unknown_service:` concatenated with [process.executable.name](https://opentelemetry.io/docs/specs/semconv/attributes-registry/process/), e.g. `unknown_service:bash`. If `process.executable.name` is not available, the value MUST be set to `unknown_service`.

**[2] `traceparent`:** MUST be in the [text format](https://www.w3.org/TR/trace-context/#traceparent-header).

**[3] `traceparent`:** `tracparent` have extremely high-cardinality. It's RECOMMENDED to propagate this info if the high-cardinality is safe for the behind databases

**Examples:**

- Query with `service.name`:

```sql
SELECT * FROM songs /* service.name=music-player:play */
```

- Query with `service.name` and `traceparent`

```sql
SELECT * FROM songs /* service.name=music-player:play, traceparent=00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 */
```

## Semantic conventions for specific database technologies

More specific Semantic Conventions are defined for the following database technologies:
Expand Down
18 changes: 18 additions & 0 deletions docs/database/sql-server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ linkTitle: SQL Server
<!-- toc -->

- [Spans](#spans)
- [Propagating context to SQL Server](#propagating-context-to-sql-server)
- [Metrics](#metrics)

<!-- tocstop -->
Expand Down Expand Up @@ -152,6 +153,23 @@ and SHOULD be provided **at span creation time** (if provided at all):
<!-- END AUTOGENERATED TEXT -->
<!-- endsemconv -->

### Propagating context to SQL Server

**Status**: [Development][DocumentStatus]

Instrumentations SHOULD propagate the context information to the SQL queries following [sqlcommenter](https://google.github.io/sqlcommenter/spec/).

| Attribute | Type | Description | Require level | Stability |
|----------------|--------|---------------------------------------|---------------|----------------------------------------------------------------|
| `service.name` | string | Logical name of the service [1] | `Required` | ![Development](https://img.shields.io/badge/-development-blue) |
| `traceparent` | string | The trace context of current span [2] | `Recommended` | ![Development](https://img.shields.io/badge/-development-blue) |

**[1] `service.name`:** MUST be the same for all instances of horizontally scaled services. If the value was not specified, SDKs MUST fall back to `unknown_service:` concatenated with [process.executable.name](https://opentelemetry.io/docs/specs/semconv/attributes-registry/process/), e.g. `unknown_service:bash`. If `process.executable.name` is not available, the value MUST be set to `unknown_service`.

**[2] `traceparent`:** MUST be in the [text format](https://www.w3.org/TR/trace-context/#traceparent-header).

Instrumentations SHOULD make use of [SET CONTEXT_INFO](https://learn.microsoft.com/en-us/sql/t-sql/statements/set-context-info-transact-sql?view=sql-server-ver16) to add high-cardinality information(e.g. `traceparent`) as adding sql comments to queries changes their unique identifier in SQL Server.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add to the guidance here that instrumentations will need to copy over the transaction (if present) to the new set context_info command. When we didn't do this, we ran into customer issues when there was an active transaction

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate more about what copy over the transaction is actually doing? Could we just run set context_info before every query?

Copy link
Contributor

@zacharycmontoya zacharycmontoya May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the LOC I'm referencing for our C# client : https://github.com/DataDog/dd-trace-dotnet/pull/6233/files#diff-29ead36b0177265c725705ce75f5a441417f4c22e99f7080bc8ef0f486b723a7R145-R146

When creating a new set context_info command that we run before the original query, we simply copy the Transaction from the original DbCommand object to the new one. Otherwise, when executing the set context_info command we encountered a InvalidOperationException

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SET context_info must run on the same physical connection as the SQL statement it’s meant to instrument.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated


## Metrics

Microsoft SQL Server client instrumentations SHOULD collect metrics according to the general
Expand Down
Loading