From 3c66198247e23c6f66f2ae1a297264781ccac6d0 Mon Sep 17 00:00:00 2001 From: Zach Groves Date: Thu, 20 Nov 2025 17:19:58 -0500 Subject: [PATCH 01/11] remove private and beta, update to 1.37+ since 1.38 is out --- .../instrumentation/otel_instrumentation.md | 28 ++++++++----------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index 2a60e77a1e8b8..b0fe3f444025b 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -1,23 +1,18 @@ --- title: OpenTelemetry Instrumentation -is_beta: true -private: true --- -{{< callout url="#" btn_hidden="true" >}} - OpenTelemetry instrumentation for LLM Observability is in Preview. For access, contact Datadog Support. -{{< /callout >}} - ## Overview By using OpenTelemetry's standardized semantic conventions for generative AI operations, you can instrument your LLM applications with any OpenTelemetry-compatible library or framework and visualize the traces in LLM Observability. -LLM Observability supports ingesting OpenTelemetry traces that follow the [OpenTelemetry 1.37 semantic conventions for generative AI][1]. This allows you to send LLM traces directly from OpenTelemetry-instrumented applications to Datadog without requiring the Datadog LLM Observability SDK or a Datadog Agent. +LLM Observability supports ingesting OpenTelemetry traces that follow the [OpenTelemetry 1.37+ semantic conventions for generative AI][1]. This allows you to send LLM traces directly from OpenTelemetry-instrumented applications to Datadog without requiring the Datadog LLM Observability SDK or a Datadog Agent. ## Prerequisites - A [Datadog API key][2] -- An application instrumented with OpenTelemetry that emits traces following the [OpenTelemetry 1.37 semantic conventions for generative AI][1] -- Access to the OpenTelemetry instrumentation Preview feature ([contact support][4] to request access) +- An application instrumented with OpenTelemetry that emits traces following the [OpenTelemetry 1.37+ semantic conventions for generative AI][1] + +**Note**: [External evaluations][6] are not currently supported for OpenTelemetry spans. ## Setup @@ -41,30 +36,30 @@ If your framework previously supported a pre-1.37 OpenTelemetry specification ve OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental ``` -This environment variable enables version 1.37-compliant OpenTelemetry traces for frameworks that now support the version 1.37 semantic conventions, but previously supported older versions (such as [strands-agents][5]). +This environment variable enables version 1.37+-compliant OpenTelemetry traces for frameworks that now support the version 1.37+ semantic conventions, but previously supported older versions (such as [strands-agents][5]). **Note**: If you are using an OpenTelemetry library other than the default OpenTelemetry SDK, you may need to configure the endpoint, protocol, and headers differently depending on the library's API. Refer to your library's documentation for the appropriate configuration method. #### Using strands-agents -If you are using the [`strands-agents` library][5], you need to set an additional environment variable to enable traces that are compliant with OpenTelemetry v1.37: +If you are using the [`strands-agents` library][5], you need to set an additional environment variable to enable traces that are compliant with OpenTelemetry v1.37+: ``` OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental ``` -This environment variable ensures that `strands-agents` emits traces following the OpenTelemetry v1.37 semantic conventions for generative AI, which are required by LLM Observability. +This environment variable ensures that `strands-agents` emits traces following the OpenTelemetry v1.37+ semantic conventions for generative AI, which are required by LLM Observability. ### Instrumentation To generate traces compatible with LLM Observability, do one of the following: -- Use an OpenTelemetry library or instrumentation package that emits spans following the [OpenTelemetry 1.37 semantic conventions for generative AI][1]. +- Use an OpenTelemetry library or instrumentation package that emits spans following the [OpenTelemetry 1.37+ semantic conventions for generative AI][1]. - Create custom OpenTelemetry instrumentation that produces spans with the required `gen_ai.*` attributes, as defined in the semantic conventions. After your application starts sending data, the traces automatically appear in the [**LLM Observability Traces** page][3]. To search for your traces in the UI, use the `ml_app` attribute, which is automatically set to the value of your OpenTelemetry root span's `service` attribute. -
OpenInference and OpenLLMetry are not supported, as they have not been updated to support OpenTelemetry 1.37 semantic conventions for generative AI.
+
OpenInference and OpenLLMetry are not currently supported, as they have not been updated to support OpenTelemetry 1.37+ semantic conventions for generative AI.
**Note**: There may be a 3-5 minute delay between sending traces and seeing them appear on the LLM Observability Traces page. If you have APM enabled, traces appear immediately in the APM Traces page. @@ -72,7 +67,7 @@ After your application starts sending data, the traces automatically appear in t #### Using strands-agents -The following example demonstrates a complete application using strands-agents with the OpenTelemetry integration. This same approach works with any framework that supports OpenTelemetry version 1.37 semantic conventions for generative AI. +The following example demonstrates a complete application using strands-agents with the OpenTelemetry integration. This same approach works with any framework that supports OpenTelemetry version 1.37+ semantic conventions for generative AI. ```python from strands import Agent @@ -203,7 +198,7 @@ After running this example, search for `ml_app:simple-llm-example` in the LLM Ob ## Supported semantic conventions -LLM Observability supports spans that follow the OpenTelemetry 1.37 semantic conventions for generative AI, including: +LLM Observability supports spans that follow the OpenTelemetry 1.37+ semantic conventions for generative AI, including: - LLM operations with `gen_ai.provider.name`, `"gen_ai.operation.name"`, `gen_ai.request.model`, and other gen_ai attributes - Operation inputs/outputs on direct span attributes or via span events @@ -217,4 +212,5 @@ For the complete list of supported attributes and their specifications, see the [3]: https://app.datadoghq.com/llm/traces [4]: /help/ [5]: https://pypi.org/project/strands-agents/ +[6]: /llm_observability/evaluations/external_evaluations From 909a68f9a963bf883f4fb7f27671436e25b82ee5 Mon Sep 17 00:00:00 2001 From: Zach Groves Date: Thu, 20 Nov 2025 18:12:28 -0500 Subject: [PATCH 02/11] add lack of support for external evaluations to evaluations page --- .../en/llm_observability/evaluations/external_evaluations.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/en/llm_observability/evaluations/external_evaluations.md b/content/en/llm_observability/evaluations/external_evaluations.md index 94766a6070b42..12af0850bfe7d 100644 --- a/content/en/llm_observability/evaluations/external_evaluations.md +++ b/content/en/llm_observability/evaluations/external_evaluations.md @@ -21,6 +21,8 @@ further_reading: In the context of LLM applications, it's important to track user feedback and evaluate the quality of your LLM application's responses. While LLM Observability provides a few out-of-the-box evaluations for your traces, you can submit your own evaluations to LLM Observability in two ways: with Datadog's [SDK](#submitting-evaluations-with-the-sdk), or with the [LLM Observability API](#submitting-evaluations-with-the-api). Use this naming convention for the evaluation label: +**Note**: External evaluations are not currently supported for [OpenTelemetry spans][5]. + * Evaluation labels must start with a letter. * Evaluation labels must only contain ASCII alphanumerics or underscores. * Other characters, including spaces, are converted to underscores. @@ -104,3 +106,4 @@ You can use the evaluations API provided by LLM Observability to send evaluation [2]: /llm_observability/setup/api/?tab=model#evaluations-api [3]: /llm_observability/setup/sdk/python/#evaluations [4]: /llm_observability/setup/sdk/nodejs/#evaluations +[5]: /llm_observability/instrumentation/otel_instrumentation From 59909f6b91202d1de8a23c7dbdf25caf15bc364f Mon Sep 17 00:00:00 2001 From: Zachary Groves <32471391+ZStriker19@users.noreply.github.com> Date: Fri, 21 Nov 2025 11:31:21 -0500 Subject: [PATCH 03/11] Update content/en/llm_observability/evaluations/external_evaluations.md Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com> --- .../en/llm_observability/evaluations/external_evaluations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/evaluations/external_evaluations.md b/content/en/llm_observability/evaluations/external_evaluations.md index 12af0850bfe7d..7929d23a36b80 100644 --- a/content/en/llm_observability/evaluations/external_evaluations.md +++ b/content/en/llm_observability/evaluations/external_evaluations.md @@ -21,7 +21,7 @@ further_reading: In the context of LLM applications, it's important to track user feedback and evaluate the quality of your LLM application's responses. While LLM Observability provides a few out-of-the-box evaluations for your traces, you can submit your own evaluations to LLM Observability in two ways: with Datadog's [SDK](#submitting-evaluations-with-the-sdk), or with the [LLM Observability API](#submitting-evaluations-with-the-api). Use this naming convention for the evaluation label: -**Note**: External evaluations are not currently supported for [OpenTelemetry spans][5]. +**Note**: External evaluations are not supported for [OpenTelemetry spans][5]. * Evaluation labels must start with a letter. * Evaluation labels must only contain ASCII alphanumerics or underscores. From b2f30c49ffc0176d15ec9fadfda10ae371bf2d8a Mon Sep 17 00:00:00 2001 From: Zachary Groves <32471391+ZStriker19@users.noreply.github.com> Date: Fri, 21 Nov 2025 11:31:28 -0500 Subject: [PATCH 04/11] Update content/en/llm_observability/instrumentation/otel_instrumentation.md Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com> --- .../llm_observability/instrumentation/otel_instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index b0fe3f444025b..1ecbf7b13ca6f 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -12,7 +12,7 @@ LLM Observability supports ingesting OpenTelemetry traces that follow the [OpenT - A [Datadog API key][2] - An application instrumented with OpenTelemetry that emits traces following the [OpenTelemetry 1.37+ semantic conventions for generative AI][1] -**Note**: [External evaluations][6] are not currently supported for OpenTelemetry spans. +**Note**: [External evaluations][6] are not supported for OpenTelemetry spans. ## Setup From 0f114bbe0a8b6da430ac45fa11befa75f1e71dca Mon Sep 17 00:00:00 2001 From: Zachary Groves <32471391+ZStriker19@users.noreply.github.com> Date: Fri, 21 Nov 2025 11:31:35 -0500 Subject: [PATCH 05/11] Update content/en/llm_observability/instrumentation/otel_instrumentation.md Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com> --- .../llm_observability/instrumentation/otel_instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index 1ecbf7b13ca6f..5910ed6f3b063 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -59,7 +59,7 @@ To generate traces compatible with LLM Observability, do one of the following: After your application starts sending data, the traces automatically appear in the [**LLM Observability Traces** page][3]. To search for your traces in the UI, use the `ml_app` attribute, which is automatically set to the value of your OpenTelemetry root span's `service` attribute. -
OpenInference and OpenLLMetry are not currently supported, as they have not been updated to support OpenTelemetry 1.37+ semantic conventions for generative AI.
+
OpenInference and OpenLLMetry are not supported, as they have not been updated to support OpenTelemetry 1.37+ semantic conventions for generative AI.
**Note**: There may be a 3-5 minute delay between sending traces and seeing them appear on the LLM Observability Traces page. If you have APM enabled, traces appear immediately in the APM Traces page. From 2ace09b795961578724a7bf21d4407a603b83c1e Mon Sep 17 00:00:00 2001 From: Zachary Groves <32471391+ZStriker19@users.noreply.github.com> Date: Fri, 21 Nov 2025 11:32:46 -0500 Subject: [PATCH 06/11] Update content/en/llm_observability/instrumentation/otel_instrumentation.md Co-authored-by: Barry Eom <31739208+barieom@users.noreply.github.com> --- .../llm_observability/instrumentation/otel_instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index 5910ed6f3b063..6818f56d621c0 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -65,7 +65,7 @@ After your application starts sending data, the traces automatically appear in t ### Examples -#### Using strands-agents +#### Using Strands Agents The following example demonstrates a complete application using strands-agents with the OpenTelemetry integration. This same approach works with any framework that supports OpenTelemetry version 1.37+ semantic conventions for generative AI. From f4e2a660ab667896dcf577c88182d125a81d3039 Mon Sep 17 00:00:00 2001 From: Zachary Groves <32471391+ZStriker19@users.noreply.github.com> Date: Fri, 21 Nov 2025 11:33:01 -0500 Subject: [PATCH 07/11] Update content/en/llm_observability/instrumentation/otel_instrumentation.md Co-authored-by: Barry Eom <31739208+barieom@users.noreply.github.com> --- .../llm_observability/instrumentation/otel_instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index 6818f56d621c0..8d8a49e42242b 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -67,7 +67,7 @@ After your application starts sending data, the traces automatically appear in t #### Using Strands Agents -The following example demonstrates a complete application using strands-agents with the OpenTelemetry integration. This same approach works with any framework that supports OpenTelemetry version 1.37+ semantic conventions for generative AI. +The following example demonstrates a complete application using [Strands Agents][7] with the OpenTelemetry integration. This same approach works with any framework that supports OpenTelemetry version 1.37+ semantic conventions for generative AI. ```python from strands import Agent From 047bb8fea5b689273f7f677bce7153a9e7c1f234 Mon Sep 17 00:00:00 2001 From: Zach Groves Date: Fri, 21 Nov 2025 11:52:19 -0500 Subject: [PATCH 08/11] resolve merge conflict --- .../evaluations/external_evaluations.md | 9 ++++++--- .../instrumentation/otel_instrumentation.md | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/content/en/llm_observability/evaluations/external_evaluations.md b/content/en/llm_observability/evaluations/external_evaluations.md index 7929d23a36b80..cadad7ee83369 100644 --- a/content/en/llm_observability/evaluations/external_evaluations.md +++ b/content/en/llm_observability/evaluations/external_evaluations.md @@ -21,15 +21,18 @@ further_reading: In the context of LLM applications, it's important to track user feedback and evaluate the quality of your LLM application's responses. While LLM Observability provides a few out-of-the-box evaluations for your traces, you can submit your own evaluations to LLM Observability in two ways: with Datadog's [SDK](#submitting-evaluations-with-the-sdk), or with the [LLM Observability API](#submitting-evaluations-with-the-api). Use this naming convention for the evaluation label: -**Note**: External evaluations are not supported for [OpenTelemetry spans][5]. - * Evaluation labels must start with a letter. * Evaluation labels must only contain ASCII alphanumerics or underscores. * Other characters, including spaces, are converted to underscores. * Unicode is not supported. * Evaluation labels must not exceed 200 characters. Fewer than 100 is preferred from a UI perspective. -
Evaluation labels must be unique for a given LLM application (ml_app) and organization.
+
+ +- Evaluation labels must be unique for a given LLM application (ml_app) and organization. +- External evaluations are not currently supported for [OpenTelemetry spans][5]. + +
## Submitting external evaluations with the SDK diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index 8d8a49e42242b..d713f1fcc24a9 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -12,7 +12,7 @@ LLM Observability supports ingesting OpenTelemetry traces that follow the [OpenT - A [Datadog API key][2] - An application instrumented with OpenTelemetry that emits traces following the [OpenTelemetry 1.37+ semantic conventions for generative AI][1] -**Note**: [External evaluations][6] are not supported for OpenTelemetry spans. +
[External evaluations][6] in LLM Observability are not currently applied to OpenTelemetry spans. Evaluations are only available for spans generated with the Datadog LLM Observability SDK or submitted directly to the HTTP API intake.
## Setup From 5425af8db44608c442828c2185ae98d4a5e5318c Mon Sep 17 00:00:00 2001 From: Zach Groves Date: Fri, 21 Nov 2025 11:58:03 -0500 Subject: [PATCH 09/11] add strands url, put warnings in banners --- .../instrumentation/otel_instrumentation.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index d713f1fcc24a9..b5f403c8d6f0c 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -59,9 +59,12 @@ To generate traces compatible with LLM Observability, do one of the following: After your application starts sending data, the traces automatically appear in the [**LLM Observability Traces** page][3]. To search for your traces in the UI, use the `ml_app` attribute, which is automatically set to the value of your OpenTelemetry root span's `service` attribute. -
OpenInference and OpenLLMetry are not supported, as they have not been updated to support OpenTelemetry 1.37+ semantic conventions for generative AI.
+
-**Note**: There may be a 3-5 minute delay between sending traces and seeing them appear on the LLM Observability Traces page. If you have APM enabled, traces appear immediately in the APM Traces page. +- OpenInference and OpenLLMetry are not supported, as they have not been updated to support OpenTelemetry 1.37+ semantic conventions for generative AI. +- There may be a 3-5 minute delay between sending traces and seeing them appear on the LLM Observability Traces page. If you have APM enabled, traces appear immediately in the APM Traces page. + +
### Examples @@ -213,4 +216,5 @@ For the complete list of supported attributes and their specifications, see the [4]: /help/ [5]: https://pypi.org/project/strands-agents/ [6]: /llm_observability/evaluations/external_evaluations +[7]: https://strandsagents.com/latest/ From ffb376958bc4ccc02d99be3b1d8b6bcde8855536 Mon Sep 17 00:00:00 2001 From: Zachary Groves <32471391+ZStriker19@users.noreply.github.com> Date: Fri, 21 Nov 2025 17:10:32 -0500 Subject: [PATCH 10/11] Update content/en/llm_observability/instrumentation/otel_instrumentation.md Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com> --- .../llm_observability/instrumentation/otel_instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/instrumentation/otel_instrumentation.md b/content/en/llm_observability/instrumentation/otel_instrumentation.md index b5f403c8d6f0c..685cae372c3cb 100644 --- a/content/en/llm_observability/instrumentation/otel_instrumentation.md +++ b/content/en/llm_observability/instrumentation/otel_instrumentation.md @@ -12,7 +12,7 @@ LLM Observability supports ingesting OpenTelemetry traces that follow the [OpenT - A [Datadog API key][2] - An application instrumented with OpenTelemetry that emits traces following the [OpenTelemetry 1.37+ semantic conventions for generative AI][1] -
[External evaluations][6] in LLM Observability are not currently applied to OpenTelemetry spans. Evaluations are only available for spans generated with the Datadog LLM Observability SDK or submitted directly to the HTTP API intake.
+
External evaluations in LLM Observability are not applied to OpenTelemetry spans. Evaluations are only available for spans generated with the Datadog LLM Observability SDK or submitted directly to the HTTP API intake.
## Setup From 5d6d6b2cd6d6de50b40f011e176e011f023d49aa Mon Sep 17 00:00:00 2001 From: Zachary Groves <32471391+ZStriker19@users.noreply.github.com> Date: Fri, 21 Nov 2025 17:11:06 -0500 Subject: [PATCH 11/11] Update content/en/llm_observability/evaluations/external_evaluations.md Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com> --- .../en/llm_observability/evaluations/external_evaluations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/evaluations/external_evaluations.md b/content/en/llm_observability/evaluations/external_evaluations.md index cadad7ee83369..3f64f0e81d210 100644 --- a/content/en/llm_observability/evaluations/external_evaluations.md +++ b/content/en/llm_observability/evaluations/external_evaluations.md @@ -30,7 +30,7 @@ While LLM Observability provides a few out-of-the-box evaluations for your trace
- Evaluation labels must be unique for a given LLM application (ml_app) and organization. -- External evaluations are not currently supported for [OpenTelemetry spans][5]. +- External evaluations are not supported for [OpenTelemetry spans][5].