3.16.1

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix resolves an issue where tests would be incorrectly detected as third-party code if a third-party package containing a folder with the same name as the tests folder was installed. For instance, the sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.

• grpc: This fix resolves an issue where the internal span was left active in the caller when using the future interface.

• ray: This fix resolves an issue where submitting Ray jobs caused an AttributeError crash in certain configurations.

• ray: This fix resolves an issue where long-running spans did not preserve the correct resource name when being recreated.

• ray: This fix resolves an issue where long-running job spans could remain unfinished when an exception occurred during job submission.

• AAP: This PR is a tentative fix for rare memory problems with libddwaf that we were unable to reproduce for now.

• Internal: Fix some modules being unloaded too soon when using pytest + ddtrace + gevent.

3.16.0

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• This change updates library injection logic to work under Python 3.14.
• This change adds support and tests for Python 3.14 to much of the library's functionality. The following products and integrations still do not work with Python 3.14:
  • Profiling
  • IAST
  • datastreams
  • ci_visibility
  • pytest
  • django - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django_hosts - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • djangorestframework - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django:celery - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • dramatiq - dramatiq doesn't yet have a release supporting 3.14
  • grpc_aio - some tests in the suite don't work with pytest-asyncio >= 1.0
  • rq - rq doesn't work with python 3.14
  • sqlite3 - pysqlite3-binary doesn't yet support python 3.14
  • opentelemetry - opentelemetry-exporter-otlp doesn't yet work with Python 3.14
  • openai - tiktoken doesn't yet work with Python 3.14
  • ai_guard_langchain - tiktoken doesn't yet work with Python 3.14
  • openai_agents
  • langchain
  • langgraph - tiktoken doesn't yet work with Python 3.14
  • litellm - tiktoken doesn't yet work with Python 3.14
  • google_generativeai - protobuf doesn't yet work with Python 3.14
  • vertexai
  • crewai - tiktoken doesn't yet work with Python 3.14
  • ray - ray doesn't yet work with Python 3.14
  • kafka - confluent-kafka doesn't yet work with Python 3.14
  • aws_lambda - datadog-lambda doesn't yet work with Python 3.14
  • llmobs - ragas doesn't yet work with Python 3.14
  • appsec_integrations_fastapi

Deprecation Notes

• vertica: The vertica integration is deprecated and will be removed in a future version, around the same time that ddtrace drops support for Python 3.9.

New Features

• opentelemetry: Adds default configurations for the OpenTelemetry Metrics API implementation to improve the Datadog user experience. This includes the following configurations:

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to the default Datadog Agent endpoint, or localhost if not found
  • OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set to delta
  • OTEL_METRIC_EXPORT_INTERVAL is set to 10000
  • OTEL_METRIC_EXPORT_TIMEOUT is set to 7500

• LLM Observability: MCP integration also traces ClientSession contexts, ClientSession.initialize, and ClientSession.list_tools.

• ray: This introduces a Ray core integration that traces Ray jobs, remote tasks, and actor method calls. Supported for Ray >= 2.46.0.

  To enable tracing, start the Ray head with --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing then submit jobs as usual.

Bug Fixes

• AAP: This fix resolves an issue where stream endpoints with daphne/django where unresponsive due to an asyncio error.
• CI Visibility: This fix resolves an issue where code imported at module level but not executed during a test would not be considered by Test Impact Analysis as impacting the test. For example, a test using a constant imported from some other module would not count the constant definition among its impacting lines, because the constant definition is not executed during the test, but rather when the module was imported. With this change, code executed at import time is also included among the impacted lines of a test.
• google-adk: Fixes an AttributeError that could occur when tracing Google ADK agent runs, due to the agent model attribute not being defined for SequentialAgent class.
• opentelemetry: Fixes the parsing of OTLP metrics exporter configurations and the operation to automatically append the v1/metrics path to HTTP OTLP endpoints.
• langchain: Resolves an issue where langchain patching would throw an ImportError for when using langchain_core>=0.3.76.
• LLM Observability
  • ensures APM is disabled when DD_APM_TRACING_ENABLED=0 when using LLM Observability.
  • Resolves an issue where model IDs were not being parsed correctly if the model ID was an inference profile ID in the bedrock integration.
  • enable the backend to differentiate AI Obs spans from other DJM spans, so that customers are not billed for AI Observability spans as part of their APM bill.

Other Changes

• sampling: Add more debug logs to help debug sampling issues.

3.16.0rc2

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• This change updates library injection logic to work under Python 3.14.
• This change adds support and tests for Python 3.14 to much of the library's functionality. The following products and integrations still do not work with Python 3.14:
  • Profiling
  • IAST
  • datastreams
  • ci_visibility
  • pytest
  • django - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django_hosts - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • djangorestframework - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django:celery - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • dramatiq - dramatiq doesn't yet have a release supporting 3.14
  • grpc_aio - some tests in the suite don't work with pytest-asyncio >= 1.0
  • rq - rq doesn't work with python 3.14
  • sqlite3 - pysqlite3-binary doesn't yet support python 3.14
  • opentelemetry - opentelemetry-exporter-otlp doesn't yet work with Python 3.14
  • openai - tiktoken doesn't yet work with Python 3.14
  • ai_guard_langchain - tiktoken doesn't yet work with Python 3.14
  • openai_agents
  • langchain
  • langgraph - tiktoken doesn't yet work with Python 3.14
  • litellm - tiktoken doesn't yet work with Python 3.14
  • google_generativeai - protobuf doesn't yet work with Python 3.14
  • vertexai
  • crewai - tiktoken doesn't yet work with Python 3.14
  • ray - ray doesn't yet work with Python 3.14
  • kafka - confluent-kafka doesn't yet work with Python 3.14
  • aws_lambda - datadog-lambda doesn't yet work with Python 3.14
  • llmobs - ragas doesn't yet work with Python 3.14
  • appsec_integrations_fastapi

Deprecation Notes

• vertica: The vertica integration is deprecated and will be removed in a future version, around the same time that ddtrace drops support for Python 3.9.

New Features

• opentelemetry: Adds default configurations for the OpenTelemetry Metrics API implementation to improve the Datadog user experience. This includes the following configurations:

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to the default Datadog Agent endpoint, or localhost if not found
  • OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set to delta
  • OTEL_METRIC_EXPORT_INTERVAL is set to 10000
  • OTEL_METRIC_EXPORT_TIMEOUT is set to 7500

• LLM Observability: MCP integration also traces ClientSession contexts, ClientSession.initialize, and ClientSession.list_tools.

• ray: This introduces a Ray core integration that traces Ray jobs, remote tasks, and actor method calls. Supported for Ray >= 2.46.0.

  To enable tracing, start the Ray head with --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing then submit jobs as usual.

Bug Fixes

• AAP: This fix resolves an issue where stream endpoints with daphne/django where unresponsive due to an asyncio error.
• CI Visibility: This fix resolves an issue where code imported at module level but not executed during a test would not be considered by Test Impact Analysis as impacting the test. For example, a test using a constant imported from some other module would not count the constant definition among its impacting lines, because the constant definition is not executed during the test, but rather when the module was imported. With this change, code executed at import time is also included among the impacted lines of a test.
• google-adk: Fixes an AttributeError that could occur when tracing Google ADK agent runs, due to the agent model attribute not being defined for SequentialAgent class.
• opentelemetry: Fixes the parsing of OTLP metrics exporter configurations and the operation to automatically append the v1/metrics path to HTTP OTLP endpoints.
• langchain: Resolves an issue where langchain patching would throw an ImportError for when using langchain_core>=0.3.76.
• LLM Observability
  • ensures APM is disabled when DD_APM_TRACING_ENABLED=0 when using LLM Observability.
  • Resolves an issue where model IDs were not being parsed correctly if the model ID was an inference profile ID in the bedrock integration.
  • enable the backend to differentiate AI Obs spans from other DJM spans, so that customers are not billed for AI Observability spans as part of their APM bill.

Other Changes

• sampling: Add more debug logs to help debug sampling issues.

3.16.0rc1

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• This change updates library injection logic to work under Python 3.14.
• This change adds support and tests for Python 3.14 to much of the library's functionality. The following products and integrations still do not work with Python 3.14:
  • Profiling
  • IAST
  • datastreams
  • ci_visibility
  • pytest
  • django - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django_hosts - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • djangorestframework - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django:celery - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • dramatiq - dramatiq doesn't yet have a release supporting 3.14
  • grpc_aio - some tests in the suite don't work with pytest-asyncio >= 1.0
  • rq - rq doesn't work with python 3.14
  • sqlite3 - pysqlite3-binary doesn't yet support python 3.14
  • opentelemetry - opentelemetry-exporter-otlp doesn't yet work with Python 3.14
  • openai - tiktoken doesn't yet work with Python 3.14
  • ai_guard_langchain - tiktoken doesn't yet work with Python 3.14
  • openai_agents
  • langchain
  • langgraph - tiktoken doesn't yet work with Python 3.14
  • litellm - tiktoken doesn't yet work with Python 3.14
  • google_generativeai - protobuf doesn't yet work with Python 3.14
  • vertexai
  • crewai - tiktoken doesn't yet work with Python 3.14
  • ray - ray doesn't yet work with Python 3.14
  • kafka - confluent-kafka doesn't yet work with Python 3.14
  • aws_lambda - datadog-lambda doesn't yet work with Python 3.14
  • llmobs - ragas doesn't yet work with Python 3.14
  • appsec_integrations_fastapi

Deprecation Notes

• vertica: The vertica integration is deprecated and will be removed in a future version, around the same time that ddtrace drops support for Python 3.9.

New Features

• opentelemetry: Adds default configurations for the OpenTelemetry Metrics API implementation to improve the Datadog user experience. This includes the following configurations:

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to the default Datadog Agent endpoint, or localhost if not found
  • OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set to delta
  • OTEL_METRIC_EXPORT_INTERVAL is set to 10000
  • OTEL_METRIC_EXPORT_TIMEOUT is set to 7500

• LLM Observability: MCP integration also traces ClientSession contexts, ClientSession.initialize, and ClientSession.list_tools.

• ray: This introduces a Ray core integration that traces Ray jobs, remote tasks, and actor method calls. Supported for Ray >= 2.46.0.

  To enable tracing, start the Ray head with --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing then submit jobs as usual.

Bug Fixes

• AAP: This fix resolves an issue where stream endpoints with daphne/django where unresponsive due to an asyncio error.
• CI Visibility: This fix resolves an issue where code imported at module level but not executed during a test would not be considered by Test Impact Analysis as impacting the test. For example, a test using a constant imported from some other module would not count the constant definition among its impacting lines, because the constant definition is not executed during the test, but rather when the module was imported. With this change, code executed at import time is also included among the impacted lines of a test.
• google-adk: Fixes an AttributeError that could occur when tracing Google ADK agent runs, due to the agent model attribute not being defined for SequentialAgent class.
• llmobs: Resolves an issue where model IDs were not being parsed correctly if the model ID was an inference profile ID in the bedrock integration.
• opentelemetry: Fixes the parsing of OTLP metrics exporter configurations and the operation to automatically append the v1/metrics path to HTTP OTLP endpoints.
• langchain: Resolves an issue where langchain patching would throw an ImportError for when using langchain_core>=0.3.76.
• LLM Observability: ensures APM is disabled when DD_APM_TRACING_ENABLED=0 when using LLM Observability.

Other Changes

• sampling: Add more debug logs to help debug sampling issues.

3.14.4

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• AAP:

  • This fix resolves an issue where stream endpoints with daphne/django where unresponsive due to an asyncio error.
  • This fix resolves an issue where the endpoint discovery feature could generate a crash for flask at startup.

• CI Visibility: This fix solves an issue where the ITR skip count metric was aggregating skipped tests even when skipping level was set to suite. It will now count appropriately (skipped suites or skipped tests) depending on ITR skip level.

3.15.0

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

New Features

• google-adk: Adds APM tracing and LLM Observability support for the Google ADK library (google-adk). Support includes APM tracing and LLM Observability for agent runs, tool calls, and code execution.
• django: This introduces the DD_DJANGO_TRACING_MINIMAL environment variable for performance-sensitive applications. When enabled, this disables Django ORM, cache, and template instrumentation while keeping middleware instrumentation enabled. This significantly reduces overhead by removing Django-specific spans while preserving visibility into the underlying database drivers, cache clients, and other integrations. For example, with this enabled, Django ORM query spans are disabled but database driver spans (e.g., psycopg, MySQLdb) will still be created. To enable minimal tracing, set DD_DJANGO_TRACING_MINIMAL=true.
• AWS: adds aws.partition tag onto AWS traces based on the region for the boto, botocore, and aiobotocore integrations.
• AAP: This extends downstream request analysis (API10) to the requests package. Previously, downstream request analysis was only supported in the standard cpython api (urllib).
• dynamic instrumentation/exception replay/code origin for spans: added support for the latest Datadog agent intake for snapshots. This requires a minimum agent version of 7.49.0.
• CI Visibility: This introduces the env var DD_CIVISIBILITY_ENABLED (with default value True) so it can be disabled to avoid sending traces to the Test Visibility product from the test runners.
• azure_servicebus: Add distributed tracing support for sending batches with Azure Service Bus producers.
• azure_functions: Use span links to connect Service Bus trigger consumers to the producers that send the messages.
• tracing: Added support for resource renaming, an experimental feature that lets the Datadog platform adjust the resource field on web request spans when the endpoint cannot be correctly deduced. Enable the feature by setting DD_TRACE_RESOURCE_RENAMING_ENABLED="true"
• Code Security (IAST)
  • Untrusted Serialization detection, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
  • Reduce false positives if md5 or sha1 functions have the parameter usedforsecurity=False.
• LLM Observability: Extends the prompt structure to add tags and chat_template, and a new Prompt TypedDict class that would be used in annotation and annotation_context.
• LLM Observability: Datasets & Experiments SDK now has summary evaluators support.

Bug Fixes

• CI Visibility: This fix solves an issue where the ITR skip count metric was aggregating skipped tests even when skipping level was set to suite. It will now count appropriately (skipped suites or skipped tests) depending on ITR skip level.
• sampling: This change prevents the DatadogSampler from getting recreated whenever the SpanAggregator is reset, and instead updates the rate limiter that the sampler uses.
• dynamic instrumentation: fix an issue that prevented multiple probes on the same location from being instrumented.
• exception replay
  • prevent Celery from crashing when a task raises a custom exception with mandatory arguments.
  • ensure that value capture starts from the leaf frame of the innermost exception.
• tracing: Fixes encoding bytes objects as span attributes by truncating byte string, rather than throwing PyErr_Format.
• AAP
  • This fix resolves an issue where the endpoint discovery feature could generate a crash for flask at startup.
  • This fix disables grpc threat monitoring, as it could generate false positives.
• libinjection: allow python module executed with -m entries in the denylist.
• profiling
  • Upgrades echion to resolve segmentation faults that can happen on services with a lot of asyncio.Tasks.
  • Fix crash in memory profiling when garbage collection is triggered while sampling a PyObject_Realloc call, which can lead to accessing freed memory.
  • Profiling won't load if --skip-atexit is not set when --lazy or --lazy-apps is set on uWSGI<2.0.30. This is to prevent crashes from profiling native extension modules. See unbit/uwsgi#2726 for details.
• RemoteConfig: Fixes an issue introduced in Python 3.13 where creating a shared array with the c_char type raised a TypeError, this now uses the 'c' typecode for better compatibility across versions.
• source code integration: check that DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.

3.15.0rc1

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

New Features

• google-adk: Adds APM tracing and LLM Observability support for the Google ADK library (google-adk). Support includes APM tracing and LLM Observability for agent runs, tool calls, and code execution.
• django: This introduces the DD_DJANGO_TRACING_MINIMAL environment variable for performance-sensitive applications. When enabled, this disables Django ORM, cache, and template instrumentation while keeping middleware instrumentation enabled. This significantly reduces overhead by removing Django-specific spans while preserving visibility into the underlying database drivers, cache clients, and other integrations. For example, with this enabled, Django ORM query spans are disabled but database driver spans (e.g., psycopg, MySQLdb) will still be created. To enable minimal tracing, set DD_DJANGO_TRACING_MINIMAL=true.
• AWS: adds aws.partition tag onto AWS traces based on the region for the boto, botocore, and aiobotocore integrations.
• AAP: This extends downstream request analysis (API10) to the requests package. Previously, downstream request analysis was only supported in the standard cpython api (urllib).
• dynamic instrumentation/exception replay/code origin for spans: added support for the latest Datadog agent intake for snapshots. This requires a minimum agent version of 7.49.0.
• CI Visibility: This introduces the env var DD_CIVISIBILITY_ENABLED (with default value True) so it can be disabled to avoid sending traces to the Test Visibility product from the test runners.
• azure_servicebus: Add distributed tracing support for sending batches with Azure Service Bus producers.
• azure_functions: Use span links to connect Service Bus trigger consumers to the producers that send the messages.
• tracing: Added support for resource renaming, an experimental feature that lets the Datadog platform adjust the resource field on web request spans when the endpoint cannot be correctly deduced. Enable the feature by setting DD_TRACE_RESOURCE_RENAMING_ENABLED="true"
• Code Security (IAST)
  • Untrusted Serialization detection, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
  • Reduce false positives if md5 or sha1 functions have the parameter usedforsecurity=False.
• LLM Observability: Extends the prompt structure to add tags and chat_template, and a new Prompt TypedDict class that would be used in annotation and annotation_context.

Bug Fixes

• CI Visibility: This fix solves an issue where the ITR skip count metric was aggregating skipped tests even when skipping level was set to suite. It will now count appropriately (skipped suites or skipped tests) depending on ITR skip level.
• sampling: This change prevents the DatadogSampler from getting recreated whenever the SpanAggregator is reset, and instead updates the rate limiter that the sampler uses.
• dynamic instrumentation: fix an issue that prevented multiple probes on the same location from being instrumented.
• exception replay
  • prevent Celery from crashing when a task raises a custom exception with mandatory arguments.
  • ensure that value capture starts from the leaf frame of the innermost exception.
• tracing: Fixes encoding bytes objects as span attributes by truncating byte string, rather than throwing PyErr_Format.
• AAP
  • This fix resolves an issue where the endpoint discovery feature could generate a crash for flask at startup.
  • This fix disables grpc threat monitoring, as it could generate false positives.
• libinjection: allow python module executed with -m entries in the denylist.
• profiling
  • Upgrades echion to resolve segmentation faults that can happen on services with a lot of asyncio.Tasks.
  • Fix crash in memory profiling when garbage collection is triggered while sampling a PyObject_Realloc call, which can lead to accessing freed memory.
  • Profiling won't load if --skip-atexit is not set when --lazy or --lazy-apps is set on uWSGI<2.0.30. This is to prevent crashes from profiling native extension modules. See unbit/uwsgi#2726 for details.
• RemoteConfig: Fixes an issue introduced in Python 3.13 where creating a shared array with the c_char type raised a TypeError, this now uses the 'c' typecode for better compatibility across versions.
• source code integration: check that DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.

3.14.3

Estimated end-of-life date, accurate to within three months: 09-2026
See the support level definitions for more information.

Bug Fixes

• exception replay: prevent Celery from crashing when a task raises a custom exception with mandatory arguments.

• tracing: Fixes encoding bytes objects as span attributes by truncating byte string, rather than throwing PyErr_Format.

• libinjection: allow deny listing python modules executed with python -m and deny py_compile.

3.13.4

Estimated end-of-life date, accurate to within three months: 09-2026
See the support level definitions for more information.

Bug Fixes

• tracing: Fixes encoding bytes objects as span attributes by truncating byte string, rather than throwing PyErr_Format.

• libinjection: allow deny listing python modules executed with python -m and deny py_compile.

3.12.9

Estimated end-of-life date, accurate to within three months: 09-2026
See the support level definitions for more information.

Bug Fixes

• libinjection: allow deny listing python modules executed with python -m and deny py_compile.