3.17.3

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix addresses a performance issue where repository tags were fetched during the unshallow process to extract commit metadata, causing slowdowns in repositories with many tags.

• LLM Observability: Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.

• opentelemetry:

  • Fixed circular import when enabling multiple OpenTelemetry signals (metrics + logs) simultaneously.
  • Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.

• ray: This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().

• tracer: This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0

3.18.0

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• openai: Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
  will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.

Deprecation Notes

• tracing
  • Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.set_tag_str is deprecated and will be removed in version 4.0.0.
    As an alternative to Span.set_tag_str, you can use Span.set_tag instead.
• profiling: The V1 stack profiler is deprecated and will be removed in 4.0. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.

New Features

• AAP
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling: Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+
• kafka: This fix resolves an issue where only the first message in a batch was dispatched to Data Streams Monitoring (DSM) when consuming multiple Kafka messages
• langchain
  • This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.
  • Fixes an issue where streamed responses that end before the first chunk is received would result in an IndexError.
• LLM Observability
  • Corrected the description of the assessment argument in submit_evaluation().
    assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
    The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.
• openai: This fix resolves an issue where using async iteration with paginated methods (e.g., async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.
• pytest plugin: fix for potential KeyError exceptions in test runs when gevent is detected within the environment.
• code origin: ensure that code location information is added to entry spans when Code Origin is enabled remotely.
• ray
  • This fix resolves an issue where exceptions raised in Ray child spans were not properly recorded in the trace.
  • This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().
  • This fix stops instrumenting internal Ray actors (those starting with underscore) that were causing excessive noise, and adds ray.data._internal to the module denylist.
• IAST: Fixed an issue where using weak hashing or cipher algorithms outside of a request context (e.g., during application startup) could raise an unhandled exception. The fix ensures proper error handling when IAST operations are performed without an active request context.
• otel
  • Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.
  • Fixed circular import when enabling multiple OpenTelemetry signals (metrics + logs) simultaneously.
• Fix a potential race condition in the tracer.
• profiling
  • This fix resolves an issue where AssertionError exceptions were silently suppressed in the _acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)
  • Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.
• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.
• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.
• tracer
  • This fix ensures compatibility with wrapt 2.0.0
  • This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0

3.18.0rc2

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• openai: Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
  will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.

Deprecation Notes

• tracing
  • Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.set_tag_str is deprecated and will be removed in version 4.0.0.
    As an alternative to Span.set_tag_str, you can use Span.set_tag instead.
• profiling: The V1 stack profiler is deprecated and will be removed in 4.0. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.

New Features

• AAP
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling: Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+
• kafka: This fix resolves an issue where only the first message in a batch was dispatched to Data Streams Monitoring (DSM) when consuming multiple Kafka messages
• langchain
  • This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.
  • Fixes an issue where streamed responses that end before the first chunk is received would result in an IndexError.
• LLM Observability
  • Corrected the description of the assessment argument in submit_evaluation().
    assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
    The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.
• openai: This fix resolves an issue where using async iteration with paginated methods (e.g., async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.
• pytest plugin: fix for potential KeyError exceptions in test runs when gevent is detected within the environment.
• code origin: ensure that code location information is added to entry spans when Code Origin is enabled remotely.
• ray
  • This fix resolves an issue where exceptions raised in Ray child spans were not properly recorded in the trace.
  • This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().
  • This fix stops instrumenting internal Ray actors (those starting with underscore) that were causing excessive noise, and adds ray.data._internal to the module denylist.
• IAST: Fixed an issue where using weak hashing or cipher algorithms outside of a request context (e.g., during application startup) could raise an unhandled exception. The fix ensures proper error handling when IAST operations are performed without an active request context.
• otel
  • Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.
  • Fixed circular import when enabling multiple OpenTelemetry signals (metrics + logs) simultaneously.
• Fix a potential race condition in the tracer.
• profiling
  • This fix resolves an issue where AssertionError exceptions were silently suppressed in the _acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)
  • Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.
• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.
• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.
• tracer
  • This fix ensures compatibility with wrapt 2.0.0
  • This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0

3.18.0rc1

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Deprecation Notes

• tracing
  • Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.set_tag_str is deprecated and will be removed in version 4.0.0.
    As an alternative to Span.set_tag_str, you can use Span.set_tag instead.
• profiling: The V1 stack profiler is deprecated and will be removed in 4.0. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.

New Features

• AAP
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling: Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+
• kafka: This fix resolves an issue where only the first message in a batch was dispatched to Data Streams Monitoring (DSM) when consuming multiple Kafka messages
• langchain
  • This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.
  • Fixes an issue where streamed responses that end before the first chunk is received would result in an IndexError.
• LLM Observability
  • Corrected the description of the assessment argument in submit_evaluation().
    assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
    The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.
• openai: This fix resolves an issue where using async iteration with paginated methods (e.g., async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.
• pytest plugin: fix for potential KeyError exceptions in test runs when gevent is detected within the environment.
• code origin: ensure that code location information is added to entry spans when Code Origin is enabled remotely.
• ray
  • This fix resolves an issue where exceptions raised in Ray child spans were not properly recorded in the trace.
  • This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().
  • This fix stops instrumenting internal Ray actors (those starting with underscore) that were causing excessive noise, and adds ray.data._internal to the module denylist.
• IAST: Fixed an issue where using weak hashing or cipher algorithms outside of a request context (e.g., during application startup) could raise an unhandled exception. The fix ensures proper error handling when IAST operations are performed without an active request context.
• otel: Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.
• Fix a potential race condition in the tracer.
• profiling
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)
  • Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.
• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.
• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.
• tracer: This fix ensures compatibility with wrapt 2.0.0

3.17.2

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• langchain: This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.

• openai: This fix resolves an issue where using async iteration with paginated methods (e.g., async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.

• langchain: Fixes an issue where streamed responses that end before the first chunk is received would result in an IndexError.

• Fix a potential race condition in the tracer.

• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.

• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.

3.16.4

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• langchain: This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.

• Fix a potential race condition in the tracer.

• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.

• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.

3.17.1

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+

• LLM Observability

  • Corrected the description of the assessment argument in submit_evaluation().
    assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled. The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.

• IAST: Fixed an issue where using weak hashing or cipher algorithms outside of a request context (e.g., during application startup) could raise an unhandled exception. The fix ensures proper error handling when IAST operations are performed without an active request context.

• profiling: Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.

3.16.3

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+

• LLM Observability: Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled. The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.

• profiling: Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.

2.21.12

Estimated end-of-life date, accurate to within three months: 11-2025
See the support level definitions for more information.

Bug Fixes

• Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.

3.17.0

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• LLM Observability: Experiments can now be created to be stored under a different project from the project defined in LLMObs.enable

Deprecation Notes

• LLM Observability: LLMObs.submit_evaluation_for() has been deprecated and will be removed in a future version. It will be replaced with LLMObs.submit_evaluation() which will take the signature of the original LLMObs.submit_evaluation_for() method in ddtrace version 4.0. Please use LLMObs.submit_evaluation() for submitting evaluations moving forward.
  To migrate:
  • LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)
  • LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)
• tracing: Tracer.on_start_span and Tracer.deregister_on_start_span are deprecated and will be removed in v4.0.0 with no planned replacement.
• Support for ddtrace with Python 3.8 is deprecated and will be removed in version 4.0.0.

New Features

• CI Visibility: This introduces Test Impact Analysis code coverage support for Python 3.13.
• azure_eventhubs: Add support for Azure Event Hubs producers.
• azure_functions: Add support for Event Hubs triggers.
• LLM Observability
  • Introduces automatic tracing context propagation for LLM Observability traces involving asynchronous tasks created via asyncio.create_task().
  • The asyncio and futures integrations are now enabled by default on LLMObs.enable(), which enables asynchronous context propagation for those libraries.
  • The LLMObs.submit_evaluation() and LLMObs.submit_evaluation_for() methods now accept a reasoning argument to denote an explanation of the evaluation results.
  • The OpenAI integration now submits LLM spans to LLM Observability for parse() methods used for structured outputs.
  • The LLMObs.submit_evaluation_for() method now accepts a assessment argument to denote
    whether or not the evaluation is valid or correct. Accepted values are either "pass" or "fail".
• openai: Adds support for tracing the parse() methods for structured outputs on chat.completions and responses endpoints (available in OpenAI SDK >= 1.92.0).
• AAP
  • This introduces track_user_id in the ATO SDK, which is equivalent to track_user but does not require the login, only the user id.
  • This introduces supports for custom scanners for data classification.

Bug Fixes

• AAP
  • This fix resolves an issue where downstream request analysis would not match headers in rules when using requests with urllib3\<2.
  • This PR is a tentative fix for rare memory problems with libddwaf that we were unable to reproduce for now.
• Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.
• CI Visibility
  • This fix resolves an issue where tests would be incorrectly detected as third-party code if a third-party package containing a folder with the same name as the tests folder was installed. For instance, the sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.
  • This fix resolves an issue with our coverage implementation for Python versions 3.12+ that affects generated bytecode that isn't mapped to a line in the code
• LLM Observability: Resolves an issue with the Google GenAI integration where processing token metrics would sometimes be skipped if the LLM message had no text part.
• grpc: This fix resolves an issue where the internal span was left active in the caller when using the future interface.
• Profiling: prevent potential deadlocks with thread pools.
• ray
  • This fix resolves an issue where submitting Ray jobs caused an AttributeError crash in certain configurations.
  • This fix resolves an issue where long-running job spans could remain unfinished when an exception occurred during job submission.
  • This fix resolves an issue where long-running spans did not preserve the correct resource name when being recreated.
• otel: Ensures the /v1/logs path is correctly added to prevent log payloads from being dropped by the Agent when using OTEL_EXPORTER_OTLP_ENDPOINT configuration. Metrics and traces are unaffected.