2.6.10

Bug Fixes

• ASM: This fix resolves an issue where django login failure events may send wrong information of user existence.
• Code Security: fix setting the wrong source on map elements tainted from taint_structure.
• datastreams: Changed DSM processor error logs to debug logs for a statement which is retried. If all retries fail, the stack trace is included
• Code Security: Fixes an issue where the AST patching process fails when the origin of a module is reported as None, raising a FileNotFoundError.
• CI Visibility: fixes an issue where tests were less likely to be skipped due to ITR skippable tests requests timing out earlier than they should
• internal: This fix resolves an issue where importing the ddtrace.contrib.botocore.services module would fail raising an ImportError
• starlette: Fix a bug that crashed background tasks started from functions without a __name__ attribute
• Code Security: Fixed an issue with AES functions from the pycryptodome package that caused the application to crash and stop.
• Code Security: This fix addresses an issue where tainting objects may fail due to context not being created in the current span.
• Code Security: Some native exceptions were not being caught correctly by the python tracer. This fix remove those exceptions to avoid fatal error executions.
• ASM: This fix removes unrequired API security metrics.
• structlog: Fixes error where multiple loggers would duplicate processors. Also adds processors injection when resetting to defaults.

2.7.7

Bug Fixes

• ASM: This fix resolves an issue where django login failure events may send wrong information of user existence.
• datastreams: Changed DSM processor error logs to debug logs for a statement which is retried. If all retries fail, the stack trace is included
• internal: This fix resolves an issue where importing the ddtrace.internal.peer_service module would fail raising an ImportError
• starlette: Fix a bug that crashed background tasks started from functions without a __name__ attribute
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where tainting objects may fail due to context not being created in the current span.
• Vulnerability Management for Code-level (IAST): Some native exceptions were not being caught correctly by the python tracer.
  This fix removes those exceptions to avoid fatal error executions.
• kafka: This fix resolves an issue where an empty message list returned from consume calls could cause crashes in the Kafka integration.
  Empty lists from consume can occur when the call times out.

2.8.0

Prelude

tracing: This release adds support for lazy sampling, essentially moving when we make a sampling decision for a trace to the latest possible moment. These include the following:

1. Before encoding a trace chunk to be sent to the agent
2. Before making an outgoing request via HTTP, gRPC, or a DB call for any automatically instrumented integration
3. Before running os.fork()

For most users this change shouldn't have any impact on their traces, but it does allow for more flexibility in sampling (see features release note). It should be noted that if a user has application egress points that are not automatically instrumented, to other Datadog components (downstream instrumented services, databases, or execution context changes), and rely on the Python tracer to make the sampling decision (don't have an upstream service doing this), they will need to manually run the sampler for those traces, or use HttpPropagator.inject(). For more information please see the following: https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#distributed-tracing https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#tracing-context-management

Deprecation Notes

• tracing: Deprecates support for ddtrace.contrib.asyncio.AsyncioContextProvider. ddtrace fully supports tracing across asyncio tasks. Asyncio no longer requires additional configurations.
• tracing: tracer.sampler is deprecated and will be removed in the next major version release. To manually sample please call tracer.sample instead.
• gevent: Deprecates ddtrace.contrib.gevent.provider.GeventContextProvider. Drops support for gevent<20.12.0 and greenlet<1.0.

New Features

• DSM: Adds base64 format for encoding and decoding DSM context hash.
• botocore: adds dsm payload size stats for botocore messaging services of kinesis, sqs and sns.
• botocore: Adds support to the bedrock integration for tagging input and output messages.
• langchain: This introduces support for langchain==0.1.0. Note that this does not have tracing support for deprecated langchain operations. Please follow the langchain upgrade guide or the langchain integration :ref: docs<langchain> to enable full tracing support.
• dramatiq: Adds automatic tracing of the dramatiq library.
• tracing: Added support for lazy sampling, the benefit of which is the ability to make a sampling decision using DD_TRACE_SAMPLING_RULES based on any span attribute (service, resource, tags, name) regardless of when the value for the attribute is set. This change is particularly beneficial for sampling on tags, since the vast majority of tags are set after the span is created. Since sampling was previously done at span creation time, this meant that those tags could not be used for sampling decisions.
• openai: Adds support for tagging streamed responses for completion and chat completion endpoints.
• profiling: implement an experimental stack sampling feature, which can be enabled by setting DD_PROFILING_STACK_V2_ENABLED=true. This new sampler should resolve segfault issues on Python 3.11 and later, while also decreasing the latency contribution of the profiler in many situations, and also improving the accuracy of stack-sampling data. This feature is currently only available on Linux using CPython 3.8 or greater. Requires DD_PROFILING_EXPORT_LIBDD_ENABLED=true to be set.
• botocore: Changes botocore aws kinesis contrib to set DSM pathway using extracted DSM context, if found, instead of always using a new pathway with default context.
• kafka: Adds tracing and DSM support for confluent_kafka.Consumer.consume(). Previously only confluent_kafka.Consumer.poll was instrumented.

Bug Fixes

• otel: Ensures that the last datadog parent_id is added to w3c distributed tracing headers generated by the OpenTelemetry API.
• ASM: This fix resolves an issue where a valid user may trigger a failed login event.
• ASM: always clear the DDWaf context at the end of the span to avoid gc-induced latency spikes at the end of some requests.
• ASM: This fix resolves an issue where django login failure events may send wrong information of user existence.
• Vulnerability Management for Code-level (IAST): Some native exceptions were not being caught correctly by the python tracer. This fix remove those exceptions to avoid fatal error executions.
• CI Visibility: fixes an issue where git author or committer names containing commas (eg: "Lastname, Firstname") would not work (and log an error) due to the use of comma as a separator.
• propagation: This fix resolves an issue where the sampling decision-maker tag in tracestate propagation headers was clobbered by a default value.
• datastreams: Changed DSM processor error logs to debug logs for a statement which is retried. If all retries fail, the stack trace is included
• internal telemetry: Ensures heartbeat events are sent at regular intervals even when no other events are being sent.
• Fix an incompatibility between the handling of namespace module imports and parts of the functionalities of the standard library importlib module.
• internal: This fix resolves an issue where importing the ddtrace.appsec._iast._patches module would fail raising an ImportError
• internal: This fix resolves an issue where importing the ddtrace.internal.peer_service module would fail raising an ImportError
• langchain: Ensures langchain vision APIs are correctly instrumented
• Fix for the declaration of dependencies for the package.
• internal: This fix resolves an issue where importing the ddtrace.contrib.botocore.services module would fail raising an ImportError
• profiling: handle unexpected stack data to prevent the profiler from stopping.
• starlette: Fix a bug that crashed background tasks started from functions without a __name__ attribute
• ASM: This fix resolves an issue where the asgi middleware could crash with a RuntimeError "Unexpected message received".
• ASM: This fix resolves an issue with Flask instrumentation causing CPU leak with ASM, API Security and Telemetry enabled.
• Vulnerability Management for Code-level (IAST): Addresses an issue where the IAST native module was imported even though IAST was not enabled.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where tainting objects may fail due to context not being created in the current span.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where AST patching would generate code that fails to compile, thereby preventing the application from starting correctly.
• Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where other subscript operations than Load were being unintentionally patched, leading to compilation errors for the patched module.
• Vulnerability Management for Code-level (IAST): Fixes an issue where an atexit handler could lead to a segmentation fault.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where a vulnerability would be reported at line 0 if we couldn't extract the proper line number, whereas the default line number should be -1.
• kafka: This fix resolves an issue where None messages from confluent-kafka could cause crashes in the Kafka integration.
• appsec: This fix resolves an issue in which the library attempted to finalize twice a context object used by the Application Security Management product.
• tracing: Removes allow_false argument from ddtrace samplers. allow_false allows datadog samplers to return a value that differs from the sampling decision, this behavior is not supported.
• profiling: This fixes a free(): invalid pointer error which would arise as a result of incorrectly linking the C++ runtime.
• starlette: Ensures correct URL tag is set for starlette v0.34.0 and above.
• structlog: Fixes error where multiple loggers would duplicate processors. Also adds processors injection when resetting to defaults.

2.8.0rc2

Prelude

tracing: This release adds support for lazy sampling, essentially moving when we make a sampling decision for a trace to the latest possible moment. These include the following:

1. Before encoding a trace chunk to be sent to the agent
2. Before making an outgoing request via HTTP, gRPC, or a DB call for any automatically instrumented integration
3. Before running os.fork()

For most users this change shouldn't have any impact on their traces, but it does allow for more flexibility in sampling (see features release note). It should be noted that if a user has application egress points that are not automatically instrumented, to other Datadog components (downstream instrumented services, databases, or execution context changes), and rely on the Python tracer to make the sampling decision (don't have an upstream service doing this), they will need to manually run the sampler for those traces, or use HttpPropagator.inject(). For more information please see the following: https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#distributed-tracing https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#tracing-context-management

Deprecation Notes

• tracing: Deprecates support for ddtrace.contrib.asyncio.AsyncioContextProvider. ddtrace fully supports tracing across asyncio tasks. Asyncio no longer requires additional configurations.
• tracing: tracer.sampler is deprecated and will be removed in the next major version release. To manually sample please call tracer.sample instead.
• gevent: Deprecates ddtrace.contrib.gevent.provider.GeventContextProvider. Drops support for gevent<20.12.0 and greenlet<1.0.

New Features

• DSM: Adds base64 format for encoding and decoding DSM context hash.
• botocore: adds dsm payload size stats for botocore messaging services of kinesis, sqs and sns.
• botocore: Adds support to the bedrock integration for tagging input and output messages.
• langchain: This introduces support for langchain==0.1.0. Note that this does not have tracing support for deprecated langchain operations. Please follow the langchain upgrade guide or the langchain integration :ref: docs<langchain> to enable full tracing support.
• dramatiq: Adds automatic tracing of the dramatiq library.
• tracing: Added support for lazy sampling, the benefit of which is the ability to make a sampling decision using DD_TRACE_SAMPLING_RULES based on any span attribute (service, resource, tags, name) regardless of when the value for the attribute is set. This change is particularly beneficial for sampling on tags, since the vast majority of tags are set after the span is created. Since sampling was previously done at span creation time, this meant that those tags could not be used for sampling decisions.
• openai: Adds support for tagging streamed responses for completion and chat completion endpoints.
• profiling: implement an experimental stack sampling feature, which can be enabled by setting DD_PROFILING_STACK_V2_ENABLED=true. This new sampler should resolve segfault issues on Python 3.11 and later, while also decreasing the latency contribution of the profiler in many situations, and also improving the accuracy of stack-sampling data. This feature is currently only available on Linux using CPython 3.8 or greater. Requires DD_PROFILING_EXPORT_LIBDD_ENABLED=true to be set.
• botocore: Changes botocore aws kinesis contrib to set DSM pathway using extracted DSM context, if found, instead of always using a new pathway with default context.
• kafka: Adds tracing and DSM support for confluent_kafka.Consumer.consume(). Previously only confluent_kafka.Consumer.poll was instrumented.

Bug Fixes

• otel: Ensures that the last datadog parent_id is added to w3c distributed tracing headers generated by the OpenTelemetry API.
• ASM: This fix resolves an issue where a valid user may trigger a failed login event.
• ASM: always clear the DDWaf context at the end of the span to avoid gc-induced latency spikes at the end of some requests.
• ASM: This fix resolves an issue where django login failure events may send wrong information of user existence.
• Vulnerability Management for Code-level (IAST): Some native exceptions were not being caught correctly by the python tracer. This fix remove those exceptions to avoid fatal error executions.
• CI Visibility: fixes an issue where git author or committer names containing commas (eg: "Lastname, Firstname") would not work (and log an error) due to the use of comma as a separator.
• propagation: This fix resolves an issue where the sampling decision-maker tag in tracestate propagation headers was clobbered by a default value.
• datastreams: Changed DSM processor error logs to debug logs for a statement which is retried. If all retries fail, the stack trace is included
• internal telemetry: Ensures heartbeat events are sent at regular intervals even when no other events are being sent.
• Fix an incompatibility between the handling of namespace module imports and parts of the functionalities of the standard library importlib module.
• internal: This fix resolves an issue where importing the ddtrace.appsec._iast._patches module would fail raising an ImportError
• internal: This fix resolves an issue where importing the ddtrace.internal.peer_service module would fail raising an ImportError
• langchain: Ensures langchain vision APIs are correctly instrumented
• Fix for the declaration of dependencies for the package.
• internal: This fix resolves an issue where importing the ddtrace.contrib.botocore.services module would fail raising an ImportError
• profiling: handle unexpected stack data to prevent the profiler from stopping.
• starlette: Fix a bug that crashed background tasks started from functions without a __name__ attribute
• ASM: This fix resolves an issue where the asgi middleware could crash with a RuntimeError "Unexpected message received".
• ASM: This fix resolves an issue with Flask instrumentation causing CPU leak with ASM, API Security and Telemetry enabled.
• Vulnerability Management for Code-level (IAST): Addresses an issue where the IAST native module was imported even though IAST was not enabled.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where tainting objects may fail due to context not being created in the current span.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where AST patching would generate code that fails to compile, thereby preventing the application from starting correctly.
• Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where other subscript operations than Load were being unintentionally patched, leading to compilation errors for the patched module.
• Vulnerability Management for Code-level (IAST): Fixes an issue where an atexit handler could lead to a segmentation fault.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where a vulnerability would be reported at line 0 if we couldn't extract the proper line number, whereas the default line number should be -1.
• kafka: This fix resolves an issue where None messages from confluent-kafka could cause crashes in the Kafka integration.
• appsec: This fix resolves an issue in which the library attempted to finalize twice a context object used by the Application Security Management product.
• tracing: Removes allow_false argument from ddtrace samplers. allow_false allows datadog samplers to return a value that differs from the sampling decision, this behavior is not supported.
• profiling: This fixes a free(): invalid pointer error which would arise as a result of incorrectly linking the C++ runtime.
• starlette: Ensures correct URL tag is set for starlette v0.34.0 and above.
• structlog: Fixes error where multiple loggers would duplicate processors. Also adds processors injection when resetting to defaults.

2.7.6

Bug Fixes

• Profiling: This fix resolves an issue where the profiler was forcing protobuf to load in injected environments,
  causing crashes in configurations which relied on older protobuf versions. The profiler will now detect when injection is used and try loading with the native exporter. If that fails, it will self-disable rather than loading protobuf.

2.8.0rc1

Prelude

tracing: This release adds support for lazy sampling, essentially moving when we make a sampling decision for a trace to the latest possible moment. These include the following:

1. Before encoding a trace chunk to be sent to the agent
2. Before making an outgoing request via HTTP, gRPC, or a DB call for any automatically instrumented integration
3. Before running os.fork()

For most users this change shouldn't have any impact on their traces, but it does allow for more flexibility in sampling (see features release note). It should be noted that if a user has application egress points that are not automatically instrumented, to other Datadog components (downstream instrumented services, databases, or execution context changes), and rely on the Python tracer to make the sampling decision (don't have an upstream service doing this), they will need to manually run the sampler for those traces, or use HttpPropagator.inject(). For more information please see the following: https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#distributed-tracing https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#tracing-context-management

Deprecation Notes

• tracing: Deprecates support for ddtrace.contrib.asyncio.AsyncioContextProvider. ddtrace fully supports tracing across asyncio tasks. Asyncio no longer requires additional configurations.
• tracing: tracer.sampler is deprecated and will be removed in the next major version release. To manually sample please call tracer.sample instead.
• gevent: Deprecates ddtrace.contrib.gevent.provider.GeventContextProvider. Drops support for gevent<20.12.0 and greenlet<1.0.

New Features

• DSM: Adds base64 format for encoding and decoding DSM context hash.
• botocore: adds dsm payload size stats for botocore messaging services of kinesis, sqs and sns.
• botocore: Adds support to the bedrock integration for tagging input and output messages.
• langchain: This introduces support for langchain==0.1.0. Note that this does not have tracing support for deprecated langchain operations. Please follow the langchain upgrade guide or the langchain integration :ref: docs<langchain> to enable full tracing support.
• dramatiq: Adds automatic tracing of the dramatiq library.
• tracing: Added support for lazy sampling, the benefit of which is the ability to make a sampling decision using DD_TRACE_SAMPLING_RULES based on any span attribute (service, resource, tags, name) regardless of when the value for the attribute is set. This change is particularly beneficial for sampling on tags, since the vast majority of tags are set after the span is created. Since sampling was previously done at span creation time, this meant that those tags could not be used for sampling decisions.
• openai: Adds support for tagging streamed responses for completion and chat completion endpoints.
• profiling: implement an experimental stack sampling feature, which can be enabled by setting DD_PROFILING_STACK_V2_ENABLED=true. This new sampler should resolve segfault issues on Python 3.11 and later, while also decreasing the latency contribution of the profiler in many situations, and also improving the accuracy of stack-sampling data. This feature is currently only available on Linux using CPython 3.8 or greater. Requires DD_PROFILING_EXPORT_LIBDD_ENABLED=true to be set.
• botocore: Changes botocore aws kinesis contrib to set DSM pathway using extracted DSM context, if found, instead of always using a new pathway with default context.
• kafka: Adds tracing and DSM support for confluent_kafka.Consumer.consume(). Previously only confluent_kafka.Consumer.poll was instrumented.

Bug Fixes

• otel: Ensures that the last datadog parent_id is added to w3c distributed tracing headers generated by the OpenTelemetry API.
• ASM: This fix resolves an issue where a valid user may trigger a failed login event.
• ASM: always clear the DDWaf context at the end of the span to avoid gc-induced latency spikes at the end of some requests.
• ASM: This fix resolves an issue where django login failure events may send wrong information of user existence.
• CI Visibility: fixes an issue where git author or committer names containing commas (eg: "Lastname, Firstname") would not work (and log an error) due to the use of comma as a separator.
• propagation: This fix resolves an issue where the sampling decision-maker tag in tracestate propagation headers was clobbered by a default value.
• datastreams: Changed DSM processor error logs to debug logs for a statement which is retried. If all retries fail, the stack trace is included
• internal telemetry: Ensures heartbeat events are sent at regular intervals even when no other events are being sent.
• Fix an incompatibility between the handling of namespace module imports and parts of the functionalities of the standard library importlib module.
• internal: This fix resolves an issue where importing the ddtrace.appsec._iast._patches module would fail raising an ImportError
• internal: This fix resolves an issue where importing the ddtrace.internal.peer_service module would fail raising an ImportError
• langchain: Ensures langchain vision APIs are correctly instrumented
• Fix for the declaration of dependencies for the package.
• internal: This fix resolves an issue where importing the ddtrace.contrib.botocore.services module would fail raising an ImportError
• profiling: handle unexpected stack data to prevent the profiler from stopping.
• starlette: Fix a bug that crashed background tasks started from functions without a __name__ attribute
• ASM: This fix resolves an issue where the asgi middleware could crash with a RuntimeError "Unexpected message received".
• ASM: This fix resolves an issue with Flask instrumentation causing CPU leak with ASM, API Security and Telemetry enabled.
• Vulnerability Management for Code-level (IAST): Addresses an issue where the IAST native module was imported even though IAST was not enabled.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where tainting objects may fail due to context not being created in the current span.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where AST patching would generate code that fails to compile, thereby preventing the application from starting correctly.
• Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where other subscript operations than Load were being unintentionally patched, leading to compilation errors for the patched module.
• Vulnerability Management for Code-level (IAST): Fixes an issue where an atexit handler could lead to a segmentation fault.
• Vulnerability Management for Code-level (IAST): This fix addresses an issue where a vulnerability would be reported at line 0 if we couldn't extract the proper line number, whereas the default line number should be -1.
• kafka: This fix resolves an issue where None messages from confluent-kafka could cause crashes in the Kafka integration.
• appsec: This fix resolves an issue in which the library attempted to finalize twice a context object used by the Application Security Management product.
• tracing: Removes allow_false argument from ddtrace samplers. allow_false allows datadog samplers to return a value that differs from the sampling decision, this behavior is not supported.
• profiling: This fixes a free(): invalid pointer error which would arise as a result of incorrectly linking the C++ runtime.
• starlette: Ensures correct URL tag is set for starlette v0.34.0 and above.
• structlog: Fixes error where multiple loggers would duplicate processors. Also adds processors injection when resetting to defaults.

2.7.5

New Features

• kafka: Adds tracing and DSM support for confluent_kafka.Consumer.consume(). Previously only confluent_kafka.Consumer.poll was instrumented.

Bug Fixes

• ASM: always clear the DDWaf context at the end of the span to avoid gc-induced latency spikes at the end of some requests.
• internal: This fix resolves an issue where importing the ddtrace.contrib.botocore.services module would fail raising an ImportError
• setuptools_scm version: Updates the setuptools_scm versioning method to "guess-next-dev" from "release-branch-semver", which was affecting the CI
• structlog: Fixes error where multiple loggers would duplicate processors. Also adds processors injection when resetting to defaults.

2.6.9

Bug Fixes

• propagation: This fix resolves an issue where the sampling decision-maker tag in tracestate propagation headers was clobbered by a default value.
• langchain: Ensures langchain vision APIs are correctly instrumented
• ASM: This fix resolves an issue where the asgi middleware could crash with a RuntimeError "Unexpected message received".
• kafka: This fix resolves an issue where None messages from confluent-kafka could cause crashes in the Kafka integration.

2.7.4

Bug Fixes

• ASM: This fix resolves an issue where a valid user may trigger a failed login event.
• propagation: This fix resolves an issue where the sampling decision-maker tag in tracestate propagation headers was clobbered by a default value.
• langchain: Ensures langchain vision APIs are correctly instrumented
• ASM: This fix resolves an issue where the asgi middleware could crash with a RuntimeError "Unexpected message received".
• kafka: This fix resolves an issue where None messages from confluent-kafka could cause crashes in the Kafka integration.

2.7.3

Bug Fixes

• otel: Ensures that the last datadog parent_id is added to w3c distributed tracing headers generated by the OpenTelemetry API.
• internal telemetry: Ensures heartbeat events are sent at regular intervals even when no other events are being sent.
• Fix an incompatibility between the handling of namespace module imports and parts of the functionalities of the standard library importlib module.
• Fix for the declaration of dependencies for the package.
• appsec: This fix resolves an issue in which the library attempted to finalize twice a context object used by the Application Security Management product.
• profiling: This fixes a free(): invalid pointer error which would arise as a result of incorrectly linking the C++ runtime.