2.11.0rc2

Bug Fixes

• LLM Observability: Fixes an issue in the OpenAI integration where integration metrics would still be submitted even if LLMObs.enable(agentless_enabled=True) was set.
• Code Security: Adds null pointer checks when creating new objects ids.

2.10.4

Bug Fixes

• SSI: This fixes incorrect file permissions on lib-injection images for 2.10.x releases.
• profiling: show lock init location in Lock Name and hide profiler internal frames from Stack Frame in Timeline Details tab.

2.11.0rc1

Deprecation Notes

• ASM: The environment variable DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING is deprecated and will be removed in the next major release. Instead of DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING, you should use DD_APPSEC_AUTO_USER_INSTRUMENTATION_MODE. The "safe" and "extended" modes are deprecated and have been replaced by "anonymization" and "identification", respectively.
• integrations: All methods in ddtrace/contrib/*/patch.py except patch() and unpatch() are deprecated and will be removed in version 3.0.0 for the following integrations:
  • botocore
  • consul
  • psycopg
  • pylibmc
  • pymemcache
  • pymongo
  • pymysql
  • pynamodb
  • pyodbc
  • pyramid
• exception replay: The DD_EXCEPTION_DEBUGGING_ENABLED environment variable has been deprecated in favor of DD_EXCEPTION_REPLAY_ENABLED. The old environment variable will be removed in a future major release.
• ASM: This removes the partial auto instrumentation of flask login. It was giving only partial and possibly confusing picture of the login activity. We recommend customers to switch to manual instrumentation.

New Features

• ASM: Introduces new Auto User Events support.

  ASM’s Account TakeOver (ATO) detection is now automatically monitoring all compatible user authentication frameworks to detect attempted or leaked user credentials during an ATO campaign.

  To do so, the monitoring of the user activity is extended to now collect all forms of user IDs, including non-numerical forms such as usernames or emails. This is configurable with 3 different working modes: identification to send the user IDs in clear text; anonymization to send anonymized user IDs; or disabled to completely turn off any type of user ID collection (which leads to the disablement of the ATO detection).

  The default collection mode being used is identification and this is configurable in your remote service configuration settings in the service catalog (clicking on a service), or with the service environment variable DD_APPSEC_AUTO_USER_INSTRUMENTATION_MODE.

  You can read more here.

  New local configuration environment variables include:

  • DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING_ENABLED: Can be set to "true"/"1" (default if missing) or "false"/"0" (default if set to any other value). If set to false, the feature is completely disabled. If enabled, the feature is active.
  • DD_APPSEC_AUTO_USER_INSTRUMENTATION_MODE: Can be set to "identification" (default if missing), "anonymization", or "disabled" (default if the environment variable is set to any other value). The values can be modified via remote configuration if the feature is active. If set to "disabled", user events are not collected. Otherwise, user events are collected, using either plain text user_id (in identification mode) or hashed user_id (in anonymization mode).

  Additionally, an optional argument for the public API track_user_login_success_event and track_user_login_failure_event: login_events_mode="auto". This allows manual instrumentation to follow remote configuration settings, enabling or disabling manual instrumentation with a single remote action on the Datadog UI.

  Also prevents non numerical user ids to be reported by default without user instrumentation in Django.

• Anthropic: Adds support for tracing message calls using tools.

• LLM Observability: Adds support for tracing Anthropic messages using tool calls.

• botocore: Adds support for overriding the default service name in botocore by either setting the environment variable DD_BOTOCORE_SERVICE or configuring it via ddtrace.config.botocore["service"].

• azure: Removes the restrictions on the tracer to only run the mini-agent on the consumption plan. The mini-agent now runs regardless of the hosting plan.

• ASM: Adds Threat Monitoring support for gRPC.

• Code Security: Adds propagation for gRPC server sources.

• LLM Observability: Introduces improved support for capturing tool call responses from the OpenAI and Anthropic integrations.

• LLM Observability: Introduces the agentless mode configuration for LLM Observability. To enable agentless mode, set the environment variable DD_LLMOBS_AGENTLESS_ENABLED=1, or use the enable option LLMObs.enable(agentless_enabled=True).

• LLM Observability: Function decorators now support tracing asynchronous functions.

• LLM Observability: Introduces automatic input/output annotation for task/tool/workflow/agent/retrieval spans traced by function decorators. Note that manual annotations for input/output values will override automatic annotations.

• LLM Observability: The OpenAI integration now submits embedding spans to LLM Observability.

• LLM Observability: All OpenAI model parameters specified in a completion/chat completion request are now captured.

• LLM Observability: Changes OpenAI-generated LLM Observability span names from openai.request to openai.createCompletion, openai.createChatCompletion, and openai.createEmbedding for completions, chat completions, and embeddings spans, respectively.

• LLM Observability: Introduces the agent proxy mode for LLM Observability. By default, LLM Observability spans will be sent to the Datadog agent and then forwarded to LLM Observability. To continue submitting data directly to LLM Observability without the Datadog agent, set DD_LLMOBS_AGENTLESS_ENABLED=1 or set programmatically using LLMObs.enable(agentless_enabled=True).

• LLM Observability: The Langchain integration now submits embedding spans to LLM Observability.

• LLM Observability: The LLMObs.annotate() method now replaces non-JSON serializable values with a placeholder string [Unserializable object: <string representation of object>] instead of rejecting the annotation entirely.

• pylibmc: Adds traces for memcached add command

• ASM: Introduces fingerprinting with libddwaf 1.19.1

• Database Monitoring: Adds Database Monitoring (DBM) trace propagation for postgres databases used through Django.

• langchain: Tags tool calls on chat completions.

• LLM Observability: Adds retry logic to the agentless span writer to mitigate potential networking issues, like timeouts or dropped connections.

• ASM: Introduces Command Injection support for Exploit Prevention on os.system only.

• ASM: Introduces suspicious attacker blocking with libddwaf 1.19.1

Bug Fixes

• Code Security: Adds encodings.idna to the IAST patching denylist to avoid problems with gevent.
• Code Security: Adds the boto package to the IAST patching denylist.
• Code Security: Fixes two small memory leaks with Python 3.11 and 3.12.
• CI Visibility: Fixes an issue where the pytest plugin would crash if the git binary was absent
• CI Visibility: Fixes incorrect URL for telemetry intake in EU that was causing missing telemetry data and SSL error log messages.
• celery: Changes error.message span tag to no longer include the traceback that is already included in the error.stack span tag.
• CI Visibility: Fixes source file information that would be incorrect in certain decorated / wrapped scenarios and forces paths to be relative to the repository root, if present.
• futures: Fixes inconsistent behavior with concurrent.futures.ThreadPoolExecutor context propagation by passing the current trace context instead of the currently active span to tasks. This prevents edge cases of disconnected spans when the task executes after the parent span has finished.
• kafka: Fixes ArgumentError raised when injecting span context into non-existent Kafka message headers.
• botocore: Fixes botocore Kinesis span parenting to use active trace context if a propagated child context is not found instead of empty context.
• langchain: Resolves an issue where the wrong langchain class name was being used to check for Pinecone vectorstore instances.
• LLM Observability: Resolves a typing hint error in the ddtrace.llmobs.utils.Documents helper class constructor where type hints did not accept input dictionaries with integer or float values.
• LLM Observability: Resolves an issue where the OpenAI, Anthropic, and AWS Bedrock integrations were always setting temperature and max_tokens parameters to LLM invocations. The OpenAI integration in particular was setting the wrong temperature default values. These parameters are now only set if provided in the request.
• opentelemetry: Resolves circular imports raised by the OpenTelemetry API when the ddcontextvars_context entrypoint is loaded. This resolves an incompatibility introduced in opentelemetry-api==1.25.0.
• opentelemetry: Resolves an issue where the get_tracer function would raise a TypeError when called with the attribute argument. This resolves an incompatibility introduced in opentelemetry-api==1.26.0.
• psycopg: Ensures traced async cursors return an asynchronous iterator object.
• redis: Resolves an issue in the redis exception handling where an UnboundLocalError was raised instead of the expected BaseException.
• ASM: Resolves an issue where the requests integration would not propagate when APM is opted out (i.e. in ASM Standalone).
• profiling: Fixes an issue where task information coming from echion was encoded improperly, which could segfault the application.
• tracing: Fixes a potential crash where using partial flushes and tracer.configure() could result in an ...

2.10.3

Bug Fixes

• ASM: This fix resolves an issue where the WAF could be disabled if the ASM_DD rule file was not found in Remote Config.
• CI Visibility: Fixes an issue where the pytest plugin would crash if the git binary was absent
• CI Visibility: Fixes incorrect URL for telemetry intake in EU that was causing missing telemetry data and SSL error log messages.
• Code Security: Add encodings.idna to the IAST patching denylist to avoid problems with gevent.
• internal: Fixes an issue where some pathlib functions return OSError on Windows.
• opentelemetry: Resolves an edge case where distributed tracing headers could be generated before a sampling decision is made, resulting in dropped spans in downstream services.

2.9.5

Bug Fixes

• ASM: This fix resolves an issue where the WAF could be disabled if the ASM_DD rule file was not found in Remote Config.
• CI Visibility: Fixes an issue where the pytest plugin would crash if the git binary was absent
• CI Visibility: Fixes incorrect URL for telemetry intake in EU that was causing missing telemetry data and SSL error log messages.
• Code Security: fix potential infinite loop with path traversal when the analyze quota has been exceeded.
• opentelemetry: Resolves an edge case where distributed tracing headers could be generated before a sampling decision is made, resulting in dropped spans in downstream services.
• profiling: captures lock usages with with context managers, e.g. with lock:
• profiling: propagates runtime_id tag to libdatadog exporter. It is a unique string identifier for the profiled process. For example, Thread Timeline visualization uses it to distinguish different processes.
• psycopg: Ensures traced async cursors return an asynchronous iterator object.

2.8.6

Bug Fixes

• ASM: This fix resolves an issue where an org could not customize actions through remote config.
• Code Security: add the boto package to the IAST patching denylist.
• CI Visibility: Fixes an issue where the pytest plugin would crash if the git binary was absent
• CI Visibility: fixes source file information that would be incorrect in certain decorated / wrapped scenarios and forces paths to be relative to the repository root, if present.
• CI Visibility: fixes that traces were not properly being sent in agentless mode, and were otherwise not properly attached to the test that started them
• openai: This fix resolves an issue where specifying None for streamed chat completions resulted in a TypeError.
• openai: This fix removes patching for the edits and fine tunes endpoints, which have been removed from the OpenAI API.
• openai: This fix resolves an issue where streamed OpenAI responses raised errors when being used as context managers.
• profiling: Fixes an issue where task information coming from echion was encoded improperly, which could segfault the application.
• tracing: fixes a potential crash where using partial flushes and tracer.configure() could result in an IndexError
• tracing: Fixes an issue where DD_TRACE_SPAN_TRACEBACK_MAX_SIZE was not applied to exception tracebacks.
• tracing: This fix resolves an issue where importing asyncio after a trace has already been started will reset the currently active span.
• flask: Fix scenarios when using flask-like frameworks would cause a crash because of patching issues on startup.
• profiling: captures lock usages with with context managers, e.g. with lock:
• profiling: propagates runtime_id tag to libdatadog exporter. It is a unique string identifier for the profiled process. For example, Thread Timeline visualization uses it to distinguish different processes.

2.10.2

Bug Fixes

• lib-injection: This fix resolves an issue with docker layer caching and the final lib-injection image size.
• psycopg: Ensures traced async cursors return an asynchronous iterator object.
• tracer: This fix resolves an issue where the tracer was not starting properly on a read-only file system.
• Code Security: fix potential infinite loop with path traversal when the analyze quota has been exceeded.
• profiling: captures lock usages with with context managers, e.g. with lock:
• profiling: propagates runtime_id tag to libdatadog exporter. It is a unique string identifier for the profiled process. For example, Thread Timeline visualization uses it to distinguish different processes.

2.10.1

Bug Fixes

• langchain: This fix resolves an issue where the wrong langchain class name was being used to check for Pinecone vectorstore instances.
• opentelemetry: Resolves circular imports raised by the OpenTelemetry API when the ddcontextvars_context entrypoint is loaded. This resolves an incompatibility introduced in opentelemetry-api==1.25.0.
• opentelemetry: Resolves an issue where the get_tracer function would raise a TypeError when called with the attribute argument. This resolves an incompatibility introduced in opentelemetry-api==1.26.0.
• ASM: This fix resolves an issue where ASM one click feature could fail to deactivate ASM.

2.9.4

Bug Fixes

• langchain: This fix resolves an issue where the wrong langchain class name was being used to check for Pinecone vectorstore instances.
• opentelemetry: Resolves circular imports raised by the OpenTelemetry API when the ddcontextvars_context entrypoint is loaded. This resolves an incompatibility introduced in opentelemetry-api==1.25.0.
• opentelemetry: Resolves an issue where the get_tracer function would raise a TypeError when called with the attribute argument. This resolves an incompatibility introduced in opentelemetry-api==1.26.0.
• redis: This fix resolves an issue in the redis exception handling where an UnboundLocalError was raised instead of the expected BaseException.
• Code Security: Logs warning instead of throwing an exception in the native module if IAST is not enabled by env var.
• langchain: This fixes an issue of langchain patching errors due to the langchain-community module becoming an optional dependency in langchain>=0.2.0. The langchain integration now conditionally patches langchain-community methods if it is available. See the langchain integration docs for more details.
• langchain: Resolves incompatibilities with langchain==0.2.0
• ASM: Resolves an issue where ASM one click feature could fail to deactivate ASM.

2.10.0

New Features

• botocore: Adds support for overriding the default service name in botocore by either setting the environment variable DD_BOTOCORE_SERVICE or configuring it via ddtrace.config.botocore["service"].

• Database Monitoring: Adds Database Monitoring (DBM) trace propagation for postgres databases used through Django.

• Anthropic: Adds support for tracing message calls using tools.

• LLM Observability: Adds support for tracing Anthropic messages using tool calls.

• azure: Removes the restrictions on the tracer to only run the mini-agent on the consumption plan. The mini-agent now runs regardless of the hosting plan

• Anthropic: Adds support for tracing synchronous and asynchronous message streaming.

• LLM Observability: Adds support for tracing synchronous and asynchronous message streaming.

• SSI: Introduces generic safeguards for automatic instrumentation when using single step install in the form of early exit conditions. Early exit from instrumentation is triggered if a version of software in the environment is not explicitly supported by ddtrace. The Python runtime itself and many Python packages are checked for explicit support on the basis of their version.

• langchain: This introduces support for langchain==0.2.0 by conditionally patching the langchain-community module if available, which is an optional dependency for langchain>=0.2.0. See the langchain integration :ref: docs<langchain> for more details.

• LLM Observability: Adds support to automatically submit Anthropic chat messages to LLM Observability.

• tracer: This introduces the tracer flare functionality. Currently the tracer flare includes the tracer logs and tracer configurations.

• Expand SSRF vulnerability support for Code Security and Exploit Prevention for the modules urllib3, http.client, webbrowser and urllib.request.

• ASM: This introduces full support for exploit prevention in the python tracer.

  • LFI (via standard API open)
  • SSRF (via standard API urllib or third party requests)

  with monitoring and blocking feature, telemetry and span metrics reports.

• ASM: This introduces SQL injection support for exploit prevention.

• anthropic: This introduces tracing support for anthropic chat messages.
  See the docs for more information.

• ASM: This introduces "Standalone ASM", a feature that disables APM in the tracer but keeps ASM enabled. In order to enable it, set the environment variables DD_APPSEC_ENABLED=1 and DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED=1.

• LLM Observability: This introduces the LLM Observability SDK, which enhances the observability of Python-based LLM applications. See the LLM Observability Overview or the SDK documentation for more information about this feature.

• opentelemetry: Adds support for span events.

• tracing: Ensures the following OpenTelemetry environment variables are mapped to an equivalent Datadog configuration (datadog environment variables taking precedence in cases where both are configured):

  OTEL_SERVICE_NAME -> DD_SERVICE
  OTEL_LOG_LEVEL -> DD_TRACE_DEBUG
  OTEL_PROPAGATORS -> DD_TRACE_PROPAGATION_STYLE
  OTEL_TRACES_SAMPLER -> DD_TRACE_SAMPLE_RATE
  OTEL_TRACES_EXPORTER -> DD_TRACE_ENABLED
  OTEL_METRICS_EXPORTER -> DD_RUNTIME_METRICS_ENABLED
  OTEL_LOGS_EXPORTER -> none
  OTEL_RESOURCE_ATTRIBUTES -> DD_TAGS
  OTEL_SDK_DISABLED -> DD_TRACE_OTEL_ENABLED
  

• otel: adds support for generating Datadog trace metrics using OpenTelemetry instrumentations

Known Issues

• Code Security: Security tracing for the builtins.open function is experimental and may not be stable. This aspect is not replaced by default.
• grpc: Tracing for the grpc.aio clients and servers is experimental and may not be stable. This integration is now disabled by default.

Deprecation Notes

• Removes the deprecated sqlparse dependency.
• LLM Observability: DD_LLMOBS_APP_NAME is deprecated and will be removed in the next major version of ddtrace. As an alternative to DD_LLMOBS_APP_NAME, you can use DD_LLMOBS_ML_APP instead. See the SDK setup documentation for more details on how to configure the LLM Observability SDK.

Bug Fixes

• Code Security: Logs warning instead of throwing an exception in the native module if IAST is not enabled by env var.

• redis: This fix resolves an issue in redis utils where a variable may not be declared within a try/catch

• Code Security: add the boto package to the IAST patching denylist.

• celery: changes error.message span tag to no longer include the traceback that is already included in the error.stack span tag.

• CI Visibility: fixes source file information that would be incorrect in certain decorated / wrapped scenarios and forces paths to be relative to the repository root, if present.

• LLM Observability: This resolves a typing hint error in the ddtrace.llmobs.utils.Documents helper class constructor where type hints did not accept input dictionaries with integer or float values.

• LLM Observability: This fix resolves an issue where the OpenAI, Anthropic, and AWS Bedrock integrations were always setting temperature and max_tokens parameters to LLM invocations. The OpenAI integration in particular was setting the wrong temperature default values. These parameters are now only set if provided in the request.

• redis: This fix resolves an issue in the redis exception handling where an UnboundLocalError was raised instead of the expected BaseException.

• ASM: This fix resolves an issue where the requests integration would not propagate when apm is opted out (i.e. in ASM Standalone).

• profiling: Fixes an issue where task information coming from echion was encoded improperly, which could segfault the application.

• tracing: fixes a potential crash where using partial flushes and tracer.configure() could result in an IndexError

• flask: Fix scenarios when using flask-like frameworks would cause a crash because of patching issues on startup.

• wsgi: Ensures the status of wsgi Spans are not set to error when a StopIteration exception is raised marked the span as an error. With this change, StopIteration exceptions in this context will be ignored.

• langchain: tag non-dict inputs to LCEL chains appropriately. Non-dict inputs are stringified, and dict inputs are tagged by key-value pairs.

• langchain: This fixes an issue of langchain patching errors due to the langchain-community module becoming an optional dependency in langchain>=0.2.0. The langchain integration now conditionally patches langchain-community methods if it is available. See the langchain integration :ref: docs<langchain> for more details.

• ASM: This fix resolves an issue where an org could not customize actions through remote config.

• ASM: protect against potentially returning None when tainting a gRPC message.

• botocore: This fix adds additional key name checking and appropriate defaults for responses from Cohere and Amazon models.

• This fix resolves an issue where importing asyncio after a trace has already been started will reset the currently active span.

• CI Visibility: fixes that traces were not properly being sent in agentless mode, and were otherwise not properly attached to the test that started them

• fix(grpc): This fix a bug in the grpc.aio support specific to streaming responses.

• openai: This fix resolves an issue where specifying n=None for streamed chat completions resulted in a TypeError.

• openai: This fix removes patching for the edits and fine tunes endpoints, which have been removed from the OpenAI API.

• openai: This fix resolves an issue where streamed OpenAI responses raised errors when being used as context managers.

• tracing: Ensures span links generated by distributed tracing headers record the correct sampling decision.

• telemetry: This fix resolves an issue when using pytest + gevent where the telemetry writer was eager initialized by pytest entrypoints loading of our plugin causing a potential dead lock.

• tracing: Fixes an issue where DD_TRACE_SPAN_TRACEBACK_MAX_SIZE was not applied to exception tracebacks.

• Code Security: This fixes a bug in the AST patching process where ImportError exceptions were being caught, interfering with the proper application cycle if an ImportError was expected."

• Code Security: Ensure IAST propagation does not raise side effects related to Magic methods.

• Code Security: fix a potential memory corruption when the context was reset.

• langchain: This fix resolves an issue where specifying inputs as a keyword argument for batching on chains caused a crash.

• Code Security: avoid calling terminate on the extend and join aspect when an exception is raised.

• tracing: Ensures spans are rate limited at the expected rate (100 spans per second by default). Previously long running spans would set the rate limiter to set an invalid window and this could cause the next trace to be dropped.

• RemoteConfig: This fix resolves an issue where remote config did not work for the tracer when using an agent that would add a flare item to the remote config payload. With this fix, the tracer will now correctly pull out the lib_config we need from the payload in order to implement remote config changes properly.

• opentelemetry: Records exceptions on spans in a manner that is consistent with the otel specification
  ...