3.1.0

New Features

• Dynamic Instrumentation

  • Symbol Database is now enabled by default.

• LLM Observability

  • Properly labels Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.
  • Introduces an improved automated context management system for LLM Observability-specific spans. Also modifies LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.
  • Introduces automatic distributed tracing support for LLM Observability traces involving the concurrent.futures.thread module.
  • Introduces metadata argument to LLMObs.submit_evaluation_for, which enables users to attach arbitrary structured metadata to custom evaluations.

• Tracing

  • api gateway: Adds preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.
  • graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.
  • graphql: Adds user-provided extensions to graphql error span events through the DD_TRACE_GRAPHQL_ERROR_EXTENSIONS environment variable. This is disabled by default; when set it allows users to capture their extensions.
  • Introduces a record_exception method that adds an exception to a Span as a span event. Refer to Span.record_exception for more details.

Bug Fixes

• ASM

  • Resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

• Code Security (IAST)

  • In some applications, unique cookie names are used per request, leading to a high number of unique cookie vulnerabilities. This fix improves detection accuracy by identifying the vulnerability at the set_cookies function and adding more context. Warning: This change updates the vulnerability hash, meaning that if it was previously silenced in the web dashboard, it may reappear as a new vulnerability.

• LLM Observability

  • botocore: Moves bedrock token usage metrics on APM spans from the meta field under bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.
  • Resolves an issue where the Gemini and Vertex AI integrations were not properly submitting spans for streamed responses in the case that the generator is not fully consumed.
  • Resolves an issue with correlating traces between APM and LLM Observability where trace IDs were always being converted to hexadecimal values which broke APM links for 64-bit trace IDs.
  • Fixes non-LLM message inputs and outputs to be rendered as json rather than Python strings.

• Profiling

  • Fixes a crash in memory allocation profiler when tracking a large number of live allocations, caused by unsigned integer wraparound which lead to invalid memory access
  • Fixes an issue where the profiler unnecessarily outputs log messages when it was unable to record a lock acquire event.

• Single-Step Instrumentation

  • Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.
  • Resolves an issue in which incorrect runtime version constraints prevented injection into processes using Python 3.13.1+

• Tracing

  • asyncio: Fixes an issue where the name of a coroutine was being overridden by a ddtrace function.
  • baggage: Fixes an issue where ot-baggage headers in wsgi apps were not properly recognized due to the http_ header prefix. This ensures consistent baggage propagation in wsgi compatible web frameworks.
  • configurations: This fix resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly
  • configurations: Resolves an issue where accessing ddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.
  • internal: Require bytecode>=0.15.1 for Python 3.12 to avoid known bug.
  • internal: Fix ddtrace internal logger initialization mutating an unlocked shared resource.
  • library: Updates the exception raised when ddtrace.patch(...) is called with an invalid argument.
  • pymongo: Changes the mongodb.query span tag to json so it can be parsed by the agent.
  • runtime_metrics: Ensures only the global service name is tagged on runtime metrics, instead of every service name found in the process.
  • span: Fixes issue where spans weren't being handled correctly and were not being sent when using a custom Exception class that raises an exception in __str__.
  • Captures the execution of async functions traced via DD_TRACE_METHODS, not just coroutine creation. This change increases span durations, which may affect latency-based metrics.
  • Resolves ModuleNotFoundError errors raised when a OpenTracing tracer is initialized with a AsyncioScopeManager.