3.7.0

New Features

• AAP (ASM is now AAP)

  • Introduces a new user event sdk available through ddtrace.appsec.track_user_sdk for manual instrumentation.

• CI Visibility

  • Introduces the ability to gzip the payload when using the agentless setup, incurring in less network bandwidth consumption.

• LLM Observability

  • Adds support to automatically submit LiteLLM SDK requests to LLM Observability.

• Tracing

  • baggage: Adds support for automatically adding baggage key-value pairs to span tags. Baggage items from incoming HTTP headers are attached to spans as tags with a baggage. prefix The DD_TRACE_BAGGAGE_TAG_KEYS configuration allows users to specify a comma-separated list of baggage keys for span tagging, by default the value is set to user.id,account.id,session.id. When set to \*, all baggage keys will be converted into span tags. Setting it to an empty value disables baggage tagging.

Bug Fixes

• Code Security

  • The field representing the class name in IAST vulnerability location reporting was previously incorrectly named as class_name. This fix standardizes the naming and ensures that the correct field name is used (class).

• Dynamic Instrumentation

  • Fixes incompatibility between code origin and dynamic instrumentation probe on exit span functions.

• LLM Observability

  • openai: This fix resolves an issue where using client.beta.chat.completions.stream with openai patching caused an attribute error.
  • Resolves an issue where using client.beta.chat.completions.stream with LLM Observability enabled caused an attribute error
  • Resolves an issue where tool call names were not being captured for bedrock converse_stream calls.
  • Resolves an issue where bedrock converse_stream calls contained an extra empty output message.

• Profiling

  • Fixes an issue where inconsistent state in the native profiler code was causing excessive log messages to be emitted at an unprecedented rate, significantly increasing the size of end-user log files.

• Tracing

  • azure_functions: Resolves an issue where async functions throw an error when instrumented.
  • datastreams: Resolves an issue where failure to decode the data streams context caused infinite loops in data streams checkpoints.
  • futures: Resolves an edge case where trace context was not propagated to spans generated by the ThreadPoolExecutor.
  • kafka: Fixes an issue where a producer or consumer initialized with an unpacked config resulted in TypeError, causing a failed connection. confluent-kafka supports both unpacked and packed config; this change allows initialization with either.
  • telemetry: Improves periodic telemetry writer performance by removing unnecessary calls to importlib.metadata for reporting imported dependencies.
  • Fixes an issue where empty sampling rules list in remote configuration events caused an AttributeError when reinitializing the DatadogSampler. This prevented sampling rules from being reset. Note: This only affected cases where sampling rules were an empty list. It did not impact cases with at least one rule or when rules were set to null.
  • Fixes a bug in the sampling rule matcher where the pattern ?* was not being matched correctly for DD_TRACE_SAMPLING_RULES tags, due to it matching on spans with no tag matching the specified key.
  • Fixes an issue where span attributes were not truncated before encoding, leading to runtime error and causing spans to be dropped. Spans with resource name, tag key or value larger than 25000 characters will be truncated to 2500 characters.
  • Fixes RuntimeWarning from an unwaited coroutine during tab completion in IPython REPL when asyncio integration is active. Tracer now wraps an asyncio coroutine only when there is an active trace context.

3.7.0rc2

New Features

• CI Visibility
  • Introduces the ability to gzip the payload when using the agentless setup, incurring in less network bandwidth consumption.

Bug Fixes

• Code Security

  • The field representing the class name in IAST vulnerability location reporting was previously incorrectly named as class_name. This fix standardizes the naming and ensures that the correct field name is used (class).

• Tracing

  • futures: Resolves an edge case where trace context was not propagated to spans generated by the ThreadPoolExecutor.
  • Fixes RuntimeWarning from an unwaited coroutine during tab completion in IPython REPL when asyncio integration is active. Tracer now wraps an asyncio coroutine only when there is an active trace context.

3.7.0rc1

New Features

• AAP (ASM is now AAP)

  • Introduces a new user event sdk available through ddtrace.appsec.track_user_sdk for manual instrumentation.

• LLM Observability

  • Adds support to automatically submit LiteLLM SDK requests to LLM Observability.

• Tracing

  • baggage: Adds support for automatically adding baggage key-value pairs to span tags. Baggage items from incoming HTTP headers are attached to spans as tags with a baggage. prefix The DD_TRACE_BAGGAGE_TAG_KEYS configuration allows users to specify a comma-separated list of baggage keys for span tagging, by default the value is set to user.id,account.id,session.id. When set to \*, all baggage keys will be converted into span tags. Setting it to an empty value disables baggage tagging.

Bug Fixes

• Dynamic Instrumentation

  • Fixes incompatibility between code origin and dynamic instrumentation probe on exit span functions.

• LLM Observability

  • openai: This fix resolves an issue where using client.beta.chat.completions.stream with openai patching caused an attribute error.
  • Resolves an issue where using client.beta.chat.completions.stream with LLM Observability enabled caused an attribute error
  • Resolves an issue where tool call names were not being captured for bedrock converse_stream calls.
  • Resolves an issue where bedrock converse_stream calls contained an extra empty output message.

• Profiling

  • Fixes an issue where inconsistent state in the native profiler code was causing excessive log messages to be emitted at an unprecedented rate, significantly increasing the size of end-user log files.

• Tracing

  • azure_functions: Resolves an issue where async functions throw an error when instrumented.
  • datastreams: Resolves an issue where failure to decode the data streams context caused infinite loops in data streams checkpoints.
  • kafka: Fixes an issue where a producer or consumer initialized with an unpacked config resulted in TypeError, causing a failed connection. confluent-kafka supports both unpacked and packed config; this change allows initialization with either.
  • telemetry: Improves periodic telemetry writer performance by removing unnecessary calls to importlib.metadata for reporting imported dependencies.
  • Fixes an issue where empty sampling rules list in remote configuration events caused an AttributeError when reinitializing the DatadogSampler. This prevented sampling rules from being reset. Note: This only affected cases where sampling rules were an empty list. It did not impact cases with at least one rule or when rules were set to null.
  • Fixes a bug in the sampling rule matcher where the pattern ?* was not being matched correctly for DD_TRACE_SAMPLING_RULES tags, due to it matching on spans with no tag matching the specified key.
  • Fixes an issue where span attributes were not truncated before encoding, leading to runtime error and causing spans to be dropped. Spans with resource name, tag key or value larger than 25000 characters will be truncated to 2500 characters.

3.6.0

Upgrade Notes

• Profiling

  • Upgrades echion, improving the performance of stack sampler by replacing proc filesystem reads with clock_gettime().
  • Upgrades echion, improving the performance of the stack sampler by reusing memory and reducing the frequency of memory allocation.

• Tracing

  • Sampling formula updated to have the same as the one in the agent and other tracers.

New Features

• LLM Observability
  • Introduces support for submitting custom evaluations to LLM Observability via agent proxy mode.

Bug Fixes

• LLM Observability

  • Resolves an issue where annotating session_id on a LLM Observability span via LLMObs.annotate/annotation_context() as a tag only set it as a tag on the span, but did not update the span's actual session ID value.

• Profiling

  • Reduces the overhead of running the new stack-v2 sampling mechanism via adaptive sampling.
  • Upgrades echion to pick up fixes which enables the stack sampler to avoid extra calls to process_vm_readv() syscalls for Python frame objects for Python versions 3.11+.
  • Reduces overhead of the v2 stack sampler by removing unnecessary and expensive utf-8 validation
  • Reduces the overhead of code provenance reporting when stack v2 is enabled, the default stack sampler.
  • For CPython 3.10+, code provenance correctly classifies frozen modules in Python stdlib when stack v2 is enabled, the default stack sampler.
  • Upgrades echion to pick up a fix which resolves a potential crash. The stack sampler could read off of an empty queue of frames after failing to resolve specific frame information, triggering undefined behavior.

• Tracing

  • Fixes an issue where the sampling decision was hardcoded to USER_KEEP when extracting sampling priority from HTTP headers, if there was no x-datadog-sampling-priority header. The sampling decision is now made using the updated sampling formula.

3.6.0rc1

Upgrade Notes

• Profiling

  • Upgrades echion, improving the performance of stack sampler by replacing proc filesystem reads with clock_gettime().
  • Upgrades echion, improving the performance of the stack sampler by reusing memory and reducing the frequency of memory allocation.

• Tracing

  • Sampling formula updated to have the same as the one in the agent and other tracers.

New Features

• LLM Observability
  • Introduces support for submitting custom evaluations to LLM Observability via agent proxy mode.

Bug Fixes

• LLM Observability

  • Resolves an issue where annotating session_id on a LLM Observability span via LLMObs.annotate/annotation_context() as a tag only set it as a tag on the span, but did not update the span's actual session ID value.

• Profiling

  • Reduces the overhead of running the new stack-v2 sampling mechanism via adaptive sampling.
  • Upgrades echion to pick up fixes which enables the stack sampler to avoid extra calls to process_vm_readv() syscalls for Python frame objects for Python versions 3.11+.
  • Reduces overhead of the v2 stack sampler by removing unnecessary and expensive utf-8 validation
  • Reduces the overhead of code provenance reporting when stack v2 is enabled, the default stack sampler.
  • For CPython 3.10+, code provenance correctly classifies frozen modules in Python stdlib when stack v2 is enabled, the default stack sampler.
  • Upgrades echion to pick up a fix which resolves a potential crash. The stack sampler could read off of an empty queue of frames after failing to resolve specific frame information, triggering undefined behavior.

• Tracing

  • Fixes an issue where the sampling decision was hardcoded to USER_KEEP when extracting sampling priority from HTTP headers, if there was no x-datadog-sampling-priority header. The sampling decision is now made using the updated sampling formula.

3.5.1

Bug Fixes

• Tracing
  • azure_functions: Resolves an issue where spans generated from timer triggers did not include the trigger type in the resource name.

3.4.2

Bug Fixes

• Profiling

  • Fixes an infinite recursion from calling logging module when lock profiler fails to delete _self_acquired_at attribute.

• Tracing

  • subprocess: Removes unnecessary debug log variables.

2.21.6

Bug Fixes

• Code Security

  • Fixes an issue with PosixPath handling in path operations that could cause errors during taint tracking. This fix improves stability and slightly reduces import times.

• Profiling

  • Fixes an infinite recursion from calling logging module when lock profiler fails to delete _self_acquired_at attribute.

• Tracing

  • subprocess: Removes unnecessary debug log variables.

3.5.0

Deprecation Notes

• Tracing
  • opentracing: Support for OpenTracing has been deprecated and will be removed in v4.0.0. Please use the OpenTelemetry API instead.
  • DD_RUNTIME_METRICS_INTERVAL is deprecated and will be removed in the 4.0.0 release.

New Features

• LLM Observability

  • Introduces tracing for converse_stream calls to the Bedrock Converse API. converse_stream calls are traced as LLM spans with max_tokens, temperature inference parameters and input/output messages.
  • Introduces tracing support for LangGraph graph and node invocation.
  • Introduces tracing support for CrewAI crew, task, agent, and tool executions.
  • Introduces tracing support for OpenAI Agents SDK. Agent, guardrail, tool, handoff, response, and custom operations are traced. Input/output data for the trace is only set if the default Responses API is used.
  • botocore: Introduces capturing prompts, token usage, and inference parameters for converse_stream calls to the Bedrock API.
  • crewai: Introduces tracing support for the CrewAI SDK, specifically tracing crew, task, agent, and tool executions. See the docs for more information.
  • litellm: Introduces tracing support for the LiteLLM Python SDK's synchronous and asynchronous completion and text_completion methods. See the docs for more information.
  • openai_agents: Introduces tracing support for OpenAI Agents SDK. Agent, guardrail, tool, handoff, response, and custom operations are traced. See the docs for more information.

• Tracing

  • Support for Database Monitoring (DBM) propagation in pymongo. Two propagation modes are supported:

    • Full mode: Injects both service information and trace context, allowing complete correlation between traces and database monitoring samples.
    • Service mode: Injects only service information, enabling service-level correlation without trace-specific details.

    This feature can be controlled using the DD_DBM_PROPAGATION_MODE environment variable with values full, service, or disabled. Example of enabling full or service propagation mode: DD_DBM_PROPAGATION_MODE=full|service
    Note that this feature requires PyMongo 3.9 or higher.

  • A subset of the tracer configuration is now stored on disk throughout the tracer's lifetime. This enables the monitoring of which processes are instrumented.

  • azure_functions: Introduces tracing support for timer triggers.

  • langgraph: Introduces tracing support for LangGraph, specifically tracing graph and node executions. See the docs for more information.

Bug Fixes

• Code Security

  • IAST: Fixes a bug where invalid f-strings didn’t raise the expected "Unknown format code" error when IAST was enabled.
  • Fixes an issue with PosixPath handling in path operations that could cause errors during taint tracking. This fix improves stability and slightly reduces import times.

• CI Visibility

  • Resolves an issue where the CODEOWNERS file of the project being tested was looked up in the current working directory rather than the repository root.
  • Resolves an issue where an I/O operation on closed file error was raised while printing ddtrace logs during a pytest session, due to a conflict with pytest's log capture.

• Profiling

  • Fixes an infinite recursion from calling logging module when lock profiler fails to delete _self_acquired_at attribute.

• Tracing

  • Resolves an issue where Django, Flask, Cherrypy, wsgi, asgi, pyramid, molten, falcon, tornado, aiohttp, bottle, rq, and sanic integrations were unnecessarily running code to activate distributed tracing headers multiple times per request. This is fixed by only activating distributed tracing headers once per request.
  • azure_functions: Resolves an issue where the function name set by the function_name decorator was ignored.
  • azure_functions: Resolves an issue where a function would not start if the trigger_arg_nameparameter was set to anything other than req.
  • azure_functions: Resolves an issue where timer triggers had a span kind of server instead of internal
  • graphql: Resolves AttributeError raised when used with older versions of graphql-core that do not have formatted property for SourceLocation.
  • opentelemetry: Updates trace flags in opentelemetry.trace.Span.get_span_context() to match OpenTelemetry specification.
  • subprocess: Removes unnecessary debug log variables.

3.5.0rc1

Deprecation Notes

• Tracing
  • opentracing: Support for OpenTracing has been deprecated and will be removed in v4.0.0. Please use the OpenTelemetry API instead.
  • DD_RUNTIME_METRICS_INTERVAL is deprecated and will be removed in the 4.0.0 release.

New Features

• LLM Observability

  • Introduces tracing for converse_stream calls to the Bedrock Converse API. converse_stream calls are traced as LLM spans with max_tokens, temperature inference parameters and input/output messages.
  • Introduces tracing support for LangGraph graph and node invocation.
  • Introduces tracing support for CrewAI crew, task, agent, and tool executions.
  • Introduces tracing support for OpenAI Agents SDK. Agent, guardrail, tool, handoff, response, and custom operations are traced. Input/output data for the trace is only set if the default Responses API is used.
  • botocore: Introduces capturing prompts, token usage, and inference parameters for converse_stream calls to the Bedrock API.
  • crewai: Introduces tracing support for the CrewAI SDK, specifically tracing crew, task, agent, and tool executions. See the docs for more information.
  • litellm: Introduces tracing support for the LiteLLM Python SDK's synchronous and asynchronous completion and text_completion methods. See the docs for more information.
  • openai_agents: Introduces tracing support for OpenAI Agents SDK. Agent, guardrail, tool, handoff, response, and custom operations are traced. See the docs for more information.

• Tracing

  • Support for Database Monitoring (DBM) propagation in pymongo. Two propagation modes are supported:

    • Full mode: Injects both service information and trace context, allowing complete correlation between traces and database monitoring samples.
    • Service mode: Injects only service information, enabling service-level correlation without trace-specific details.

    This feature can be controlled using the DD_DBM_PROPAGATION_MODE environment variable with values full, service, or disabled. Example of enabling full or service propagation mode: DD_DBM_PROPAGATION_MODE=full|service
    Note that this feature requires PyMongo 3.9 or higher.

  • A subset of the tracer configuration is now stored on disk throughout the tracer's lifetime. This enables the monitoring of which processes are instrumented.

  • azure_functions: Introduces tracing support for timer triggers.

  • langgraph: Introduces tracing support for LangGraph, specifically tracing graph and node executions. See the docs for more information.

Bug Fixes

• Code Security

  • IAST: Fixes a bug where invalid f-strings didn’t raise the expected "Unknown format code" error when IAST was enabled.
  • Fixes an issue with PosixPath handling in path operations that could cause errors during taint tracking. This fix improves stability and slightly reduces import times.

• CI Visibility

  • Resolves an issue where the CODEOWNERS file of the project being tested was looked up in the current working directory rather than the repository root.
  • Resolves an issue where an I/O operation on closed file error was raised while printing ddtrace logs during a pytest session, due to a conflict with pytest's log capture.

• Profiling

  • Fixes an infinite recursion from calling logging module when lock profiler fails to delete _self_acquired_at attribute.

• Tracing

  • Resolves an issue where Django, Flask, Cherrypy, wsgi, asgi, pyramid, molten, falcon, tornado, aiohttp, bottle, rq, and sanic integrations were unnecessarily running code to activate distributed tracing headers multiple times per request. This is fixed by only activating distributed tracing headers once per request.
  • azure_functions: Resolves an issue where the function name set by the function_name decorator was ignored.
  • azure_functions: Resolves an issue where a function would not start if the trigger_arg_nameparameter was set to anything other than req.
  • azure_functions: Resolves an issue where timer triggers had a span kind of server instead of internal
  • graphql: Resolves AttributeError raised when used with older versions of graphql-core that do not have formatted property for SourceLocation.
  • opentelemetry: Updates trace flags in opentelemetry.trace.Span.get_span_context() to match OpenTelemetry specification.
  • subprocess: Removes unnecessary debug log variables.