Skip to content

3.0.0

Choose a tag to compare

View all tags
@erikayasuda erikayasuda released this 20 Feb 18:21
v3.0.0
a103a32
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Migration Guide: Upgrading from dd-trace-py v2 to v3

We are happy to announce the release of v3.0.0 of ddtrace. This release drops support for Python 3.7 and many previously-deprecated parts of the library interface.

The 2.x release line enters Maintenance Mode with the release of 3.0.0, meaning that it will only receive bugfix changes on its last few minor releases, and 2.21 will be the last 2.x minor version. See the versioning policy for support level definitions.

Summary of Significant Changes

  • Removed support for Python 3.7 from the library

CI Visibility

  • The new pytest plugin is now the default.
  • Module, suite, and test names are now parsed from item.nodeid.
  • Test names now include the class name (TestClass::test_method) for class-based tests.
  • Test skipping is now done at the suite level.
  • The DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable is no longer used.

LLM Observability

  • Removed support for DD_LLMOBS_APP_NAME.
  • The parameters argument in LLMObs.annotate() has been removed. Use metadata instead.
  • Dropped support for OpenAI 0.x and Langchain v0.0.x.

Tracing

  • Removed all deprecated interfaces from integrations.
  • DD_TRACE_PROPAGATION_STYLE=b3 single header is no longer supported. Use b3 instead.
  • Dropped support for multiple Tracer instances. The global tracer instance ddtrace.tracer must be used.
  • Removed deprecated parameters from Tracer.configure(...).
  • Dropped support for several deprecated tracing configurations. See “Environment Variables” section below for details.

Recommended Pre-Upgrade Steps

Before upgrading to v3, check for deprecation warnings in your application using the latest v2 release (2.21.0):

Detect Deprecation Warnings in Tests

Enable warnings as errors in pytest:

pytest -W "error::ddtrace.DDTraceDeprecationWarning" tests.py

Detect Deprecation Warnings in Applications

Set the following environment variable to enable all deprecation warnings:

PYTHONWARNINGS=all python app.py

Environment Variables

The following environment variables have been removed or replaced in v3. Use the table below to update your configuration:

v2 Removed in v3 Replaced/Changed in v3
LLM Observability
DD_LLMOBS_APP_NAME Use DD_LLMOBS_ML_APP instead
_DD_LLMOBS_EVALUATOR_SAMPLING_RULES Use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead
_DD_LLMOBS_EVALUATORS Use DD_LLMOBS_EVALUATORS instead
Tracing
DD_ANALYTICS_ENABLED No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail.
DD_HTTP_CLIENT_TAG_QUERY_STRING Use DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING.
DD_TRACE_ANALYTICS_ENABLED No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail.
DD_TRACE_METHODS (using [] notation) Must use : notation instead, For example: mod.submod:method1,method2;mod.submod:Class.method1. (docs)
DD_TRACE_PROPAGATION_STYLE=”b3 single header” Use DD_TRACE_PROPAGATION_STYLE=b3 for identical behavior
DD_TRACE_SAMPLE_RATE Use DD_TRACE_SAMPLING_RULES (docs).
DD_TRACE_SPAN_AGGREGATOR_RLOCK No replacement (feature removed). Config variable is now a no-op.
CI Visibility
DD_PYTEST_USE_NEW_PLUGIN_BETA No replacement (new pytest plugin is not in beta anymore).

Use the following patterns to identify the deprecated environment variables in a code base:

git grep -P -e "DD_LLMOBS_APP_NAME" \
  -e "_DD_LLMOBS_EVALUATOR_SAMPLING_RULES" \
  -e "_DD_LLMOBS_EVALUATORS" \
  -e "DD_TRACE_PROPAGATION_STYLE=.*b3 single header" \
  -e "DD_TRACE_SAMPLE_RATE" \
  -e "DD_TRACE_API_VERSION=v0.3" \
  -e "DD_ANALYTICS_ENABLED" \
  -e "DD_TRACE_ANALYTICS_ENABLED" \
  -e "DD_HTTP_CLIENT_TAG_QUERY_STRING" \
  -e "DD_TRACE_SPAN_AGGREGATOR_RLOCK" \
  -e "DD_TRACE_METHODS=.*\[\]"

Replace deprecated settings with the recommended alternatives where applicable.

NOTE: The changes to environment variables apply only to the configuration of the dd-trace-py library and not the Datadog Agent.

Legacy Interfaces

The following methods and module attributes have been removed or changed. Unless noted, removal means that the functionality pointed to by the removed interface has been disabled.

v2 v3
Tracing
Tracer.configure(enabled: Optional[bool] = None, hostname: Optional[str] = None, port: Optional[int] = None, uds_path: Optional[str] = None, https: Optional[bool] = None, sampler: Optional[BaseSampler] = None, context_provider: Optional[DefaultContextProvider] = None, wrap_executor: Optional[Callable] = None, priority_sampling: Optional[bool] = None, settings: Optional[Dict[str, Any]] = None, dogstatsd_url: Optional[str] = None, writer: Optional[TraceWriter] = None, partial_flush_enabled: Optional[bool] = None, partial_flush_min_spans: Optional[int] = None, api_version: Optional[str] = None, compute_stats_enabled: Optional[bool] = appsec_enabled: Optional[bool] = None, iast_enabled: Optional[bool] = None, appsec_standalone_enabled: Optional[bool] = None, trace_processors: Optional[List[TraceProcessor]] = None ) Tracer.configure(context_provider: Optional[BaseContextProvider] = None, compute_stats_enabled: Optional[bool] = None, appsec_enabled: Optional[bool] = None, iast_enabled: Optional[bool] = None, apm_tracing_disabled: Optional[bool] = None, trace_processors: Optional[List[TraceProcessor]] = None)
ddtrace.opentracer.tracer.Tracer()._dd_tracer ddtrace.tracer
ddtrace.tracer.sampler.rules[].choose_matcher(lambda: True) ddtrace.tracer.sampler.rules[].choose_matcher(“foo”) Callables and regex patterns are no longer allowed as arguments to choose_matcher
Span.sampled Span.context.sampling_priority > 0
LLM Observability
LLMObs.annotate(parameters=foo) LLMObs.annotate(metadata=foo)

Python 3.13 Support

Much of the library’s functionality has been compatible with Python 3.13 since version 2.20. Some pieces of functionality do not yet work with Python 3.13. Here’s what does not work under Python 3.13 as of dd-trace-py 3.0. 'Not tested' indicates that while the feature may be compatible with Python 3.13, its functionality has not been verified, as it is not included in our automated test coverage.

  • dd-trace-py doesn’t work on Windows with Python 3.13
  • Appsec Threat Detection is not tested against Django, Flask, or FastAPI with 3.13
  • Automatic Service Naming is not tested with 3.13
  • The following products are not tested with 3.13:
    • Code Coverage
    • Appsec IAST
    • Data Streams Monitoring
    • CI Visibility
    • Continuous Profiling
  • The following integrations are not tested with 3.13:
    • anthropic
    • consul
    • freezegun
    • gevent
    • google_generativeai
    • gunicorn
    • langchain
    • mysqlclient
    • opentracing
    • psycopg
    • psycopg2
    • pymysql
    • pytest
    • pytest-bdd
    • pytest-benchmark
    • sanic
    • selenium
    • sqlite3
    • starlette
    • tornado
    • vertexai

By following these guidelines, you can transition to dd-trace-py v3 gradually, ensuring minimal disruption to your services while taking advantage of the latest features and improvements. Happy migrating!

New Features

  • Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.

  • ASM

    • Introduces use of meta struct for appsec event reports. This will improve performances. ASM will now require at least agent version 7.35.0 to work as expected.

  • Code Security (IAST)

    • Introduces Code Security support to PyMySQL and MySQLDB integrations.
    • XSS detection for Django applications and Jinja2 (Flask and FastAPI applications), which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
    • Implements the detection of the Stacktrace-Leak vulnerability for Django, Flask and FastAPI.

  • LLM Observability

    • valkey: Adds automatic instrumentation of the Valkey package. Thank you AhmadMasry!

    • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.

    • Introduces an integration with the RAGAS evaluation framework to continuously monitor the performance of context-augmented LLM generations in production.
      The integration supports evaluating LLM inferences with the following RAGAS metrics:

      • Faithfulness: measures if the LLM response is faithful to the provided context.
      • Answer Relevancy: measures how relevant the LLM response is to the user input.
      • Context Precision: measures how effectively the context is used in the generated response.

      To learn more, see the LLM Observability evaluations guide.

Upgrade Notes

  • CI Visibility

    • Official release of the new version of the pytest plugin, introducing the following features:

      NOTE: this new version of the plugin introduces breaking changes:

      • module, suite, and test names are now parsed from the item.nodeid attribute
      • test names now include the class for class-based tests
      • Test skipping by Test Impact Analysis (formerly Intelligent Test Runner) is now done at the suite level, instead of at the test level

      A beta version of the plugin had been available since v2.18.0, and could be enabled via the DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable. The new version is now the default, and the environment variable is not used anymore.

  • LLM Observability

    • langchain: Removes the langchain.tokens.total_cost span metric for OpenAI calls. For continued cost estimation of OpenAI calls, enable LLM Observability.

    • langchain: Removes prompt-completion log sampling from the LangChain integration. To continue logging prompt completions, enable LLM Observability.

    • langchain: Removes integration metrics from the LangChain integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.

    • langchain: removed tracing support for langchain.chains.base.Chains. To maintain support, migrate to using Langchain's LCEL instead.
      - langchain: Drops tracing and LLM Observability support for Langchain v0.0.x versions.

    • openai: Removes prompt-completion log sampling from the OpenAI integration. To continue logging prompt completions, enable LLM Observability.

    • openai: Removes integration metrics from the OpenAI integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.

    • openai: Drops tracing and LLM Observability support for OpenAI 0.x versions. To maintain support, upgrade to openai>=1.0.

    • Support for the deprecated environment variable DD_LLMOBS_APP_NAME has been removed.

    • Removes the deprecated parameters argument from LLMObs annotate(). Use metadata instead.

  • Profiling

    • Bumps libdatadog dependency to v15.0.0.

  • Tracing

    • configurations: Drops support for deprecated tracing configurations. The following configurations are no longer supported:
      • DD_TRACE_SAMPLE_RATE, use DD_TRACE_SAMPLING_RULES instead.
      • DD_TRACE_API_VERSION=v0.3, the default v0.5 version is used instead.
      • DD_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.
      • DD_TRACE_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.
      • DD_HTTP_CLIENT_TAG_QUERY_STRING, DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING should be used instead.
      • DD_TRACE_SPAN_AGGREGATOR_RLOCK, disabling the span aggregator rlock is no longer supported.
    • aws_lambda: Drops support for aws_lambda<=6.105.0.
    • configurations: Drops support for configuring DD_TRACE_METHODS with the '[]' notation. Ensure DD_TRACE_METHODS use the ':' notation instead".
    • integrations: Removes deprecated interfaces from all integrations.
    • opentracing: Removes the deprecated ddtracer parameter from ddtrace.opentracer.tracer.Tracer().
    • propagation: Setting DD_TRACE_PROPAGATION_STYLE to b3 single header is no longer supported. Use b3 instead.
    • sampling: Drops support for configuring sampling rules using functions and regex in the ddtrace.tracer.sampler.rules[].choose_matcher(...) method and removes the timestamp_ns parameter from ddtrace.internal.rate_limiter.RateLimiter.is_allowed().
    • span: Removes the deprecated Span.sampled property
    • Drops support for multiple Tracer instances in the same process. Use ddtrace.trace.tracer to access the global tracer instance.
    • Removes support for overriding the global tracer in ddtrace.trace.Pin
    • Removes deprecated parameters from Tracer.configure(...) method and removes the Tracer.sampler attribute.
    • Drops support for multiple tracer instances, ddtrace.trace.Tracer can not be reinitialized.

Deprecation Notes

  • ASM

    • Deprecated env var DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING is now removed. Please use the new env var DD_APPSEC_AUTO_USER_INSTRUMENTATION_MODE instead.

  • LLM Observability

    • The _DD_LLMOBS_EVALUATORS environment variable is deprecated and will be removed in ddtrace 3.0.0. As an alternative to _DD_LLMOBS_EVALUATORS, you can use DD_LLMOBS_EVALUATORS instead. To migrate, replace _DD_LLMOBS_EVALUATORS with DD_LLMOBS_EVALUATORS.
    • _DD_LLMOBS_EVALUATOR_SAMPLING_RULES environment variable is deprecated and will be removed in ddtrace 3.0.0. As an alternative to _DD_LLMOBS_EVALUATOR_SAMPLING_RULES, you can use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead. To migrate, replace _DD_LLMOBS_EVALUATOR_SAMPLING_RULES with DD_LLMOBS_EVALUATOR_SAMPLING_RULES.

  • Tracing

    • Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.
    • Deprecates the following constants in ddtrace.constants module:
      • ANALYTICS_SAMPLE_RATE_KEY
      • SAMPLE_RATE_METRIC_KEY
      • SAMPLING_PRIORITY_KEY
      • SAMPLING_AGENT_DECISION
      • SAMPLING_RULE_DECISION
      • SAMPLING_LIMIT_DECISION
      • ORIGIN_KEY
      • USER_ID_KEY
      • HOSTNAME_KEY
      • RUNTIME_FAMILY
      • BASE_SERVICE_KEY
      • SPAN_MEASURED_KEY
      • KEEP_SPANS_RATE_KEY
      • MULTIPLE_IP_HEADERS
      • CONFIG_ENDPOINT_ENV
      • CONFIG_ENDPOINT_RETRIES_ENV
      • CONFIG_ENDPOINT_TIMEOUT_ENV
    • Internalizes the ddtrace.settings.config module and deprecates the following ddtrace.config attributes:
      • http, use DD_TRACE_HEADER_TAGS environment variable instead.
      • http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.
      • trace_headers, this attribute is internal to the tracer.
      • header_is_traced, this attribute is internal to the tracer.
      • convert_rc_trace_sampling_rules, this attribute is internal to the tracer.
      • enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.
      • get_from, use ddtrace.trace.Pin to set instance level configurations.
    • configurations: Removes deprecated attributes from ddtrace.config.

Bug Fixes

  • ASM

    • Ensures that no module from ASM are loaded when ASM is disabled or unavailable.
    • Resolves an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.

  • CI Visibility

    • Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.

  • Code security (IAST)

    • SCA: Ensures that no module from IAST are loaded when IAST is disabled or unavailable.
    • Resolves an issue where the usage of callonce decorator could trigger an import loop

  • LLM Observability

    • openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
    • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.
    • Resolves an issue where successive calls to LLMObs.annotate() on the same span caused overwriting of metrics, metadata, and prompt dictionaries. Now, calling LLMObs.annotate() will merge new metrics/metadata/prompt values with any existing ones.
    • Resolves an issue where spans were being enqueued to an inactive evaluator runner which caused noisy logs related to the evaluator runner buffer being full.
    • Resolves an issue where explicitly only using LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
    • Resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
    • Resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError.

  • Profiling

    • Fixes an issue on arm64 macOS where the profiler was not able to find native extension modules to enable the following features: DD_PROFILING_TIMELINE_ENABLED and DD_PROFILING_STACK_V2_ENABLED.
    • Fixes SystemError from the memory profiler returning NULL when collecting events

  • Tracing

    • aiohttp: Adds the environment variable DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.
    • internal: Fix performance overhead of Python distribution parsing for internal telemetry.
    • logging: Resolves an an unneeded info log being logged on process exit due to a forksafe hook being unregistered that was never registered to begin with.

Other Changes

  • Tracing
    • cassandra, cherrypy, flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.
    • Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
    • Removes the deprecated tracing modules and constants from the ddtrace package.
Assets 2
Loading