Skip to content

Commit 3c79a02

Browse files
fix(tracing): prevent google.protobuf import error when unloading modules [backport 1.16] (#6359)
Backport d29c860 from #6346 to 1.16. This fix resolves an issue where the `google` module is left in a degraded state and causes import error for `google.protobuf`. This issue only happens when module unloading is used (gevent is present, or manually enabled), and when stats computation is enabled. The example scenario is present when importing `confluent-kafka`, but is possible to happen with other integrations as well. ``` ❯ DD_TRACE_COMPUTE_STATS=true ddtrace-run python -c "import confluent_kafka" failed to import ddtrace module 'ddtrace.contrib.kafka' when patching on import Traceback (most recent call last): File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/_monkey.py", line 159, in on_import imported_module = importlib.import_module(path) File "/Users/brett.langdon/.pyenv/versions/3.10.8/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/internal/module.py", line 220, in _exec_module self.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/contrib/kafka/__init__.py", line 51, in <module> from .patch import patch File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/internal/module.py", line 220, in _exec_module self.loader.exec_module(module) File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/contrib/kafka/patch.py", line 14, in <module> from ddtrace.internal.datastreams.processor import PROPAGATION_KEY File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/internal/module.py", line 220, in _exec_module self.loader.exec_module(module) File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/internal/datastreams/processor.py", line 17, in <module> from ddsketch.pb.proto import DDSketchProto File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddtrace/internal/module.py", line 220, in _exec_module self.loader.exec_module(module) File "/private/tmp/hoki/venv/lib/python3.10/site-packages/ddsketch/pb/proto.py", line 12, in <module> pb_version = tuple(map(int, google.protobuf.__version__.split(".")[0:2])) AttributeError: module 'google' has no attribute 'protobuf' ``` ## Testing I was unable to create a reproduction case using subprocess tests, to validate manually: Via docker: ``` docker run --rm -it python:3.11 /bin/bash # pip install ddtrace confluent-kafka # DD_TRACE_COMPUTE_STATS=true DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE=true ddtrace-run python -c "import confluent_kafka" ``` Locally: ``` virtualenv --python=3.11 venv source venv/bin/activate pip install ddtrace confluent-kafka DD_TRACE_COMPUTE_STATS=true DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE=true ddtrace-run python -c "import confluent_kafka" ``` ## Checklist - [x] Change(s) are motivated and described in the PR description. - [x] Testing strategy is described if automated tests are not included in the PR. - [x] Risk is outlined (performance impact, potential for breakage, maintainability, etc). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) are followed. If no release note is required, add label `changelog/no-changelog`. - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)). - [x] Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Title is accurate. - [x] No unnecessary changes are introduced. - [x] Description motivates each change. - [x] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Testing strategy adequately addresses listed risk(s). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] Release note makes sense to a user of the library. - [x] Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment. - [x] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Brett Langdon <[email protected]>
1 parent 0297c4f commit 3c79a02

File tree

2 files changed

+5
-0
lines changed

2 files changed

+5
-0
lines changed

ddtrace/bootstrap/sitecustomize.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,7 @@ def drop(module_name):
111111
"sre_constants", # imported by re at runtime
112112
"logging",
113113
"attr",
114+
"google",
114115
"google.protobuf", # the upb backend in >= 4.21 does not like being unloaded
115116
]
116117
)
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
---
2+
fixes:
3+
- |
4+
tracing: This fix resolves a ``google.protobuf`` import error when module unloading.

0 commit comments

Comments
 (0)