chore: wrapping context support for Python 3.15#17849
Conversation
Codeowners resolved as |
|
BenchmarksBenchmark execution time: 2026-07-18 04:29:11 Comparing candidate commit c369b05 in PR branch Found 0 performance improvements and 5 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics. scenario:iastaspects-lstrip_aspect
scenario:iastaspects-swapcase_noaspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:span-start
scenario:telemetryaddmetric-1-count-metric-1-times
|
c96d872 to
a6faddd
Compare
fa647f2 to
5e26fd2
Compare
We add support for the wrapping context for 3.15 by leveraging the low-impact monitoring API. With the latest, changes the API now allows making all events local per-code object, so this allows us to set local exception unwinding events to catch per-function thrown exceptions.
2d33056 to
cd95d95
Compare
|
@P403n1x87 I've taken a look at the PR, and it makes sense to me for the most part. For the parts that I don't have much experience with, I've asked Claude for thoughts. Here are its findings, hopefully they will be helpful: Must be fixed before merging1. Hot-path callbacks iterate a mutable dict without synchronization (monitoring.py) The callbacks (_on_py_start, _on_py_return, etc.) do: Meanwhile register() and unregister() mutate that same inner dict (entries[id(handler)] = entry / existing.pop(...)) while holding _registry_lock -- but the callbacks don't acquire the lock. If a register call adds or removes an entry between two iterations of the for loop (GIL can switch between next calls on the dict view), you get RuntimeError: dictionary changed size during iteration. Fix options: (a) snapshot with list(entries.values()) in callbacks (one small allocation per event), (b) swap to a copy-on-write scheme where register/unregister replace the entire inner dict atomically (the reference assignment is GIL-atomic, so the callback always sees a consistent snapshot), or (c) hold the lock in callbacks (worst for latency). Option (b) is the best tradeoff: zero allocation on the hot path, and register/unregister are cold. Nice to have2. Tool ID allocation starts at 0, competing with debuggers (monitoring.py) IDs 0-2 are conventionally reserved (debugger, coverage, profiler). Starting from 0 means ddtrace could claim the debugger slot if no debugger is attached yet, then a later debugpy or pdb attach would fail to register. Starting from 3 (or iterating range(5, -1, -1) to prefer higher IDs) would be more neighborly. 3. _ENTER_FRAME_DEPTH = 3 is fragile (context.py)
This assumes a fixed call-stack depth from the monitored function through the monitoring dispatch to enter. If CPython changes the monitoring callback invocation depth, or if the multiplexer adds/removes a level, this silently produces the wrong frame. Consider walking the stack looking for the code object that matches self.wrapped.code rather than assuming a depth. |
|
Awesome, thanks!
The reasoning here was that it would be unlikely to have mutation while callbacks are invoked, because these are generally installed on enablement (boot). However RC might violate this assumption, so it won't cost us much to be a bit defensive here.
We are a debugger as a matter of fact 🙁 but I guess it doesn't matter where we start with the ID so we can just comply.
Deliberate choice. This value should be fixed for each Python release, so the cost is at most 1 update every year. Still much better than updating a whole bunch of opcodes 🙂 |
build_base_venvs compiles every native extension (Rust _native, Cython, IAST, profiling) at CMAKE_BUILD_PARALLEL_LEVEL=12. When a version's ext_cache is cold -- e.g. a newly added Python version such as 3.15 -- the full 12-way parallel compile peaks well above the ~5Gi the VPA tunes this job down to (observed "MemoryLimit changed from 6Gi to 5012971110"), and the job is OOMKilled (exit 137) deterministically. Pin CPU/memory and disable the VPA, matching the resource requests the native package build jobs (.build_base, "test sdist") already use. This keeps build parallelism intact and only affects cold-cache builds; warm builds barely compile anything.
## Description `build_base_venvs` is OOMKilled for cold-`ext_cache` Python versions. Currently this happens for `v3.15` only, which is new and has no warm cache. This was found while working on #17849. **Root cause** - It sets **no** `KUBERNETES_MEMORY_*` and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap **warm-cache** history. - A cold version compiles everything at 12-way parallelism, which exceeds the approximated limit. ### Changes Pin CPU/memory and disable the VPA on `build_base_venvs`, matching what `.build_base`, `test sdist` already use for the same build targets: ```yaml KUBERNETES_CPU_REQUEST: '6' KUBERNETES_MEMORY_REQUEST: '10Gi' KUBERNETES_MEMORY_LIMIT: '10Gi' DD_DISABLE_VPA: 'true' ``` ## Test plan * [Next PR](#17849) passes [dd-gitlab/build_base_venvs: [3.15]](https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-py/builds/1846141002) <img width="494" height="472" alt="Screenshot 2026-07-09 at 4 25 46 PM" src="https://github.com/user-attachments/assets/c622de21-a232-4f23-a085-728ad4461d25" /> ## Additional Notes * Warm builds are unaffected. * The failing job logs also show the GitLab **runner S3 cache returning 403 AccessDenied**, so `ext_cache` restore fails on every run and forces cold compiles every time. That's a runner/`ddbuild` platform-side credential issue. Fixing this separately would restores warm-build speed. Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
ad1e1ab to
3eef3aa
Compare
Turn check_requirements_lockfiles from a dead-end hard gate into a self-healing step. On a PR branch, when the riot lockfiles or requirements.csv drift, a new commit_requirements_lockfiles job re-applies the regenerated files (produced in the testrunner image, which has every interpreter incl. 3.15-dev) and pushes the fix back to the branch via the dd-octo-sts GitLab->GitHub token, triggering a fresh pipeline. Drift stays a hard gate on protected and merge-queue refs, where pushing is unsafe. Requires the gitlab.regenerate-requirements.push octo-sts policy, added separately in #19039 (octo-sts reads trust policies from the default branch, so it must land on main first. Documents the behavior in contributing-testing.rst. EOF )
3eef3aa to
29fa94b
Compare
6e65907 to
e9cc57d
Compare
## Description `build_base_venvs` is OOMKilled for cold-`ext_cache` Python versions. Currently this happens for `v3.15` only, which is new and has no warm cache. This was found while working on #17849. **Root cause** - It sets **no** `KUBERNETES_MEMORY_*` and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap **warm-cache** history. - A cold version compiles everything at 12-way parallelism, which exceeds the approximated limit. ### Changes Pin CPU/memory and disable the VPA on `build_base_venvs`, matching what `.build_base`, `test sdist` already use for the same build targets: ```yaml KUBERNETES_CPU_REQUEST: '6' KUBERNETES_MEMORY_REQUEST: '10Gi' KUBERNETES_MEMORY_LIMIT: '10Gi' DD_DISABLE_VPA: 'true' ``` ## Test plan * [Next PR](#17849) passes [dd-gitlab/build_base_venvs: [3.15]](https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-py/builds/1846141002) <img width="494" height="472" alt="Screenshot 2026-07-09 at 4 25 46 PM" src="https://github.com/user-attachments/assets/c622de21-a232-4f23-a085-728ad4461d25" /> ## Additional Notes * Warm builds are unaffected. * The failing job logs also show the GitLab **runner S3 cache returning 403 AccessDenied**, so `ext_cache` restore fails on every run and forces cold compiles every time. That's a runner/`ddbuild` platform-side credential issue. Fixing this separately would restores warm-build speed. Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
df31c93 to
33504a3
Compare
The monitoring-based WrappingContext path on 3.15 fixes t-string wrapping and preserves __code__ on unwrap, so strict xfails now XPASS. Strip the t-string xfails via conftest (the test file cannot be ruff-linted when staged). Skip gevent tests on 3.15 until greenlet ships a compatible wheel.
…ush_null + load_common_constant) in async coroutine/generator assembly
Description
We add support for the wrapping context for 3.15 by leveraging the low-impact monitoring API. With the latest, changes the API now allows making all events local per-code object, so this allows us to set local exception unwinding events to catch per-function thrown exceptions.