Skip to content

chore: wrapping context support for Python 3.15#17849

Draft
P403n1x87 wants to merge 34 commits into
vlad/ci-auto-regen-requirementsfrom
chore/315-wrapping-context
Draft

chore: wrapping context support for Python 3.15#17849
P403n1x87 wants to merge 34 commits into
vlad/ci-auto-regen-requirementsfrom
chore/315-wrapping-context

Conversation

@P403n1x87

Copy link
Copy Markdown
Contributor

Description

We add support for the wrapping context for 3.15 by leveraging the low-impact monitoring API. With the latest, changes the API now allows making all events local per-code object, so this allows us to set local exception unwinding events to catch per-function thrown exceptions.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented May 1, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/internal/wrapping/asyncs.py                                     @DataDog/apm-core-python

@P403n1x87 P403n1x87 added the changelog/no-changelog A changelog entry is not required for this PR. label May 1, 2026
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented May 1, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

🚦 49 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | appsec/appsec 3/4   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | ci_visibility/testing 13/21   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | ci_visibility/testing 21/21   View in Datadog   GitLab

View all 49 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c369b05 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented May 1, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-18 04:29:11

Comparing candidate commit c369b05 in PR branch chore/315-wrapping-context with baseline commit 33504a3 in branch vlad/ci-auto-regen-requirements.

Found 0 performance improvements and 5 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics.

scenario:iastaspects-lstrip_aspect

  • 🟥 execution_time [+61.631µs; +70.961µs] or [+18.610%; +21.427%]

scenario:iastaspects-swapcase_noaspect

  • 🟥 execution_time [+19.190µs; +23.554µs] or [+7.134%; +8.757%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+104.204µs; +111.891µs] or [+24.130%; +25.910%]

scenario:span-start

  • 🟥 execution_time [+1.249ms; +1.431ms] or [+8.086%; +9.264%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+230.431ns; +264.541ns] or [+11.117%; +12.762%]

@P403n1x87
P403n1x87 force-pushed the chore/315-wrapping-context branch from c96d872 to a6faddd Compare May 6, 2026 10:32
@P403n1x87
P403n1x87 force-pushed the chore/315-wrapping-context branch from fa647f2 to 5e26fd2 Compare May 14, 2026 15:26
P403n1x87 added 3 commits May 14, 2026 16:30
We add support for the wrapping context for 3.15 by leveraging the
low-impact monitoring API. With the latest, changes the API now allows
making all events local per-code object, so this allows us to set local
exception unwinding events to catch per-function thrown exceptions.
@P403n1x87
P403n1x87 force-pushed the chore/315-wrapping-context branch from 2d33056 to cd95d95 Compare May 14, 2026 15:31
@vlad-scherbich

vlad-scherbich commented May 19, 2026

Copy link
Copy Markdown
Contributor

@P403n1x87 I've taken a look at the PR, and it makes sense to me for the most part. For the parts that I don't have much experience with, I've asked Claude for thoughts. Here are its findings, hopefully they will be helpful:

Must be fixed before merging

1. Hot-path callbacks iterate a mutable dict without synchronization (monitoring.py)

The callbacks (_on_py_start, _on_py_return, etc.) do:

entries = _registry.get(code)
# ...
for e in entries.values():
...

Meanwhile register() and unregister() mutate that same inner dict (entries[id(handler)] = entry / existing.pop(...)) while holding _registry_lock -- but the callbacks don't acquire the lock. If a register call adds or removes an entry between two iterations of the for loop (GIL can switch between next calls on the dict view), you get RuntimeError: dictionary changed size during iteration.

Fix options: (a) snapshot with list(entries.values()) in callbacks (one small allocation per event), (b) swap to a copy-on-write scheme where register/unregister replace the entire inner dict atomically (the reference assignment is GIL-atomic, so the callback always sees a consistent snapshot), or (c) hold the lock in callbacks (worst for latency).

Option (b) is the best tradeoff: zero allocation on the hot path, and register/unregister are cold.

Nice to have

2. Tool ID allocation starts at 0, competing with debuggers (monitoring.py)

for tid in range(6):
    try:
        sys.monitoring.use_tool_id(tid, "ddtrace")
...

IDs 0-2 are conventionally reserved (debugger, coverage, profiler). Starting from 0 means ddtrace could claim the debugger slot if no debugger is attached yet, then a later debugpy or pdb attach would fail to register. Starting from 3 (or iterating range(5, -1, -1) to prefer higher IDs) would be more neighborly.

3. _ENTER_FRAME_DEPTH = 3 is fragile (context.py)

_ENTER_FRAME_DEPTH = 3 if sys.version_info >= (3, 15) else 1

This assumes a fixed call-stack depth from the monitored function through the monitoring dispatch to enter. If CPython changes the monitoring callback invocation depth, or if the multiplexer adds/removes a level, this silently produces the wrong frame. Consider walking the stack looking for the code object that matches self.wrapped.code rather than assuming a depth.

@P403n1x87

Copy link
Copy Markdown
Contributor Author

Awesome, thanks!

register() and unregister() mutate that same inner dict (entries[id(handler)] = entry / existing.pop(...)) while holding _registry_lock

The reasoning here was that it would be unlikely to have mutation while callbacks are invoked, because these are generally installed on enablement (boot). However RC might violate this assumption, so it won't cost us much to be a bit defensive here.

IDs 0-2 are conventionally reserved (debugger, coverage, profiler). Starting from 0 means ddtrace could claim the debugger slot if no debugger is attached yet, then a later debugpy or pdb attach would fail to register. Starting from 3 (or iterating range(5, -1, -1) to prefer higher IDs) would be more neighborly.

We are a debugger as a matter of fact 🙁 but I guess it doesn't matter where we start with the ID so we can just comply.

3. _ENTER_FRAME_DEPTH = 3 is fragile (context.py)

Deliberate choice. This value should be fixed for each Python release, so the cost is at most 1 update every year. Still much better than updating a whole bunch of opcodes 🙂

build_base_venvs compiles every native extension (Rust _native, Cython,
IAST, profiling) at CMAKE_BUILD_PARALLEL_LEVEL=12. When a version's
ext_cache is cold -- e.g. a newly added Python version such as 3.15 --
the full 12-way parallel compile peaks well above the ~5Gi the VPA tunes
this job down to (observed "MemoryLimit changed from 6Gi to 5012971110"),
and the job is OOMKilled (exit 137) deterministically.

Pin CPU/memory and disable the VPA, matching the resource requests the
native package build jobs (.build_base, "test sdist") already use. This
keeps build parallelism intact and only affects cold-cache builds; warm
builds barely compile anything.
@vlad-scherbich
vlad-scherbich changed the base branch from main to ci/build-base-venvs-oom July 9, 2026 20:35
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jul 13, 2026
## Description

`build_base_venvs` is OOMKilled for cold-`ext_cache` Python versions. Currently this happens for `v3.15` only, which is new and has no warm cache. This was found while working on #17849.

**Root cause**
- It sets **no** `KUBERNETES_MEMORY_*` and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap **warm-cache** history.
- A cold version compiles everything at 12-way parallelism, which exceeds the approximated limit.

### Changes

Pin CPU/memory and disable the VPA on `build_base_venvs`, matching what `.build_base`, `test sdist` already use for the same build targets:

```yaml
KUBERNETES_CPU_REQUEST: '6'
KUBERNETES_MEMORY_REQUEST: '10Gi'
KUBERNETES_MEMORY_LIMIT: '10Gi'
DD_DISABLE_VPA: 'true'
```

## Test plan

* [Next PR](#17849) passes [dd-gitlab/build_base_venvs: [3.15]](https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-py/builds/1846141002)

<img width="494" height="472" alt="Screenshot 2026-07-09 at 4 25 46 PM" src="https://github.com/user-attachments/assets/c622de21-a232-4f23-a085-728ad4461d25" />
 

## Additional Notes

* Warm builds are unaffected.
* The failing job logs also show the GitLab **runner S3 cache returning 403 AccessDenied**, so `ext_cache` restore fails on every run and forces cold compiles every time. That's a runner/`ddbuild` platform-side credential issue. Fixing this separately would restores warm-build speed.

Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
Base automatically changed from ci/build-base-venvs-oom to main July 13, 2026 20:51
@vlad-scherbich
vlad-scherbich changed the base branch from main to vlad/ci-auto-regen-requirements July 14, 2026 15:13
@vlad-scherbich
vlad-scherbich changed the base branch from vlad/ci-auto-regen-requirements to main July 14, 2026 15:26
@vlad-scherbich
vlad-scherbich force-pushed the chore/315-wrapping-context branch from ad1e1ab to 3eef3aa Compare July 14, 2026 15:26
Turn check_requirements_lockfiles from a dead-end hard gate into a
self-healing step. On a PR branch, when the riot lockfiles or
requirements.csv drift, a new commit_requirements_lockfiles job re-applies
the regenerated files (produced in the testrunner image, which has every
interpreter incl. 3.15-dev) and pushes the fix back to the branch via the
dd-octo-sts GitLab->GitHub token, triggering a fresh pipeline.

Drift stays a hard gate on protected and merge-queue refs, where pushing is
unsafe. Requires the gitlab.regenerate-requirements.push octo-sts policy,
added separately in #19039 (octo-sts reads trust policies from the default
branch, so it must land on main first. Documents the behavior in
contributing-testing.rst.
EOF
)
@vlad-scherbich
vlad-scherbich changed the base branch from main to vlad/ci-auto-regen-requirements July 14, 2026 15:45
@vlad-scherbich
vlad-scherbich force-pushed the vlad/ci-auto-regen-requirements branch from 6e65907 to e9cc57d Compare July 15, 2026 02:51
vlad-scherbich added a commit that referenced this pull request Jul 15, 2026
## Description

`build_base_venvs` is OOMKilled for cold-`ext_cache` Python versions. Currently this happens for `v3.15` only, which is new and has no warm cache. This was found while working on #17849.

**Root cause**
- It sets **no** `KUBERNETES_MEMORY_*` and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap **warm-cache** history.
- A cold version compiles everything at 12-way parallelism, which exceeds the approximated limit.

### Changes

Pin CPU/memory and disable the VPA on `build_base_venvs`, matching what `.build_base`, `test sdist` already use for the same build targets:

```yaml
KUBERNETES_CPU_REQUEST: '6'
KUBERNETES_MEMORY_REQUEST: '10Gi'
KUBERNETES_MEMORY_LIMIT: '10Gi'
DD_DISABLE_VPA: 'true'
```

## Test plan

* [Next PR](#17849) passes [dd-gitlab/build_base_venvs: [3.15]](https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-py/builds/1846141002)

<img width="494" height="472" alt="Screenshot 2026-07-09 at 4 25 46 PM" src="https://github.com/user-attachments/assets/c622de21-a232-4f23-a085-728ad4461d25" />
 

## Additional Notes

* Warm builds are unaffected.
* The failing job logs also show the GitLab **runner S3 cache returning 403 AccessDenied**, so `ext_cache` restore fails on every run and forces cold compiles every time. That's a runner/`ddbuild` platform-side credential issue. Fixing this separately would restores warm-build speed.

Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
@vlad-scherbich
vlad-scherbich force-pushed the vlad/ci-auto-regen-requirements branch from df31c93 to 33504a3 Compare July 17, 2026 19:36
The monitoring-based WrappingContext path on 3.15 fixes t-string wrapping
and preserves __code__ on unwrap, so strict xfails now XPASS. Strip the
t-string xfails via conftest (the test file cannot be ruff-linted when
staged). Skip gevent tests on 3.15 until greenlet ships a compatible wheel.
…ush_null + load_common_constant) in async coroutine/generator assembly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants