ci: pin build_base_venvs memory to prevent cold-cache OOM by vlad-scherbich · Pull Request #18962 · DataDog/dd-trace-py

vlad-scherbich · 2026-07-09T14:53:16Z

Description

build_base_venvs is OOMKilled for cold-ext_cache Python versions. Currently this happens for v3.15 only, which is new and has no warm cache. This was found while working on #17849.

Root cause

It sets no KUBERNETES_MEMORY_* and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap warm-cache history.
A cold version compiles everything at 12-way parallelism, which exceeds the approximated limit.

Changes

Pin CPU/memory and disable the VPA on build_base_venvs, matching what .build_base, test sdist already use for the same build targets:

KUBERNETES_CPU_REQUEST: '6'
KUBERNETES_MEMORY_REQUEST: '10Gi'
KUBERNETES_MEMORY_LIMIT: '10Gi'
DD_DISABLE_VPA: 'true'

Test plan

Next PR passes dd-gitlab/build_base_venvs: [3.15]

Additional Notes

Warm builds are unaffected.
The failing job logs also show the GitLab runner S3 cache returning 403 AccessDenied, so ext_cache restore fails on every run and forces cold compiles every time. That's a runner/ddbuild platform-side credential issue. Fixing this separately would restores warm-build speed.

build_base_venvs compiles every native extension (Rust _native, Cython, IAST, profiling) at CMAKE_BUILD_PARALLEL_LEVEL=12. When a version's ext_cache is cold -- e.g. a newly added Python version such as 3.15 -- the full 12-way parallel compile peaks well above the ~5Gi the VPA tunes this job down to (observed "MemoryLimit changed from 6Gi to 5012971110"), and the job is OOMKilled (exit 137) deterministically. Pin CPU/memory and disable the VPA, matching the resource requests the native package build jobs (.build_base, "test sdist") already use. This keeps build parallelism intact and only affects cold-cache builds; warm builds barely compile anything.

cit-pr-commenter-54b7da · 2026-07-09T14:55:00Z

Codeowners resolved as

.gitlab/templates/build-base-venvs.yml                                  @DataDog/python-guild @DataDog/apm-core-python

cit-pr-commenter-54b7da · 2026-07-09T14:55:08Z

Circular import analysis

⚠️ Existing circular imports

There are 4974 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 4974 shortest)

ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.botocore -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

datadog-datadog-prod-us1 · 2026-07-09T14:55:13Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 8 tests - 8 passed on retry

🚧 3 tests that failed were ignored due to quarantine

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 88c0f8b | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-07-09T15:41:37Z

Benchmarks

Benchmark execution time: 2026-07-13 19:23:26

Comparing candidate commit 88c0f8b in PR branch ci/build-base-venvs-oom with baseline commit cf59b87 in branch main.

Found 0 performance improvements and 6 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics.

scenario:iastaspects-index_aspect

🟥 execution_time [+10.399µs; +14.986µs] or [+8.058%; +11.612%]

scenario:iastaspects-ljust_aspect

🟥 execution_time [+76.066µs; +84.065µs] or [+13.059%; +14.433%]

scenario:iastaspects-swapcase_aspect

🟥 execution_time [+54.068µs; +61.367µs] or [+15.374%; +17.450%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+98.211µs; +106.091µs] or [+23.326%; +25.197%]

scenario:span-start

🟥 execution_time [+1.259ms; +1.440ms] or [+7.948%; +9.088%]

scenario:tracer-small

🟥 execution_time [+27.517µs; +30.661µs] or [+8.016%; +8.932%]

Copilot

Pull request overview

This PR updates the GitLab CI build_base_venvs job to avoid cold-cache OOMKills by explicitly pinning Kubernetes CPU/memory requests/limits and disabling VPA tuning, aligning its runtime constraints with other native-build jobs in the pipeline.

Changes:

Add KUBERNETES_CPU_REQUEST, KUBERNETES_MEMORY_REQUEST, and KUBERNETES_MEMORY_LIMIT to build_base_venvs.
Set DD_DISABLE_VPA to prevent VPA from clamping memory during cold-cache native extension builds.
Document the rationale inline in the job’s variables section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

## Description `build_base_venvs` is OOMKilled for cold-`ext_cache` Python versions. Currently this happens for `v3.15` only, which is new and has no warm cache. This was found while working on #17849. **Root cause** - It sets **no** `KUBERNETES_MEMORY_*` and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap **warm-cache** history. - A cold version compiles everything at 12-way parallelism, which exceeds the approximated limit. ### Changes Pin CPU/memory and disable the VPA on `build_base_venvs`, matching what `.build_base`, `test sdist` already use for the same build targets: ```yaml KUBERNETES_CPU_REQUEST: '6' KUBERNETES_MEMORY_REQUEST: '10Gi' KUBERNETES_MEMORY_LIMIT: '10Gi' DD_DISABLE_VPA: 'true' ``` ## Test plan * [Next PR](#17849) passes [dd-gitlab/build_base_venvs: [3.15]](https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-py/builds/1846141002) <img width="494" height="472" alt="Screenshot 2026-07-09 at 4 25 46 PM" src="https://github.com/user-attachments/assets/c622de21-a232-4f23-a085-728ad4461d25" /> ## Additional Notes * Warm builds are unaffected. * The failing job logs also show the GitLab **runner S3 cache returning 403 AccessDenied**, so `ext_cache` restore fails on every run and forces cold compiles every time. That's a runner/`ddbuild` platform-side credential issue. Fixing this separately would restores warm-build speed. Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>

vlad-scherbich requested review from a team as code owners July 9, 2026 14:53

vlad-scherbich requested a review from P403n1x87 July 9, 2026 14:53

vlad-scherbich added the changelog/no-changelog A changelog entry is not required for this PR. label Jul 9, 2026

vlad-scherbich requested a review from Copilot July 9, 2026 15:52

Copilot started reviewing on behalf of vlad-scherbich July 9, 2026 15:52 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread .gitlab/templates/build-base-venvs.yml

ci: clarify parallelism in build_base_venvs resource comment

9ff9307

This comment was marked as resolved.

Sign in to view

Merge branch 'main' into ci/build-base-venvs-oom

88c0f8b

gh-worker-dd-mergequeue-cf854d Bot merged commit bb53f37 into main Jul 13, 2026
1180 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the ci/build-base-venvs-oom branch July 13, 2026 20:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: pin build_base_venvs memory to prevent cold-cache OOM#18962

ci: pin build_base_venvs memory to prevent cold-cache OOM#18962
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
ci/build-base-venvs-oom

vlad-scherbich commented Jul 9, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 9, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 9, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

pr-commenter Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

vlad-scherbich commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Test plan

Additional Notes

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 9, 2026

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 9, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:iastaspects-index_aspect

scenario:iastaspects-ljust_aspect

scenario:iastaspects-swapcase_aspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:span-start

scenario:tracer-small

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vlad-scherbich commented Jul 9, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 9, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 9, 2026 •

edited by datadog-official Bot

Loading

pr-commenter Bot commented Jul 9, 2026 •

edited

Loading