fix(setup): out-of-memory error [backport 1.18] (#6804)

Backport 07f59ac from #6792 to 1.18.

When running setup.py extensions with the CMake parameter "-j", it could
potentially raise an out-of-memory error. If someone wants to expedite
the ddtrace installation, they should manually set the
"CMAKE_BUILD_PARALLEL_LEVEL" environment variable.
## Checklist

- [X] Change(s) are motivated and described in the PR description.
- [X] Testing strategy is described if automated tests are not included
in the PR.
- [X] Risk is outlined (performance impact, potential for breakage,
maintainability, etc).
- [X] Change is maintainable (easy to change, telemetry, documentation).
- [X] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed. If no release note is required, add label
`changelog/no-changelog`.
- [X] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/)).
- [X] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist

- [x] Title is accurate.
- [x] No unnecessary changes are introduced.
- [x] Description motivates each change.
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes unless absolutely necessary.
- [x] Testing strategy adequately addresses listed risk(s).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] Release note makes sense to a user of the library.
- [x] Reviewer has explicitly acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment.
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Co-authored-by: Alberto Vara <[email protected]>

Author: github-actions[bot]

.github/workflows/build_deploy.yml

@@ -75,6 +75,7 @@ jobs:
           # `platform.mac_ver()` reports incorrect MacOS version at 11.0
           # See: https://stackoverflow.com/a/65402241
           CIBW_ENVIRONMENT_MACOS: SYSTEM_VERSION_COMPAT=0
+          CMAKE_BUILD_PARALLEL_LEVEL: 12
 
       - uses: actions/upload-artifact@v3
         with:

.github/workflows/build_python_3.yml

@@ -53,6 +53,7 @@ jobs:
           CIBW_BUILD: ${{ inputs.cibw_build }}
           CIBW_SKIP: ${{ inputs.cibw_skip }}
           CIBW_PRERELEASE_PYTHONS: ${{ inputs.cibw_prerelease_pythons }}
+          CMAKE_BUILD_PARALLEL_LEVEL: 12
 
       - uses: actions/upload-artifact@v3
         with:

.github/workflows/system-tests.yml

@@ -42,6 +42,7 @@ jobs:
       # that make assertion on backend data. Using a fake key allow to run system tests on PR originating from forks.
       # If ever it's needed, a valid key exists in the repo, using ${{ secrets.DD_API_KEY }}
       DD_API_KEY: 1234567890abcdef1234567890abcdef
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     steps:
       - name: Setup python 3.9
         if: needs.needs-run.outputs.outcome == 'success'

.github/workflows/test_frameworks.yml

@@ -33,6 +33,7 @@ jobs:
     env:
       DD_PROFILING_ENABLED: true
       DD_TESTING_RAISE: true
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: bottle
@@ -80,6 +81,7 @@ jobs:
     env:
       # Regression test for DataDog/dd-trace-py/issues/5722
       DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE: true
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: sanic
@@ -131,6 +133,7 @@ jobs:
       DD_DEBUGGER_EXPL_COVERAGE_DELETE_LINE_PROBES: 1  # Delete to speed up
       DD_DEBUGGER_EXPL_CONSERVATIVE: 1
       PYTHONPATH: ../ddtrace/tests/debugging/exploration/:.
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: django
@@ -194,6 +197,7 @@ jobs:
       DD_PROFILING_ENABLED: true
       DD_TESTING_RAISE: true
       PYTHONPATH: ../ddtrace/tests/debugging/exploration/:.
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: graphene
@@ -235,6 +239,7 @@ jobs:
     env:
       DD_TESTING_RAISE: true
       DD_PROFILING_ENABLED: true
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: fastapi
@@ -277,6 +282,7 @@ jobs:
       DD_TESTING_RAISE: true
       DD_PROFILING_ENABLED: true
       PYTHONPATH: ../ddtrace/tests/debugging/exploration/
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: flask
@@ -362,6 +368,7 @@ jobs:
       DD_TESTING_RAISE: true
       DD_PROFILING_ENABLED: true
       PYTHONPATH: ../ddtrace/tests/debugging/exploration/
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: mako
@@ -407,6 +414,7 @@ jobs:
       DD_TESTING_RAISE: true
       DD_PROFILING_ENABLED: true
       PYTHONPATH: ../ddtrace/tests/debugging/exploration/
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: starlette
@@ -447,6 +455,7 @@ jobs:
     env:
       DD_TESTING_RAISE: true
       DD_PROFILING_ENABLED: true
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: requests
@@ -485,6 +494,7 @@ jobs:
     env:
       DD_TESTING_RAISE: true
       DD_PROFILING_ENABLED: true
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: asyncpg
@@ -528,6 +538,7 @@ jobs:
     env:
       DD_TESTING_RAISE: true
       PYTHONPATH: ../ddtrace/tests/debugging/exploration/
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: pylons
@@ -570,6 +581,7 @@ jobs:
     env:
       DD_TESTING_RAISE: true
       # PYTHONPATH: ../ddtrace/tests/debugging/exploration/
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: gunicorn
@@ -604,6 +616,7 @@ jobs:
       DD_TESTING_RAISE: true
       DD_PROFILING_ENABLED: true
       PYTHONPATH: ../ddtrace/tests/debugging/exploration/
+      CMAKE_BUILD_PARALLEL_LEVEL: 12
     defaults:
       run:
         working-directory: uwsgi

.gitlab/benchmarks.yml

@@ -10,6 +10,7 @@ variables:
   timeout: 1h
   script:
     - export REPORTS_DIR="$(pwd)/reports/" && (mkdir "${REPORTS_DIR}" || :)
+    - export CMAKE_BUILD_PARALLEL_LEVEL=12
     - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/".insteadOf "https://github.com/DataDog/"
     - git clone --branch dd-trace-py https://github.com/DataDog/relenv-microbenchmarking-platform /platform && cd /platform
     - ./steps/capture-hardware-software-info.sh

docs/spelling_wordlist.txt

@@ -251,3 +251,4 @@ programmatically
 DES
 Blowfish
 Gitlab
+CMake

releasenotes/notes/setup-oom-errors-426c9951d5c9ed57.yaml

@@ -0,0 +1,6 @@
+---
+issues:
+  - |
+    When running setup.py extensions with the CMake parameter "-j", it could potentially raise an out-of-memory error. 
+    If someone wants to expedite the ddtrace installation, they should manually set the "CMAKE_BUILD_PARALLEL_LEVEL" 
+    environment variable.

riotfile.py

@@ -101,6 +101,7 @@ def select_pys(min_version=MIN_PYTHON_VERSION, max_version=MAX_PYTHON_VERSION):
         "DD_CIVISIBILITY_AGENTLESS_ENABLED": "1",
         "DD_CIVISIBILITY_CODE_COVERAGE_ENABLED": "1",
         "DD_CIVISIBILITY_ITR_ENABLED": "1",
+        "CMAKE_BUILD_PARALLEL_LEVEL": "12",
     },
     venvs=[
         Venv(

setup.py

@@ -345,9 +345,6 @@ def build_extension(self, ext):
                 # DEV: -j is only supported in CMake 3.12+ only.
                 if hasattr(self, "parallel") and self.parallel:
                     build_args += ["-j{}".format(self.parallel)]
-                else:
-                    # Let CMake determine the parallelism to use
-                    build_args += ["-j"]
             try:
                 cmake_cmd_with_args = [cmake_command] + cmake_args
                 subprocess.run(cmake_cmd_with_args, cwd=tmp_iast_path, check=True)
@@ -462,7 +459,7 @@ def get_exts_for(name):
         )
 
         if sys.version_info >= (3, 6, 0):
-            ext_modules.append(Extension("ddtrace.appsec.iast._taint_tracking._native", sources=[], parallel=8))
+            ext_modules.append(Extension("ddtrace.appsec.iast._taint_tracking._native", sources=[]))
 else:
     ext_modules = []