fix(lib-injection): install from local wheels [backport #5576 to 1.11] (#5579)

Backports #5576 to 1.11.

Change the installation mechanism to only download the wheels and
install from the wheels on process start.

Formerly, the wheels were merged together. This causes issues as ddtrace
has versioned dependencies based on the Python version. eg. bytecode
0.13 is required for Python <3.8 and 0.14 is required for >3.8. These
packages were erroneously merged together.

The risk of this change is that the mechanism is back to `pip
install`-ing at runtime which means:

- Performance overhead for application startup.
- Incorrect version of `pip` / `python` are used.

The image size might also be bigger as the ddtrace files aren't deduped.
Hopefully the compression used by the image registries is smart enough
to figure this out.

These tradeoffs are better than having a broken install however.


The automated tests that we have already should cover this change.


Fixes #5565, #5512

## Checklist

- [x] Change(s) are motivated and described in the PR description.
- [x] Testing strategy is described if automated tests are not included
in the PR.
- [x] Risk is outlined (performance impact, potential for breakage,
maintainability, etc).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines)
are followed.
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/)).
- [x] PR description includes explicit acknowledgement/acceptance of the
performance implications of this PR as reported in the benchmarks PR
comment.

## Reviewer Checklist

- [x] Title is accurate.
- [x] No unnecessary changes are introduced.
- [x] Description motivates each change.
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes unless absolutely necessary.
- [x] Testing strategy adequately addresses listed risk(s).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] Release note makes sense to a user of the library.
- [x] Reviewer has explicitly acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment.

Co-authored-by: Kyle Verhoog <[email protected]>
Co-authored-by: Emmett Butler <[email protected]>

Author: 3 people

.github/workflows/lib-injection.yml

@@ -62,6 +62,7 @@ jobs:
         run: |
           docker run \
             -d \
+            --name=testagent \
             --network=test-inject \
             -p 8126:8126 \
             ghcr.io/datadog/dd-apm-test-agent/ddapm-test-agent:v1.7.2
@@ -70,14 +71,13 @@ jobs:
           cd lib-injection
           mkdir -p lib-injection/ddtrace_pkgs
           cp sitecustomize.py lib-injection/
-          ./dl_megawheel.py \
+          ./dl_wheels.py \
             --ddtrace-version=v1.10 \
             --python-version=3.11 \
             --python-version=3.10 \
             --python-version=3.9 \
             --python-version=3.8 \
             --python-version=3.7 \
-            --ddtrace-version=v1.10 \
             --arch x86_64 \
             --platform manylinux2014 \
             --output-dir ddtrace_pkgs \
@@ -94,15 +94,19 @@ jobs:
             --network test-inject \
             -p 18080:18080 \
             -e PYTHONPATH=/lib-injection \
+            -e DD_TRACE_AGENT_URL=http://testagent:8126 \
             -v $PWD/lib-injection:/lib-injection \
             ${{matrix.variant}}
           # Wait for the app to start
-          sleep 5
+          sleep 20
           docker logs ${{matrix.variant}}
       - name: Test the app
         run: |
           curl http://localhost:18080
           sleep 1  # wait for traces to be sent
-      - name: Check test agent
+      - name: Print traces
+        run: curl http://localhost:8126/test/traces
+      - name: Check test agent received a trace
         run: |
-          curl http://localhost:8126/test/traces
+          N=$(curl http://localhost:8126/test/traces | jq -r -e 'length')
+          [[ $N == "1" ]]

lib-injection/Dockerfile

@@ -6,10 +6,10 @@ ARG DDTRACE_PYTHON_VERSION
 RUN python3 -m pip install -U pip==23.0.1
 RUN python3 -m pip install packaging==23.0
 RUN mkdir -p pkgs
-ADD ./dl_megawheel.py .
+ADD ./dl_wheels.py .
 # Note that we only get Python >= 3.7. This is to keep the size of the image
 # as small as possible.
-RUN python3 dl_megawheel.py \
+RUN python3 dl_wheels.py \
         --python-version=3.11 \
         --python-version=3.10 \
         --python-version=3.9 \

lib-injection/README.md

@@ -12,9 +12,9 @@ It is responsible for providing the files necessary to run `ddtrace` in an
 arbitrary downstream application container. It also provides a script to copy
 the necessary files to a given directory.
 
-The `dl_megawheel.py` script provides a portable `ddtrace` package. It is
-responsible for downloading and merging the published wheels of `ddtrace` and
-its dependencies.
+The `dl_wheels.py` script provides a portable set of `ddtrace` wheels. It is
+responsible for downloading the published wheels of `ddtrace` and its
+dependencies.
 
 The Datadog Admission Controller injects the InitContainer with a new volume
 mount to the application deployment. The script to copy files out of the

lib-injection/dl_megawheel.py renamed to lib-injection/dl_wheels.py

@@ -13,15 +13,14 @@
 20.0.2.
 
 Usage:
-        ./dl_megawheel.py --help
+        ./dl_wheels.py --help
 
 
 The downloaded wheels can then be installed locally using:
         pip install --no-index --find-links <dir_of_downloaded_wheels> ddtrace
 """
 import argparse
 import itertools
-import os
 import subprocess
 import sys
 
@@ -105,12 +104,3 @@
 
     if not args.dry_run:
         subprocess.run(cmd, capture_output=not args.verbose, check=True)
-
-        # Unzip all the wheels into the output directory
-        wheel_files = [f for f in os.listdir(dl_dir) if f.endswith(".whl")]
-        for whl in wheel_files:
-            wheel_file = os.path.join(dl_dir, whl)
-            # -q for quieter output, else we get all of the files being unzipped.
-            subprocess.run(["unzip", "-q", "-o", wheel_file, "-d", dl_dir])
-            # Remove the wheel as it has been unpacked
-            os.remove(wheel_file)

lib-injection/sitecustomize.py

@@ -1,29 +1,50 @@
 """
-When included on the PYTHONPATH this module will initialize the PYTHONPATH to
-include the directory containing the ddtrace package and its dependencies. It
-then imports the ddtrace.bootstrap.sitecustomize module to automatically
-instrument the application.
+This module when included on the PYTHONPATH will install the ddtrace package
+from the locally available wheels that are included in the image.
 """
 import os
 import sys
 
 
-def _add_to_pythonpath(path):
-    # type: (str) -> None
-    """Adds a path to the start of PYTHONPATH."""
+def _configure_ddtrace():
+    # This import has the same effect as ddtrace-run for the current process.
+    import ddtrace.bootstrap.sitecustomize
+
+    bootstrap_dir = os.path.abspath(os.path.dirname(ddtrace.bootstrap.sitecustomize.__file__))
     prev_python_path = os.getenv("PYTHONPATH", "")
-    os.environ["PYTHONPATH"] = "%s%s%s" % (path, os.pathsep, prev_python_path)
-    sys.path.insert(0, path)
+    os.environ["PYTHONPATH"] = "%s%s%s" % (bootstrap_dir, os.path.pathsep, prev_python_path)
+
+    # Also insert the bootstrap dir in the path of the current python process.
+    sys.path.insert(0, bootstrap_dir)
+    print("datadog autoinstrumentation: successfully configured python package")
+
+
+# Avoid infinite loop when attempting to install ddtrace. This flag is set when
+# the subprocess is launched to perform the installation.
+if "DDTRACE_PYTHON_INSTALL_IN_PROGRESS" not in os.environ:
+    try:
+        import ddtrace  # noqa: F401
+
+    except ImportError:
+        import subprocess
 
+        print("datadog autoinstrumentation: installing python package")
 
-pkgs_path = os.path.join(os.path.dirname(__file__), "ddtrace_pkgs")
-bootstrap_dir = os.path.join(pkgs_path, "ddtrace", "bootstrap")
-_add_to_pythonpath(pkgs_path)
-_add_to_pythonpath(bootstrap_dir)
+        # Set the flag to avoid an infinite loop.
+        env = os.environ.copy()
+        env["DDTRACE_PYTHON_INSTALL_IN_PROGRESS"] = "true"
 
-try:
-    import ddtrace.bootstrap.sitecustomize  # noqa: F401
-except BaseException as e:
-    print("datadog autoinstrumentation: ddtrace failed to install:\n %s" % str(e))
-else:
-    print("datadog autoinstrumentation: ddtrace successfully installed")
+        # Execute the installation with the current interpreter
+        try:
+            pkgs_path = os.path.join(os.path.dirname(__file__), "ddtrace_pkgs")
+            subprocess.run(
+                [sys.executable, "-m", "pip", "install", "--find-links", pkgs_path, "ddtrace"], env=env, check=True
+            )
+        except BaseException as e:
+            print("datadog autoinstrumentation: failed to install python package %s" % str(e))
+        else:
+            print("datadog autoinstrumentation: successfully installed python package")
+            _configure_ddtrace()
+    else:
+        print("datadog autoinstrumentation: ddtrace already installed, skipping install")
+        _configure_ddtrace()

releasenotes/notes/lib-injection-wheels-fd40598c49a47c64.yaml

@@ -0,0 +1,6 @@
+---
+fixes:
+  - |
+    lib-injection: Switch installation to install from included wheels. Prior,
+      the wheels were merged together which caused conflicts between versions
+      of dependencies based on Python version.