fix(lib-injection): wait and publish release tag of image [backport #4931, #5039 to 1.8] (#5056)

## [fix(lib-injection): wait and publish release tag of
image](a4fdbad)

Currently, only dev images for commits are pushed to the container
registry. This means that images like `v1.7.0` that get published are
really just aliases of the corresponding SHA image. These images use

```
pip install git+....@sha
```
to install the image.

For example:

```
~/d/h/c/datadog ❯❯❯ docker run -it gcr.io/datadoghq/dd-lib-python-init:v1.7.0 cat /datadog-init/sitecustomize.py | head -n 12
"""
This module when included on the PYTHONPATH will install the ddtrace package from pypi
for the Python runtime being used.
"""
$ docker run -it gcr.io/datadoghq/dd-lib-python-init:v1.7.0 cat /datadog-init/sitecustomize.py | head -n 12
...
version = "git+https://github.com/Datadog/dd-trace-py@ced65df45a9b592dd36af12c37c047c1486af38a"
```

While this isn't _wrong_ it is slow for auto instrumented applications
to start since the install is from source. It also means all
installation dependencies are required and the install could fail.

The solution is to run the publishing Github workflow on release after
the pypi_upload job. This should ensure that we build an image that uses
the release tag and subsequently pip installs with that version rather
than from source.

The Gitlab job responsible for publishing the image to the other
container registries is updated to use the release tag image rather than
the SHA. There is and was a race between when the Github registry image
is published and when the Gitlab job runs. To mitigate this the Gitlab
job is delayed by 1 day to give time for the image to be pushed to the
Github registry.

## [fix(ci/lib-injection): add missing runs_on
value](d10ec21)

The `runs_on` field was missing which caused the 1.7.4 release workflow
to fail.  To avoid this problem, or problems like this from happening
again, the shared logic is pulled into a reusable workflow[0]. Now the
build and push logic is shared between the test and publish jobs which
will give us a better idea if the publish one works.


## Checklist

- [ ] Change(s) are motivated and described in the PR description.
- [ ] Testing strategy is described if automated tests are not included
in the PR.
- [ ] Risk is outlined (performance impact, potential for breakage,
maintainability, etc).
- [ ] Change is maintainable (easy to change, telemetry, documentation).
- [ ] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines)
are followed.
- [ ] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/)).


## Reviewer Checklist

- [ ] Title is accurate.
- [ ] No unnecessary changes are introduced.
- [ ] Description motivates each change.
- [ ] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes unless absolutely necessary.
- [ ] Testing strategy adequately addresses listed risk(s).
- [ ] Change is maintainable (easy to change, telemetry, documentation).
- [ ] Release note makes sense to a user of the library.

---------

Co-authored-by: Yun Kim <[email protected]>

Author: Kyle-Verhoog

.github/workflows/build-and-publish-image.yml

@@ -0,0 +1,41 @@
+name: Build and publish image
+
+on:
+  workflow_call:
+    inputs:
+      tags:
+        required: true
+        type: string
+      platforms:
+        required: true
+        type: string
+      build-args:
+        required: true
+        type: string
+      context:
+        required: true
+        type: string
+    secrets:
+      token:
+        required: true
+
+jobs:
+  build_push:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up QEMU
+      uses: docker/setup-qemu-action@v2
+    - name: Set up Docker Buildx
+      id: buildx
+      uses: docker/setup-buildx-action@v2
+    - name: Login to Docker
+      run: docker login -u publisher -p ${{ secrets.token }} ghcr.io
+    - name: Docker Build
+      uses: docker/build-push-action@v3
+      with:
+        push: true
+        tags: ${{ inputs.tags }}
+        platforms: ${{ inputs.platforms }}
+        build-args: ${{ inputs.build-args }}
+        context: ${{ inputs.context }}

.github/workflows/build_deploy.yml

@@ -211,3 +211,16 @@ jobs:
           # due to a duplicate wheel being present which will ensure that the rest
           # of the wheels will be uploaded if some are uploaded manually.
           skip_existing: true
+
+  build-and-publish-init-image:
+    needs: [upload_pypi]
+    # We have to wait for the PyPI job since the image that we publish depends on installing
+    # the package from PyPI.
+    uses: ./.github/workflows/build-and-publish-image.yml
+    with:
+      tags: ghcr.io/datadog/dd-trace-py/dd-lib-python-init:${{ github.ref_name }}
+      platforms: 'linux/amd64,linux/arm64/v8'
+      build-args: "DDTRACE_PYTHON_VERSION=${{ github.ref_name }}"
+      context: ./lib-injection
+    secrets:
+      token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/lib-injection.yml

@@ -1,53 +1,22 @@
 name: "Library Injection"
 on:
+  # Build each branch for testing
   push:
 
 jobs:
-  build-and-publish-init-image:
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v3
-
-    - name: Set up QEMU
-      uses: docker/setup-qemu-action@v2
-
-    - uses: actions/setup-python@v4
-      with:
-        python-version: '3.10'
-    # required for getting the version
-    - run: pip install setuptools-scm
-    - name: Get library version
-      id: get_version
-      run: |
-        DDTRACE_PYTHON_VERSION=$(scripts/get_install_version.py)
-        echo "library_version=$DDTRACE_PYTHON_VERSION" >> $GITHUB_OUTPUT
-
-    - name: Set up Docker Buildx
-      id: buildx
-      uses: docker/setup-buildx-action@v2
-   
-    - name: lib-injection-tags
-      id: lib-injection-tags
-      uses: DataDog/system-tests/lib-injection/docker-tags@main
-      with:
-        init-image-name: 'dd-lib-python-init'
-        main-branch-name: '1.x'
-
-    - name: Login to Docker
-      run: docker login -u publisher -p ${{ secrets.GITHUB_TOKEN }} ghcr.io
-
-    - name: Docker Build
-      uses: docker/build-push-action@v3
-      with:
-        push: true
-        tags: ${{ steps.lib-injection-tags.outputs.tag-names }}
-        platforms: 'linux/amd64,linux/arm64/v8'
-        build-args: "DDTRACE_PYTHON_VERSION=${{ steps.get_version.outputs.library_version }}"
-        context: ./lib-injection
+  build-and-publish-test-image:
+    uses: ./.github/workflows/build-and-publish-image.yml
+    with:
+      tags: 'ghcr.io/datadog/dd-trace-py/dd-lib-python-init:${{ github.sha }}'
+      platforms: 'linux/amd64,linux/arm64/v8'
+      build-args: 'DDTRACE_PYTHON_VERSION=git+https://github.com/Datadog/dd-trace-py@${{ github.sha }}'
+      context: ./lib-injection
+    secrets:
+      token: ${{ secrets.GITHUB_TOKEN }}
 
   test:
     needs:
-      - build-and-publish-init-image
+      - build-and-publish-test-image
     runs-on: ubuntu-latest
     permissions:
       contents: read

.gitlab-ci.yml

@@ -46,16 +46,24 @@ deploy_to_docker_registries:
   rules:
     - if: '$POPULATE_CACHE'
       when: never
+    # Wait 1 day to trigger the downstream job.
+    # This is a work-around since there isn't a way to trigger
+    # Gitlab from the Github workflow (build_deploy.yml:upload_pypi).
+    #
+    # The caveat here is that if there is a failure to build to PyPI
+    # and it isn't fixed in a day then this job will fail and images
+    # will not be published.
     - if: '$CI_COMMIT_TAG =~ /^v.*/'
-      when: on_success
+      when: delayed
+      start_in: 1 day
     - when: manual
       allow_failure: true
   trigger:
     project: DataDog/public-images
     branch: main
     strategy: depend
   variables:
-    IMG_SOURCES: ghcr.io/datadog/dd-trace-py/dd-lib-python-init:$CI_COMMIT_SHA
+    IMG_SOURCES: ghcr.io/datadog/dd-trace-py/dd-lib-python-init:$CI_COMMIT_TAG
     IMG_DESTINATIONS: dd-lib-python-init:$CI_COMMIT_TAG
     IMG_SIGNING: "false"
 
@@ -64,15 +72,17 @@ deploy_latest_tag_to_docker_registries:
   rules:
     - if: '$POPULATE_CACHE'
       when: never
+    # See above note in the `deploy_to_docker_registries` job.
     - if: '$CI_COMMIT_TAG =~ /^v.*/'
-      when: on_success
+      when: delayed
+      start_in: 1 day
     - when: manual
       allow_failure: true
   trigger:
     project: DataDog/public-images
     branch: main
     strategy: depend
   variables:
-    IMG_SOURCES: ghcr.io/datadog/dd-trace-py/dd-lib-python-init:$CI_COMMIT_SHA
+    IMG_SOURCES: ghcr.io/datadog/dd-trace-py/dd-lib-python-init:$CI_COMMIT_TAG
     IMG_DESTINATIONS: dd-lib-python-init:latest
     IMG_SIGNING: "false"

releasenotes/notes/lib-injection-0041b50a263e1c19.yaml

@@ -0,0 +1,7 @@
+---
+fixes:
+  - |
+    lib-injection: Use package versions published to PyPI to install the
+      library. Formerly the published image was installing the package from
+      source using the tagged commit SHA which resulted in slow and potentially
+      failing installs.