Skip to content

Commit ecf3ded

Browse files
brettlangdonbrettlangdon
authored
fix(ci): break wheel building into multiple jobs (#6391)
In order to combat the ever increasing build/release time we are breaking up the build/publish job into multiple jobs so the work can be better parallelized. Before this change the build process would take at least 4 hours (sometimes more, or timing out at 6 hours), after this change it completes in about 1.5 hours. The Python 2.7 + 3.5 build job is kept as-is with no changes. For Python 3.6+ we have created a reusable workflow and then creates one build step per-python version that we want to support (3.6, 3.7, 3.8, 3.9, 3.10, 3.11). This change also changes the `workflow_dispatch` trigger to not publish to PyPI so it can be used to validate the build process without releasing. This change was validated by running manually, and then validating that the number of release files and the release tags created match what was published on PyPI for ddtrace==1.16.1. ## Checklist - [x] Change(s) are motivated and described in the PR description. - [x] Testing strategy is described if automated tests are not included in the PR. - [x] Risk is outlined (performance impact, potential for breakage, maintainability, etc). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) are followed. If no release note is required, add label `changelog/no-changelog`. - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)). - [x] Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Title is accurate. - [x] No unnecessary changes are introduced. - [x] Description motivates each change. - [x] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Testing strategy adequately addresses listed risk(s). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] Release note makes sense to a user of the library. - [x] Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment. - [x] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
1 parent efdf170 commit ecf3ded

File tree

2 files changed

+99
-61
lines changed

2 files changed

+99
-61
lines changed

‎.github/workflows/build_deploy.yml‎

Lines changed: 36 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,7 @@ on:
1818
types:
1919
- published
2020
workflow_dispatch:
21-
inputs:
22-
expectedVersion:
23-
description: 'Expected version string'
24-
required: true
21+
# Allow manually triggering, but do NOT upload the result
2522
schedule:
2623
# Nightly builds after weekdays
2724
- cron: 0 2 * * 2-6
@@ -83,56 +80,35 @@ jobs:
8380
with:
8481
path: ./wheelhouse/*.whl
8582

86-
build_wheels_py3:
87-
name: Build and test wheels on ${{ matrix.os }} (${{ matrix.archs }})
88-
runs-on: ${{ matrix.os }}
89-
strategy:
90-
matrix:
91-
include:
92-
- os: ubuntu-latest
93-
archs: x86_64 i686
94-
- os: ubuntu-latest
95-
archs: aarch64
96-
- os: windows-latest
97-
archs: AMD64 x86
98-
- os: macos-latest
99-
archs: x86_64 universal2
100-
steps:
101-
- uses: actions/checkout@v3
102-
# Include all history and tags
103-
with:
104-
fetch-depth: 0
83+
build_wheels_py36:
84+
uses: ./.github/workflows/build_python_3.yml
85+
with:
86+
cibw_build: 'cp36*'
10587

106-
- uses: actions/setup-python@v4
107-
name: Install Python
108-
with:
109-
python-version: '3.8'
88+
build_wheels_py37:
89+
uses: ./.github/workflows/build_python_3.yml
90+
with:
91+
cibw_build: 'cp37*'
11092

111-
- name: Set up QEMU
112-
if: runner.os == 'Linux'
113-
uses: docker/setup-qemu-action@v2
114-
with:
115-
platforms: all
93+
build_wheels_py38:
94+
uses: ./.github/workflows/build_python_3.yml
95+
with:
96+
cibw_build: 'cp38*'
11697

117-
- name: Build wheels python 3.6 and above
118-
uses: pypa/[email protected]
119-
env:
120-
# configure cibuildwheel to build native archs ('auto'), and some
121-
# emulated ones
122-
CIBW_ARCHS: ${{ matrix.archs }}
123-
CIBW_BUILD: cp3*
124-
# Run a smoke test on every supported platform
125-
CIBW_TEST_COMMAND: python {project}/tests/smoke_test.py
126-
# Testing arm on MacOS is currently not supported by Github
127-
CIBW_TEST_SKIP: "*-macosx_universal2:arm64"
128-
# Workaround for Macos 11.0 versioning issue, a.k.a.
129-
# `platform.mac_ver()` reports incorrect MacOS version at 11.0
130-
# See: https://stackoverflow.com/a/65402241
131-
CIBW_ENVIRONMENT_MACOS: SYSTEM_VERSION_COMPAT=0
98+
build_wheels_py39:
99+
uses: ./.github/workflows/build_python_3.yml
100+
with:
101+
cibw_build: 'cp39*'
132102

133-
- uses: actions/upload-artifact@v3
134-
with:
135-
path: ./wheelhouse/*.whl
103+
build_wheels_py310:
104+
uses: ./.github/workflows/build_python_3.yml
105+
with:
106+
cibw_build: 'cp310*'
107+
108+
build_wheels_py311:
109+
uses: ./.github/workflows/build_python_3.yml
110+
with:
111+
cibw_build: 'cp311*'
136112

137113
build_sdist:
138114
name: Build source distribution
@@ -187,24 +163,23 @@ jobs:
187163
working-directory: /
188164

189165
upload_pypi:
190-
needs: [build_wheels_py27_35, build_wheels_py3, test_alpine_sdist]
166+
needs:
167+
- build_wheels_py27_35
168+
- build_wheels_py36
169+
- build_wheels_py37
170+
- build_wheels_py38
171+
- build_wheels_py39
172+
- build_wheels_py310
173+
- build_wheels_py311
174+
- test_alpine_sdist
191175
runs-on: ubuntu-latest
192-
if: (github.event_name == 'release' && github.event.action == 'published') || (github.event_name == 'workflow_dispatch')
176+
if: (github.event_name == 'release' && github.event.action == 'published')
193177
steps:
194178
- uses: actions/download-artifact@v3
195179
with:
196180
name: artifact
197181
path: dist
198182

199-
- uses: actions/checkout@v3
200-
if: github.event_name == 'workflow_dispatch'
201-
# Include all history and tags
202-
with:
203-
fetch-depth: 0
204-
- name: Validate deploy version
205-
if: github.event_name == 'workflow_dispatch'
206-
run: |
207-
./scripts/validate-version "${{ github.event.inputs.expectedVersion }}"
208183
- uses: pypa/gh-action-pypi-publish@master
209184
with:
210185
user: __token__

‎.github/workflows/build_python_3.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: Build Python 3.6+
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
cibw_build:
7+
required: true
8+
type: string
9+
cibw_skip:
10+
required: false
11+
type: string
12+
13+
jobs:
14+
build:
15+
runs-on: ${{ matrix.os }}
16+
strategy:
17+
matrix:
18+
include:
19+
- os: ubuntu-latest
20+
archs: x86_64 i686
21+
- os: ubuntu-latest
22+
archs: aarch64
23+
- os: windows-latest
24+
archs: AMD64 x86
25+
- os: macos-latest
26+
archs: x86_64 universal2
27+
steps:
28+
- uses: actions/checkout@v3
29+
# Include all history and tags
30+
with:
31+
fetch-depth: 0
32+
33+
- uses: actions/setup-python@v4
34+
name: Install Python
35+
with:
36+
python-version: '3.8'
37+
38+
- name: Set up QEMU
39+
if: runner.os == 'Linux'
40+
uses: docker/setup-qemu-action@v2
41+
with:
42+
platforms: all
43+
44+
- name: Build wheels python 3.6 and above
45+
uses: pypa/[email protected]
46+
env:
47+
# configure cibuildwheel to build native archs ('auto'), and some
48+
# emulated ones
49+
CIBW_ARCHS: ${{ matrix.archs }}
50+
CIBW_BUILD: ${{ inputs.cibw_build }}
51+
CIBW_SKIp: ${{ inputs.cibw_skip }}
52+
# Run a smoke test on every supported platform
53+
CIBW_TEST_COMMAND: python {project}/tests/smoke_test.py
54+
# Testing arm on MacOS is currently not supported by Github
55+
CIBW_TEST_SKIP: "*-macosx_universal2:arm64"
56+
# Workaround for Macos 11.0 versioning issue, a.k.a.
57+
# `platform.mac_ver()` reports incorrect MacOS version at 11.0
58+
# See: https://stackoverflow.com/a/65402241
59+
CIBW_ENVIRONMENT_MACOS: SYSTEM_VERSION_COMPAT=0
60+
61+
- uses: actions/upload-artifact@v3
62+
with:
63+
path: ./wheelhouse/*.whl

0 commit comments

Comments
 (0)