Skip to content

Commit 6bc8627

Browse files
lgritzlgritz
authored
ci: For python wheel generation, use ccache (#4924)
Building the python wheels takes a long time! I really hate waiting for that when testing PRs, there must be a way to speed it up. * For the wheel workflow, add cache actions to save and restore the CCACHE_DIR. Hey, it's no easy feat to figure out what the path should be to that directory, especially on Linux where the wheels are built in a container, so the paths inside the container (where the wheel is built and the C++ compilation happens) don't match the paths outside the container (where the cache restore and save actions execute). * I had a heck of a time on Linux trying to get a pre-built ccache installed and had to resort to writing a bash script to build ccache itself from scratch, which is much more expensive than I'd like, but we'll have to come back to fix that separately. * Changed our "auto-build" utility build_dependency_with_cmake to print the amount of time it takes to build each dependency. * When auto-building, pass along CMAKE_CXX_COMPILER_LAUNCHER so that the dependenencies also for sure use ccache. * Use CMAKE_BUILD_PARALLEL_LEVEL on the wheel run to use all the cores and compile in parallel. (We did that on the regular CI but I think not for the wheel building.) * Fixes to the logic in our compiler.cmake where it tries to use ccache even if the magic CMAKE_CXX_COMPILER_LAUNCHER isn't set -- I have come to believe we were doing it wrong before, it was having no effect, and all along we only got ccache working on CI because we *also* set the env variable. * For CI, set CCACHE_COMPRESSION=1 to make the caches take less space against the precious limit of how much cache we can use total on GHA. So, the result of all this: **Previous times (typical), and first wheel run for any git branch** | platform | total | compile OIIO + deps | | ----------- | ----- | ------------------- | | Linux Intel | 10:35 | 500s | | Linux ARM | 7:20 | 294s | | Mac Intel | 20:18 | 1146s | | Mac ARM | 7:19 | 388s | | Windows | 14:00 | 759s | **With ccache active, 2nd or more wheel run for a git branch** | platform | total | compile OIIO + deps | | ----------- | ----- | ------------------- | | Linux Intel | 3:33 | 98s | | Linux ARM | 3:34 | 83s | | Mac Intel | 5:01 | 212s | | Mac ARM | 2:30 | 95s | | Windows | N/A | not using ccache | The "compile OIIO + deps" column is the isolated time to build OIIO and also any auto-building of dependencies from source that we do. But it does not include any other setup, such as 40-60s of container setup on Linux, 20-40s setting up Python on Mac, or -- ick! -- 45-60 seconds to build cmake itself from scratch. So this is considerably better, speeding up the full wheel workflow by 2-4x on all platforms but Windows. Remaining room for improvement (possibly in subsequent PRs, hopefully not necessarily by me): * Is there a way to use ccache on Windows? I'm not sure if there is, when using MSVS. * Find a way to install pre-built binaries on Linux rather than building ccache from scratch, which would save almost a whole minute per job. * Organize each platform to build all of its wheels (i.e. all of the python versions we are building for on that platform) in a single job rather than as completely independent jobs, allowing (a) the fixed per-job overhead such as container initialization, and installing python, and building of certain dependencies to happen ONCE per platform instead of separately for each wheel, and (b) the magic of ccache to speed up the builds *across* those wheels, since only a tiny amount of OIIO source codes depend on python at all. Signed-off-by: Larry Gritz <[email protected]>
1 parent 6f25722 commit 6bc8627

File tree

5 files changed

+269
-53
lines changed

5 files changed

+269
-53
lines changed

‎.github/workflows/wheel.yml‎

Lines changed: 153 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,12 @@ on:
3939
workflow_dispatch:
4040
# This allows manual triggering of the workflow from the web
4141

42+
# Allow subsequent pushes to the same PR or REF to cancel any previous jobs.
43+
concurrency:
44+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
45+
cancel-in-progress: true
46+
47+
4248
jobs:
4349
# Linux jobs run in Docker containers (manylinux), so the latest OS version
4450
# is OK. macOS and Windows jobs need to be locked to specific virtual
@@ -141,20 +147,50 @@ jobs:
141147
with:
142148
python-version: '3.9'
143149

150+
- name: ccache-restore
151+
id: ccache-restore
152+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
153+
with:
154+
path: ~/.ccache
155+
key: wheel-${{runner.os}}-${{matrix.manylinux}}-${{matrix.python}}
156+
restore-keys: wheel-${{runner.os}}-${{matrix.manylinux}}-${{matrix.python}}
157+
144158
- name: Build wheels
145159
# Note: the version of cibuildwheel should be kept in sync with src/python/stubs/CMakeLists.txt
146160
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
147161
env:
148162
# pass GITHUB_ACTIONS through to the build container so that custom
149163
# processes can tell they are running in CI.
150164
CIBW_ENVIRONMENT_PASS_LINUX: GITHUB_ACTIONS
165+
CIBW_BEFORE_ALL: "source src/build-scripts/build_ccache.bash && pwd && ext/dist/bin/ccache --max-size=200M && ext/dist/bin/ccache -sv && export CMAKE_C_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache CMAKE_CXX_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache"
166+
CIBW_BEFORE_TEST: "ext/dist/bin/ccache -s"
151167
CIBW_BUILD: ${{ matrix.python }}
152168
CIBW_ARCHS: ${{ matrix.arch }}
153169
CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux }}
170+
CIBW_ENVIRONMENT: >
171+
CCACHE_DIR=/host//home/runner/.ccache
172+
CCACHE_COMPRESSION=yes
173+
CCACHE_PREBUILT=0
174+
CMAKE_BUILD_PARALLEL_LEVEL=4
175+
CTEST_PARALLEL_LEVEL=4
176+
SKBUILD_CMAKE_ARGS="-DLINKSTATIC=1"
177+
SKBUILD_CMAKE_BUILD_TYPE="MinSizeRel"
178+
SKBUILD_BUILD_DIR=/project/build
179+
CXXFLAGS="-Wno-error=stringop-overflow -Wno-pragmas"
180+
WebP_BUILD_VERSION="1.5.0"
181+
# FIXME: Getting build problems when using WebP 1.6.0, so hold it back
182+
# CMAKE_GENERATOR = "Ninja"
183+
184+
- name: ccache-save
185+
id: ccache-save
186+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
187+
with:
188+
path: ~/.ccache
189+
key: wheel-${{runner.os}}-${{matrix.manylinux}}-${{matrix.python}}
154190

155191
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
156192
with:
157-
name: cibw-wheels-${{ matrix.python }}-${{ matrix.manylinux }}
193+
name: cibw-wheels-${{matrix.manylinux}}-${{ matrix.python }}-${{ matrix.manylinux }}
158194
path: |
159195
./wheelhouse/*.whl
160196
@@ -173,12 +209,12 @@ jobs:
173209
# ---------------------------------------------------------------------------
174210

175211
linux-arm:
176-
name: Build wheels on Linux ARM
177-
runs-on: ubuntu-24.04-arm
178-
if: |
179-
github.event_name != 'schedule' ||
180-
github.repository == 'AcademySoftwareFoundation/OpenImageIO'
181-
strategy:
212+
name: Build wheels on Linux ARM
213+
runs-on: ubuntu-24.04-arm
214+
if: |
215+
github.event_name != 'schedule' ||
216+
github.repository == 'AcademySoftwareFoundation/OpenImageIO'
217+
strategy:
182218
matrix:
183219
include:
184220
# -------------------------------------------------------------------
@@ -205,37 +241,66 @@ jobs:
205241
python: cp313-manylinux_aarch64
206242
arch: aarch64
207243

208-
steps:
209-
- name: Checkout repo
210-
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
244+
steps:
245+
- name: Checkout repo
246+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
211247

212-
- name: Install Python
213-
uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3 # v5.2.0
214-
with:
215-
python-version: '3.9'
216-
217-
- name: Build wheels
218-
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
219-
env:
220-
CIBW_BUILD: ${{ matrix.python }}
221-
CIBW_ARCHS: ${{ matrix.arch }}
222-
CIBW_MANYLINUX_AARCH64_IMAGE: ${{ matrix.manylinux }}
223-
224-
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
225-
with:
226-
name: cibw-wheels-${{ matrix.python }}-${{ matrix.manylinux }}
227-
path: |
228-
./wheelhouse/*.whl
248+
- name: Install Python
249+
uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3 # v5.2.0
250+
with:
251+
python-version: '3.9'
229252

230-
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
231-
with:
232-
name: stubs-${{ matrix.python }}-${{ matrix.manylinux }}
233-
path: |
234-
./wheelhouse/OpenImageIO/__init__.pyi
235-
# if stub validation fails we want to upload the stubs for users to review.
236-
# keep the python build in sync with the version specified in tool.cibuildwheel.overrides
237-
# section of pyproject.toml
238-
if: always() && contains(matrix.python, 'cp311-manylinux')
253+
- name: ccache-restore
254+
id: ccache-restore
255+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
256+
with:
257+
path: ~/.ccache
258+
key: wheel-${{runner.os}}-${{matrix.python}}
259+
restore-keys: wheel-${{runner.os}}-${{matrix.python}}
260+
261+
- name: Build wheels
262+
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
263+
env:
264+
CIBW_ENVIRONMENT_PASS_LINUX: GITHUB_ACTIONS
265+
CIBW_BEFORE_ALL: "source src/build-scripts/build_ccache.bash && pwd && /project/ext/dist/bin/ccache --max-size=200M && /project/ext/dist/bin/ccache -sv && export CMAKE_C_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache CMAKE_CXX_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache"
266+
CIBW_BEFORE_TEST: "ext/dist/bin/ccache -s"
267+
CIBW_BUILD: ${{ matrix.python }}
268+
CIBW_ARCHS: ${{ matrix.arch }}
269+
CIBW_MANYLINUX_AARCH64_IMAGE: ${{ matrix.manylinux }}
270+
CIBW_ENVIRONMENT: >
271+
CCACHE_DIR=/host//home/runner/.ccache
272+
CCACHE_COMPRESSION=yes
273+
CCACHE_PREBUILT=0
274+
CMAKE_BUILD_PARALLEL_LEVEL=6
275+
CTEST_PARALLEL_LEVEL=6
276+
SKBUILD_CMAKE_ARGS="-DLINKSTATIC=1"
277+
SKBUILD_CMAKE_BUILD_TYPE="MinSizeRel"
278+
SKBUILD_BUILD_DIR=/project/build
279+
CXXFLAGS="-Wno-error=stringop-overflow -Wno-pragmas"
280+
WebP_BUILD_VERSION="1.5.0"
281+
282+
- name: ccache-save
283+
id: ccache-save
284+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
285+
with:
286+
path: ~/.ccache
287+
key: wheel-${{runner.os}}-${{matrix.python}}
288+
289+
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
290+
with:
291+
name: cibw-wheels-${{ matrix.python }}-${{ matrix.manylinux }}
292+
path: |
293+
./wheelhouse/*.whl
294+
295+
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
296+
with:
297+
name: stubs-${{ matrix.python }}-${{ matrix.manylinux }}
298+
path: |
299+
./wheelhouse/OpenImageIO/__init__.pyi
300+
# if stub validation fails we want to upload the stubs for users to review.
301+
# keep the python build in sync with the version specified in tool.cibuildwheel.overrides
302+
# section of pyproject.toml
303+
if: always() && contains(matrix.python, 'cp311-manylinux')
239304

240305
# ---------------------------------------------------------------------------
241306
# macOS Wheels
@@ -278,6 +343,18 @@ jobs:
278343
with:
279344
python-version: '3.9'
280345

346+
- name: ccache-restore
347+
id: ccache-restore
348+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
349+
with:
350+
path: ~/.ccache
351+
key: wheel-${{runner.os}}-${{matrix.python}}
352+
restore-keys: wheel-${{runner.os}}-${{matrix.python}}
353+
354+
- name: Install build tools
355+
run: |
356+
brew install ninja ccache || true
357+
281358
- name: Remove brew OpenEXR/Imath
282359
run: |
283360
brew uninstall --ignore-dependencies openexr imath || true
@@ -292,6 +369,18 @@ jobs:
292369
# TODO: Re-enable HEIF when we provide a build recipe that does
293370
# not include GPL-licensed dynamic libraries.
294371
USE_Libheif: 'OFF'
372+
CMAKE_BUILD_PARALLEL_LEVEL: 6
373+
CTEST_PARALLEL_LEVEL: 6
374+
SKBUILD_BUILD_DIR: "/Users/runner/work/OpenImageIO/OpenImageIO/build"
375+
CCACHE_DIR: /Users/runner/.ccache
376+
CCACHE_COMPRESSION: yes
377+
378+
- name: ccache-save
379+
id: ccache-save
380+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
381+
with:
382+
path: ~/.ccache
383+
key: wheel-${{runner.os}}-${{matrix.python}}
295384

296385
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
297386
with:
@@ -340,13 +429,36 @@ jobs:
340429
with:
341430
python-version: '3.9'
342431

432+
- name: ccache-restore
433+
id: ccache-restore
434+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
435+
with:
436+
path: ~/.ccache
437+
key: wheel-${{runner.os}}-${{matrix.python}}
438+
restore-keys: wheel-${{runner.os}}-${{matrix.python}}
343439

440+
- name: Install build tools
441+
run: |
442+
brew install ninja ccache || true
443+
344444
- name: Build wheels
345445
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
346446
env:
347447
CIBW_BUILD: ${{ matrix.python }}
348448
CIBW_ARCHS: ${{ matrix.arch }}
349449
CMAKE_GENERATOR: "Unix Makefiles"
450+
CMAKE_BUILD_PARALLEL_LEVEL: 6
451+
CTEST_PARALLEL_LEVEL: 6
452+
SKBUILD_BUILD_DIR: "/Users/runner/work/OpenImageIO/OpenImageIO/build"
453+
CCACHE_DIR: /Users/runner/.ccache
454+
CCACHE_COMPRESSION: yes
455+
456+
- name: ccache-save
457+
id: ccache-save
458+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
459+
with:
460+
path: ~/.ccache
461+
key: wheel-${{runner.os}}-${{matrix.python}}
350462

351463
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
352464
with:
@@ -400,6 +512,11 @@ jobs:
400512
CIBW_BUILD: ${{ matrix.python }}
401513
CIBW_ARCHS: ${{ matrix.arch }}
402514
CMAKE_POLICY_VERSION_MINIMUM: 3.5
515+
CMAKE_BUILD_PARALLEL_LEVEL: 4
516+
CTEST_PARALLEL_LEVEL: 4
517+
SKBUILD_BUILD_DIR: "$HOME/OpenImageIO/OpenImageIO/build"
518+
CCACHE_DIR: ~/.ccache
519+
CCACHE_COMPRESSION: yes
403520

404521
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
405522
with:

‎pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
# See docs at https://cibuildwheel.pypa.io/en/stable/options
2+
# for description of all the options in this file.
3+
14
[project]
25
name = "OpenImageIO"
36
# The build backend ascertains the version from the CMakeLists.txt file.

‎src/build-scripts/build_ccache.bash‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env bash
2+
3+
# Utility script to download or build ccacheccache
4+
#
5+
# Copyright Contributors to the OpenImageIO project.
6+
# SPDX-License-Identifier: Apache-2.0
7+
# https://github.com/AcademySoftwareFoundation/OpenImageIO
8+
9+
# Exit the whole script if any command fails.
10+
set -ex
11+
12+
echo "Building ccache"
13+
uname
14+
ARCH=`uname -m`
15+
echo "HOME=$HOME"
16+
echo "PWD=$PWD"
17+
echo "ARCH=$ARCH"
18+
19+
CCACHE_PREBULT=${CCACHE_PREBULT:=1}
20+
21+
# Repo and branch/tag/commit of ccache to download if we don't have it yet
22+
CCACHE_REPO=${CCACHE_REPO:=https://github.com/ccache/ccache}
23+
CCACHE_VERSION=${CCACHE_VERSION:=4.12}
24+
CCACHE_TAG=${CCACHE_TAG:=v${CCACHE_VERSION}}
25+
LOCAL_DEPS_DIR=${LOCAL_DEPS_DIR:=${PWD}/ext}
26+
# Where to put ccache repo source (default to the ext area)
27+
CCACHE_SRC_DIR=${CCACHE_SRC_DIR:=${LOCAL_DEPS_DIR}/ccache}
28+
# Temp build area (default to a build/ subdir under source)
29+
CCACHE_BUILD_DIR=${CCACHE_BUILD_DIR:=${CCACHE_SRC_DIR}/build}
30+
# Install area for ccache (default to ext/dist)
31+
CCACHE_INSTALL_DIR=${CCACHE_INSTALL_DIR:=${PWD}/ext/dist}
32+
CCACHE_CONFIG_OPTS=${CCACHE_CONFIG_OPTS:=}
33+
34+
35+
# if [[ `uname` == "Linux" && `uname -m` == "x86_64" ]] ; then
36+
if [[ `uname` == "Linux" ]] ; then
37+
mkdir -p ${CCACHE_SRC_DIR}
38+
pushd ${CCACHE_SRC_DIR}
39+
40+
if [[ "$CCACHE_PREBUILT" != "0" ]] ; then
41+
#
42+
# Try to download -- had trouble with this on runners
43+
#
44+
CCACHE_DESCRIPTOR="ccache-${CCACHE_VERSION}-linux-x86_64"
45+
curl --location "${CCACHE_REPO}/releases/download/${CCACHE_TAG}/${CCACHE_DESCRIPTOR}.tar.xz" -o ccache.tar.xz
46+
tar xJvf ccache.tar.xz
47+
mkdir -p ${CCACHE_INSTALL_DIR}/bin
48+
cp ${CCACHE_SRC_DIR}/${CCACHE_DESCRIPTOR}/ccache ${CCACHE_INSTALL_DIR}/bin
49+
else
50+
# Clone ccache project from GitHub and build
51+
if [[ ! -e ${CCACHE_SRC_DIR}/.git ]] ; then
52+
echo "git clone ${CCACHE_REPO} ${CCACHE_SRC_DIR}"
53+
git clone ${CCACHE_REPO} ${CCACHE_SRC_DIR}
54+
fi
55+
56+
echo "git checkout ${CCACHE_TAG} --force"
57+
git checkout ${CCACHE_TAG} --force
58+
59+
if [[ -z $DEP_DOWNLOAD_ONLY ]]; then
60+
time cmake -S . -B ${CCACHE_BUILD_DIR} -DCMAKE_BUILD_TYPE=Release \
61+
-DCMAKE_INSTALL_PREFIX=${CCACHE_INSTALL_DIR} \
62+
-DENABLE_TESTING=OFF -DENABLE_DOCUMENTATION=OFF \
63+
${CCACHE_CONFIG_OPTS}
64+
time cmake --build ${CCACHE_BUILD_DIR} --config Release --target install
65+
fi
66+
fi
67+
68+
popd
69+
ls ${CCACHE_INSTALL_DIR}
70+
ls ${CCACHE_INSTALL_DIR}/bin
71+
echo "CCACHE_INSTALL_DIR=$CCACHE_INSTALL_DIR"
72+
echo "CCACHE_DIR=$CCACHE_DIR"
73+
mkdir -p $CCACHE_DIR
74+
ls $CCACHE_DIR || true
75+
export PATH=${CCACHE_INSTALL_DIR}/bin:$PATH
76+
${CCACHE_INSTALL_DIR}/bin/ccache -sv
77+
fi

0 commit comments

Comments
 (0)