Skip to content

Commit 93d0725

Browse files
lgritzlgritz
committed
ci: For python wheel generation, use ccache (#4924)
Building the python wheels takes a long time! I really hate waiting for that when testing PRs, there must be a way to speed it up. * For the wheel workflow, add cache actions to save and restore the CCACHE_DIR. Hey, it's no easy feat to figure out what the path should be to that directory, especially on Linux where the wheels are built in a container, so the paths inside the container (where the wheel is built and the C++ compilation happens) don't match the paths outside the container (where the cache restore and save actions execute). * I had a heck of a time on Linux trying to get a pre-built ccache installed and had to resort to writing a bash script to build ccache itself from scratch, which is much more expensive than I'd like, but we'll have to come back to fix that separately. * Changed our "auto-build" utility build_dependency_with_cmake to print the amount of time it takes to build each dependency. * When auto-building, pass along CMAKE_CXX_COMPILER_LAUNCHER so that the dependenencies also for sure use ccache. * Use CMAKE_BUILD_PARALLEL_LEVEL on the wheel run to use all the cores and compile in parallel. (We did that on the regular CI but I think not for the wheel building.) * Fixes to the logic in our compiler.cmake where it tries to use ccache even if the magic CMAKE_CXX_COMPILER_LAUNCHER isn't set -- I have come to believe we were doing it wrong before, it was having no effect, and all along we only got ccache working on CI because we *also* set the env variable. * For CI, set CCACHE_COMPRESSION=1 to make the caches take less space against the precious limit of how much cache we can use total on GHA. So, the result of all this: **Previous times (typical), and first wheel run for any git branch** | platform | total | compile OIIO + deps | | ----------- | ----- | ------------------- | | Linux Intel | 10:35 | 500s | | Linux ARM | 7:20 | 294s | | Mac Intel | 20:18 | 1146s | | Mac ARM | 7:19 | 388s | | Windows | 14:00 | 759s | **With ccache active, 2nd or more wheel run for a git branch** | platform | total | compile OIIO + deps | | ----------- | ----- | ------------------- | | Linux Intel | 3:33 | 98s | | Linux ARM | 3:34 | 83s | | Mac Intel | 5:01 | 212s | | Mac ARM | 2:30 | 95s | | Windows | N/A | not using ccache | The "compile OIIO + deps" column is the isolated time to build OIIO and also any auto-building of dependencies from source that we do. But it does not include any other setup, such as 40-60s of container setup on Linux, 20-40s setting up Python on Mac, or -- ick! -- 45-60 seconds to build cmake itself from scratch. So this is considerably better, speeding up the full wheel workflow by 2-4x on all platforms but Windows. Remaining room for improvement (possibly in subsequent PRs, hopefully not necessarily by me): * Is there a way to use ccache on Windows? I'm not sure if there is, when using MSVS. * Find a way to install pre-built binaries on Linux rather than building ccache from scratch, which would save almost a whole minute per job. * Organize each platform to build all of its wheels (i.e. all of the python versions we are building for on that platform) in a single job rather than as completely independent jobs, allowing (a) the fixed per-job overhead such as container initialization, and installing python, and building of certain dependencies to happen ONCE per platform instead of separately for each wheel, and (b) the magic of ccache to speed up the builds *across* those wheels, since only a tiny amount of OIIO source codes depend on python at all. Signed-off-by: Larry Gritz <[email protected]>
1 parent 8879c19 commit 93d0725

File tree

5 files changed

+270
-50
lines changed

5 files changed

+270
-50
lines changed

‎.github/workflows/wheel.yml‎

Lines changed: 153 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,12 @@ on:
3939
workflow_dispatch:
4040
# This allows manual triggering of the workflow from the web
4141

42+
# Allow subsequent pushes to the same PR or REF to cancel any previous jobs.
43+
concurrency:
44+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
45+
cancel-in-progress: true
46+
47+
4248
jobs:
4349
# Linux jobs run in Docker containers (manylinux), so the latest OS version
4450
# is OK. macOS and Windows jobs need to be locked to specific virtual
@@ -149,20 +155,50 @@ jobs:
149155
with:
150156
python-version: '3.9'
151157

158+
- name: ccache-restore
159+
id: ccache-restore
160+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
161+
with:
162+
path: ~/.ccache
163+
key: wheel-${{runner.os}}-${{matrix.manylinux}}-${{matrix.python}}
164+
restore-keys: wheel-${{runner.os}}-${{matrix.manylinux}}-${{matrix.python}}
165+
152166
- name: Build wheels
153167
# Note: the version of cibuildwheel should be kept in sync with src/python/stubs/CMakeLists.txt
154168
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
155169
env:
156170
# pass GITHUB_ACTIONS through to the build container so that custom
157171
# processes can tell they are running in CI.
158172
CIBW_ENVIRONMENT_PASS_LINUX: GITHUB_ACTIONS
173+
CIBW_BEFORE_ALL: "source src/build-scripts/build_ccache.bash && pwd && ext/dist/bin/ccache --max-size=200M && ext/dist/bin/ccache -sv && export CMAKE_C_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache CMAKE_CXX_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache"
174+
CIBW_BEFORE_TEST: "ext/dist/bin/ccache -s"
159175
CIBW_BUILD: ${{ matrix.python }}
160176
CIBW_ARCHS: ${{ matrix.arch }}
161177
CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.manylinux }}
178+
CIBW_ENVIRONMENT: >
179+
CCACHE_DIR=/host//home/runner/.ccache
180+
CCACHE_COMPRESSION=yes
181+
CCACHE_PREBUILT=0
182+
CMAKE_BUILD_PARALLEL_LEVEL=4
183+
CTEST_PARALLEL_LEVEL=4
184+
SKBUILD_CMAKE_ARGS="-DLINKSTATIC=1"
185+
SKBUILD_CMAKE_BUILD_TYPE="MinSizeRel"
186+
SKBUILD_BUILD_DIR=/project/build
187+
CXXFLAGS="-Wno-error=stringop-overflow -Wno-pragmas"
188+
WebP_BUILD_VERSION="1.5.0"
189+
# FIXME: Getting build problems when using WebP 1.6.0, so hold it back
190+
# CMAKE_GENERATOR = "Ninja"
191+
192+
- name: ccache-save
193+
id: ccache-save
194+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
195+
with:
196+
path: ~/.ccache
197+
key: wheel-${{runner.os}}-${{matrix.manylinux}}-${{matrix.python}}
162198

163199
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
164200
with:
165-
name: cibw-wheels-${{ matrix.python }}-${{ matrix.manylinux }}
201+
name: cibw-wheels-${{matrix.manylinux}}-${{ matrix.python }}-${{ matrix.manylinux }}
166202
path: |
167203
./wheelhouse/*.whl
168204
@@ -181,12 +217,12 @@ jobs:
181217
# ---------------------------------------------------------------------------
182218

183219
linux-arm:
184-
name: Build wheels on Linux ARM
185-
runs-on: ubuntu-24.04-arm
186-
if: |
187-
github.event_name != 'schedule' ||
188-
github.repository == 'AcademySoftwareFoundation/OpenImageIO'
189-
strategy:
220+
name: Build wheels on Linux ARM
221+
runs-on: ubuntu-24.04-arm
222+
if: |
223+
github.event_name != 'schedule' ||
224+
github.repository == 'AcademySoftwareFoundation/OpenImageIO'
225+
strategy:
190226
matrix:
191227
include:
192228
# -------------------------------------------------------------------
@@ -217,37 +253,66 @@ jobs:
217253
python: cp313-manylinux_aarch64
218254
arch: aarch64
219255

220-
steps:
221-
- name: Checkout repo
222-
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
256+
steps:
257+
- name: Checkout repo
258+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
223259

224-
- name: Install Python
225-
uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3 # v5.2.0
226-
with:
227-
python-version: '3.9'
228-
229-
- name: Build wheels
230-
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
231-
env:
232-
CIBW_BUILD: ${{ matrix.python }}
233-
CIBW_ARCHS: ${{ matrix.arch }}
234-
CIBW_MANYLINUX_AARCH64_IMAGE: ${{ matrix.manylinux }}
235-
236-
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
237-
with:
238-
name: cibw-wheels-${{ matrix.python }}-${{ matrix.manylinux }}
239-
path: |
240-
./wheelhouse/*.whl
260+
- name: Install Python
261+
uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3 # v5.2.0
262+
with:
263+
python-version: '3.9'
241264

242-
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
243-
with:
244-
name: stubs-${{ matrix.python }}-${{ matrix.manylinux }}
245-
path: |
246-
./wheelhouse/OpenImageIO/__init__.pyi
247-
# if stub validation fails we want to upload the stubs for users to review.
248-
# keep the python build in sync with the version specified in tool.cibuildwheel.overrides
249-
# section of pyproject.toml
250-
if: always() && contains(matrix.python, 'cp311-manylinux')
265+
- name: ccache-restore
266+
id: ccache-restore
267+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
268+
with:
269+
path: ~/.ccache
270+
key: wheel-${{runner.os}}-${{matrix.python}}
271+
restore-keys: wheel-${{runner.os}}-${{matrix.python}}
272+
273+
- name: Build wheels
274+
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
275+
env:
276+
CIBW_ENVIRONMENT_PASS_LINUX: GITHUB_ACTIONS
277+
CIBW_BEFORE_ALL: "source src/build-scripts/build_ccache.bash && pwd && /project/ext/dist/bin/ccache --max-size=200M && /project/ext/dist/bin/ccache -sv && export CMAKE_C_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache CMAKE_CXX_COMPILER_LAUNCHER=/project/ext/dist/bin/ccache"
278+
CIBW_BEFORE_TEST: "ext/dist/bin/ccache -s"
279+
CIBW_BUILD: ${{ matrix.python }}
280+
CIBW_ARCHS: ${{ matrix.arch }}
281+
CIBW_MANYLINUX_AARCH64_IMAGE: ${{ matrix.manylinux }}
282+
CIBW_ENVIRONMENT: >
283+
CCACHE_DIR=/host//home/runner/.ccache
284+
CCACHE_COMPRESSION=yes
285+
CCACHE_PREBUILT=0
286+
CMAKE_BUILD_PARALLEL_LEVEL=6
287+
CTEST_PARALLEL_LEVEL=6
288+
SKBUILD_CMAKE_ARGS="-DLINKSTATIC=1"
289+
SKBUILD_CMAKE_BUILD_TYPE="MinSizeRel"
290+
SKBUILD_BUILD_DIR=/project/build
291+
CXXFLAGS="-Wno-error=stringop-overflow -Wno-pragmas"
292+
WebP_BUILD_VERSION="1.5.0"
293+
294+
- name: ccache-save
295+
id: ccache-save
296+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
297+
with:
298+
path: ~/.ccache
299+
key: wheel-${{runner.os}}-${{matrix.python}}
300+
301+
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
302+
with:
303+
name: cibw-wheels-${{ matrix.python }}-${{ matrix.manylinux }}
304+
path: |
305+
./wheelhouse/*.whl
306+
307+
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
308+
with:
309+
name: stubs-${{ matrix.python }}-${{ matrix.manylinux }}
310+
path: |
311+
./wheelhouse/OpenImageIO/__init__.pyi
312+
# if stub validation fails we want to upload the stubs for users to review.
313+
# keep the python build in sync with the version specified in tool.cibuildwheel.overrides
314+
# section of pyproject.toml
315+
if: always() && contains(matrix.python, 'cp311-manylinux')
251316

252317
# ---------------------------------------------------------------------------
253318
# macOS Wheels
@@ -290,6 +355,18 @@ jobs:
290355
with:
291356
python-version: '3.9'
292357

358+
- name: ccache-restore
359+
id: ccache-restore
360+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
361+
with:
362+
path: ~/.ccache
363+
key: wheel-${{runner.os}}-${{matrix.python}}
364+
restore-keys: wheel-${{runner.os}}-${{matrix.python}}
365+
366+
- name: Install build tools
367+
run: |
368+
brew install ninja ccache || true
369+
293370
- name: Remove brew OpenEXR/Imath
294371
run: |
295372
brew uninstall --ignore-dependencies openexr imath || true
@@ -304,6 +381,18 @@ jobs:
304381
# TODO: Re-enable HEIF when we provide a build recipe that does
305382
# not include GPL-licensed dynamic libraries.
306383
USE_Libheif: 'OFF'
384+
CMAKE_BUILD_PARALLEL_LEVEL: 6
385+
CTEST_PARALLEL_LEVEL: 6
386+
SKBUILD_BUILD_DIR: "/Users/runner/work/OpenImageIO/OpenImageIO/build"
387+
CCACHE_DIR: /Users/runner/.ccache
388+
CCACHE_COMPRESSION: yes
389+
390+
- name: ccache-save
391+
id: ccache-save
392+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
393+
with:
394+
path: ~/.ccache
395+
key: wheel-${{runner.os}}-${{matrix.python}}
307396

308397
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
309398
with:
@@ -355,13 +444,36 @@ jobs:
355444
with:
356445
python-version: '3.8'
357446

447+
- name: ccache-restore
448+
id: ccache-restore
449+
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
450+
with:
451+
path: ~/.ccache
452+
key: wheel-${{runner.os}}-${{matrix.python}}
453+
restore-keys: wheel-${{runner.os}}-${{matrix.python}}
358454

455+
- name: Install build tools
456+
run: |
457+
brew install ninja ccache || true
458+
359459
- name: Build wheels
360460
uses: pypa/cibuildwheel@d4a2945fcc8d13f20a1b99d461b8e844d5fc6e23 # v2.21.1
361461
env:
362462
CIBW_BUILD: ${{ matrix.python }}
363463
CIBW_ARCHS: ${{ matrix.arch }}
364464
CMAKE_GENERATOR: "Unix Makefiles"
465+
CMAKE_BUILD_PARALLEL_LEVEL: 6
466+
CTEST_PARALLEL_LEVEL: 6
467+
SKBUILD_BUILD_DIR: "/Users/runner/work/OpenImageIO/OpenImageIO/build"
468+
CCACHE_DIR: /Users/runner/.ccache
469+
CCACHE_COMPRESSION: yes
470+
471+
- name: ccache-save
472+
id: ccache-save
473+
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
474+
with:
475+
path: ~/.ccache
476+
key: wheel-${{runner.os}}-${{matrix.python}}
365477

366478
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
367479
with:
@@ -418,6 +530,11 @@ jobs:
418530
CIBW_BUILD: ${{ matrix.python }}
419531
CIBW_ARCHS: ${{ matrix.arch }}
420532
CMAKE_POLICY_VERSION_MINIMUM: 3.5
533+
CMAKE_BUILD_PARALLEL_LEVEL: 4
534+
CTEST_PARALLEL_LEVEL: 4
535+
SKBUILD_BUILD_DIR: "$HOME/OpenImageIO/OpenImageIO/build"
536+
CCACHE_DIR: ~/.ccache
537+
CCACHE_COMPRESSION: yes
421538

422539
- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
423540
with:

‎pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
# See docs at https://cibuildwheel.pypa.io/en/stable/options
2+
# for description of all the options in this file.
3+
14
[project]
25
name = "OpenImageIO"
36
# The build backend ascertains the version from the CMakeLists.txt file.

‎src/build-scripts/build_ccache.bash‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env bash
2+
3+
# Utility script to download or build ccacheccache
4+
#
5+
# Copyright Contributors to the OpenImageIO project.
6+
# SPDX-License-Identifier: Apache-2.0
7+
# https://github.com/AcademySoftwareFoundation/OpenImageIO
8+
9+
# Exit the whole script if any command fails.
10+
set -ex
11+
12+
echo "Building ccache"
13+
uname
14+
ARCH=`uname -m`
15+
echo "HOME=$HOME"
16+
echo "PWD=$PWD"
17+
echo "ARCH=$ARCH"
18+
19+
CCACHE_PREBULT=${CCACHE_PREBULT:=1}
20+
21+
# Repo and branch/tag/commit of ccache to download if we don't have it yet
22+
CCACHE_REPO=${CCACHE_REPO:=https://github.com/ccache/ccache}
23+
CCACHE_VERSION=${CCACHE_VERSION:=4.12}
24+
CCACHE_TAG=${CCACHE_TAG:=v${CCACHE_VERSION}}
25+
LOCAL_DEPS_DIR=${LOCAL_DEPS_DIR:=${PWD}/ext}
26+
# Where to put ccache repo source (default to the ext area)
27+
CCACHE_SRC_DIR=${CCACHE_SRC_DIR:=${LOCAL_DEPS_DIR}/ccache}
28+
# Temp build area (default to a build/ subdir under source)
29+
CCACHE_BUILD_DIR=${CCACHE_BUILD_DIR:=${CCACHE_SRC_DIR}/build}
30+
# Install area for ccache (default to ext/dist)
31+
CCACHE_INSTALL_DIR=${CCACHE_INSTALL_DIR:=${PWD}/ext/dist}
32+
CCACHE_CONFIG_OPTS=${CCACHE_CONFIG_OPTS:=}
33+
34+
35+
# if [[ `uname` == "Linux" && `uname -m` == "x86_64" ]] ; then
36+
if [[ `uname` == "Linux" ]] ; then
37+
mkdir -p ${CCACHE_SRC_DIR}
38+
pushd ${CCACHE_SRC_DIR}
39+
40+
if [[ "$CCACHE_PREBUILT" != "0" ]] ; then
41+
#
42+
# Try to download -- had trouble with this on runners
43+
#
44+
CCACHE_DESCRIPTOR="ccache-${CCACHE_VERSION}-linux-x86_64"
45+
curl --location "${CCACHE_REPO}/releases/download/${CCACHE_TAG}/${CCACHE_DESCRIPTOR}.tar.xz" -o ccache.tar.xz
46+
tar xJvf ccache.tar.xz
47+
mkdir -p ${CCACHE_INSTALL_DIR}/bin
48+
cp ${CCACHE_SRC_DIR}/${CCACHE_DESCRIPTOR}/ccache ${CCACHE_INSTALL_DIR}/bin
49+
else
50+
# Clone ccache project from GitHub and build
51+
if [[ ! -e ${CCACHE_SRC_DIR}/.git ]] ; then
52+
echo "git clone ${CCACHE_REPO} ${CCACHE_SRC_DIR}"
53+
git clone ${CCACHE_REPO} ${CCACHE_SRC_DIR}
54+
fi
55+
56+
echo "git checkout ${CCACHE_TAG} --force"
57+
git checkout ${CCACHE_TAG} --force
58+
59+
if [[ -z $DEP_DOWNLOAD_ONLY ]]; then
60+
time cmake -S . -B ${CCACHE_BUILD_DIR} -DCMAKE_BUILD_TYPE=Release \
61+
-DCMAKE_INSTALL_PREFIX=${CCACHE_INSTALL_DIR} \
62+
-DENABLE_TESTING=OFF -DENABLE_DOCUMENTATION=OFF \
63+
${CCACHE_CONFIG_OPTS}
64+
time cmake --build ${CCACHE_BUILD_DIR} --config Release --target install
65+
fi
66+
fi
67+
68+
popd
69+
ls ${CCACHE_INSTALL_DIR}
70+
ls ${CCACHE_INSTALL_DIR}/bin
71+
echo "CCACHE_INSTALL_DIR=$CCACHE_INSTALL_DIR"
72+
echo "CCACHE_DIR=$CCACHE_DIR"
73+
mkdir -p $CCACHE_DIR
74+
ls $CCACHE_DIR || true
75+
export PATH=${CCACHE_INSTALL_DIR}/bin:$PATH
76+
${CCACHE_INSTALL_DIR}/bin/ccache -sv
77+
fi

0 commit comments

Comments
 (0)