Skip to content

Commit 0e7b3e6

Browse files
cedrik-fuoco-adskcedrik-fuoco-adsk
authored
ci: SG-42352: Optimize CI strategy (#1155)
### ci: Optimize CI strategy ### Linked issues n/a ### Summarize your change. - Split CI matrix into tiered jobs per platform to reduce PR build times. - Extracted build steps into reusable composite actions to avoid duplication across jobs. - Added cmake change detection to trigger Debug builds on PRs when build system files are modified. ### Describe the reason for the change. The full CI matrix (16 jobs) runs on every PR, wasting runner minutes on Debug and CY2023 builds that aren't needed for basic PR validation. I explored dynamic matrix generation using fromJSON, but it had issues. The inline JSON matrices were unreadable and fragile (a missing comma causes opaque runtime errors), and fromJSON requires a specific {"include": [...]} structure that's easy to get wrong. Job-level if conditions with matrix context were also ruled out since GitHub Actions evaluates if before matrix expansion, causing "Unrecognized named-value: 'matrix'" errors. The final approach splits each platform workflow into separate jobs with explicit YAML matrices, keeping everything readable and maintainable. ### Describe what you have tested and on which operating system. CI Add a list of changes, and note any that might need special attention during the review. - `.github/workflows/ci.yml`: Added detect-changes job `using dorny/paths-filter` to detect cmake file changes (cmake/**, **/CMakeLists.txt, **/*.cmake). Passes `full_matrix` and `run_debug` inputs to all platform workflows. All jobs have AcademySoftwareFoundation owner check. - `.github/workflows/ci-linux.yml`: Three jobs: `rocky-linux-pr (always, CY2024 Release)`, `rocky-linux-debug` (when cmake changes or full matrix, CY2024 Debug), `rocky-linux-cy2023` (full matrix only, CY2023 Release+Debug). - `.github/workflows/ci-macos.yml`: Two jobs — `macos-pr` (always, CY2024 Release), `macos-debug `(when cmake changes or full matrix, CY2024 Debug). **No CY2023 job since macOS was already CY2024-only.** - `.github/workflows/ci-windows.yml`: Three jobs: `windows-pr` (always, CY2024 Release), `windows-debug` (when cmake changes or full matrix, CY2024 Debug), `windows-cy2023` (full matrix only, CY2023 Release+Debug). - `.github/actions/build-{linux,macos,windows}/action.yml`: New composite actions containing all platform-specific build steps. Qt archive/module constants are defined as input defaults. Note: actions/checkout must run in the workflow job before calling these actions. --------- Signed-off-by: Cédrik Fuoco <cedrik.fuoco@autodesk.com>
1 parent 946e73c commit 0e7b3e6

File tree

7 files changed

+1112
-755
lines changed

7 files changed

+1112
-755
lines changed

‎.github/actions/build-linux/action.yml‎

Lines changed: 310 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,310 @@
1+
name: Build OpenRV on Linux
2+
description: Build and test OpenRV on Rocky Linux
3+
4+
inputs:
5+
rocky-version:
6+
required: true
7+
build-type:
8+
required: true
9+
qt-version:
10+
required: true
11+
cmake-version:
12+
required: true
13+
python-version:
14+
required: true
15+
vfx-platform:
16+
required: true
17+
extra_repo:
18+
required: true
19+
qt5_modules:
20+
required: true
21+
rocky_qt5_archives:
22+
required: false
23+
default: 'icu qt3d qtbase qtconnectivity qtdeclarative qtgraphicaleffects qtimageformats qtlocation qtmultimedia qtquickcontrols qtquickcontrols2 qtremoteobjects qtscxml qtsensors qtsvg qttools qttranslations qtwayland qtwebchannel qtwebsockets qtwebview qtx11extras qtxmlpatterns'
24+
rocky_qt6_modules:
25+
required: false
26+
default: 'debug_info qt3d qt5compat qtcharts qtconnectivity qtdatavis3d qtgrpc qthttpserver qtimageformats qtlanguageserver qtlocation qtlottie qtmultimedia qtnetworkauth qtpdf qtpositioning qtquick3d qtquick3dphysics qtquickeffectmaker qtquicktimeline qtremoteobjects qtscxml qtsensors qtserialbus qtserialport qtshadertools qtspeech qtvirtualkeyboard qtwaylandcompositor qtwebchannel qtwebengine qtwebsockets qtwebview'
27+
rocky_qt6_archives:
28+
required: false
29+
default: 'icu qtbase qtdeclarative qtsvg qttools qttranslations qtwayland'
30+
31+
runs:
32+
using: "composite"
33+
steps:
34+
- name: Display disk space
35+
run: df -h /
36+
shell: bash
37+
38+
- name: Clear up some spaces on disk
39+
run: |
40+
# Delete files from host through the volumes
41+
# It will have some "cannot remove" message, but but atleast 10GB will be cleared.
42+
rm -rf /github/home/android || true
43+
rm -rf /github/home/dotnet || true
44+
rm -rf /github/home/ghc || true
45+
rm -rf /github/home/.ghcup || true
46+
rm -rf /github/home/graalvm || true
47+
rm -rf /github/home/powershell || true
48+
rm -rf /github/home/chromium || true
49+
rm -rf /github/home/node_modules || true
50+
rm -rf /github/home/CodeQL || true
51+
rm -rf /github/home/boost || true
52+
shell: bash
53+
54+
- name: Display disk space
55+
run: df -h /
56+
shell: bash
57+
58+
- name: Install system dependencies
59+
run: |
60+
retry_count=0
61+
max_retries=5
62+
63+
until [ $retry_count -ge $max_retries ]; do
64+
echo "Attempt $((retry_count + 1)) of $max_retries"
65+
66+
if dnf install -y epel-release && \
67+
dnf config-manager --set-enabled ${{ inputs.extra_repo }} devel && \
68+
dnf groupinstall "Development Tools" -y && \
69+
dnf install -y alsa-lib-devel autoconf automake avahi-compat-libdns_sd-devel bison bzip2-devel cmake-gui curl-devel flex gcc gcc-c++ git libXcomposite libXi-devel libaio-devel libffi-devel nasm ncurses-devel nss libtool libxkbcommon libXcomposite libXdamage libXrandr libXtst libXcursor mesa-libOSMesa mesa-libOSMesa-devel meson openssl-devel patch pulseaudio-libs pulseaudio-libs-glib2 ocl-icd ocl-icd-devel opencl-headers qt5-qtbase-devel readline-devel sqlite-devel systemd-devel tcl-devel tcsh tk-devel yasm zip zlib-devel wget patchelf pcsc-lite libxkbfile perl-IPC-Cmd && \
70+
dnf install -y libX11-devel libXext-devel libXrender-devel libXrandr-devel libXcursor-devel libXi-devel libXxf86vm-devel libxkbcommon-devel && \
71+
dnf install -y xz-devel mesa-libGLU mesa-libGLU-devel && \
72+
dnf clean all; then
73+
echo "Dependencies installed successfully"
74+
break
75+
else
76+
retry_count=$((retry_count + 1))
77+
if [ $retry_count -lt $max_retries ]; then
78+
echo "Attempt $retry_count failed. Retrying in 5 seconds..."
79+
sleep 5
80+
fi
81+
fi
82+
done
83+
84+
if [ $retry_count -ge $max_retries ]; then
85+
echo "Failed to install dependencies after $max_retries attempts"
86+
exit 1
87+
fi
88+
shell: bash
89+
90+
- name: Install other system dependencies
91+
if: ${{ inputs.rocky-version == '9' }}
92+
run: |
93+
retry_count=0
94+
max_retries=5
95+
96+
until [ $retry_count -ge $max_retries ]; do
97+
echo "Attempt $((retry_count + 1)) of $max_retries"
98+
99+
if dnf install -y perl-CPAN && \
100+
cpan FindBin; then
101+
echo "Dependencies installed successfully"
102+
break
103+
else
104+
retry_count=$((retry_count + 1))
105+
if [ $retry_count -lt $max_retries ]; then
106+
echo "Attempt $retry_count failed. Retrying in 5 seconds..."
107+
sleep 5
108+
fi
109+
fi
110+
done
111+
112+
if [ $retry_count -ge $max_retries ]; then
113+
echo "Failed to install dependencies after $max_retries attempts"
114+
exit 1
115+
fi
116+
shell: bash
117+
118+
- name: Install GCC 11 toolchain for Rocky Linux 8
119+
if: ${{ inputs.rocky-version == '8' }}
120+
run: |
121+
retry_count=0
122+
max_retries=5
123+
124+
until [ $retry_count -ge $max_retries ]; do
125+
echo "Attempt $((retry_count + 1)) of $max_retries"
126+
127+
if dnf install -y gcc-toolset-11-toolchain; then
128+
echo "GCC 11 toolchain installed successfully"
129+
break
130+
else
131+
retry_count=$((retry_count + 1))
132+
if [ $retry_count -lt $max_retries ]; then
133+
echo "Attempt $retry_count failed. Retrying in 5 seconds..."
134+
sleep 5
135+
fi
136+
fi
137+
done
138+
139+
if [ $retry_count -ge $max_retries ]; then
140+
echo "Failed to install GCC 11 toolchain after $max_retries attempts"
141+
exit 1
142+
fi
143+
shell: bash
144+
145+
- name: Enable GCC 11 toolchain for Rocky Linux 8
146+
if: ${{ inputs.rocky-version == '8' }}
147+
run: |
148+
source /opt/rh/gcc-toolset-11/enable
149+
echo "/opt/rh/gcc-toolset-11/root/usr/bin" >> $GITHUB_PATH
150+
echo "CC=/opt/rh/gcc-toolset-11/root/usr/bin/gcc" >> $GITHUB_ENV
151+
echo "CXX=/opt/rh/gcc-toolset-11/root/usr/bin/g++" >> $GITHUB_ENV
152+
echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> $GITHUB_ENV
153+
gcc --version
154+
g++ --version
155+
shell: bash
156+
157+
- name: Add repository as safe directory
158+
run: git config --global --add safe.directory /__w/OpenRV/OpenRV
159+
shell: bash
160+
161+
- name: Install pyenv
162+
run: |
163+
curl https://pyenv.run | bash
164+
165+
export PYENV_ROOT="$HOME/.pyenv"
166+
echo "$PYENV_ROOT/shims" >> $GITHUB_PATH
167+
echo "$PYENV_ROOT/bin" >> $GITHUB_PATH
168+
shell: bash
169+
170+
- name: Setup pyenv
171+
run: |
172+
echo $PATH
173+
pyenv install ${{ inputs.python-version }}
174+
pyenv global ${{ inputs.python-version }}
175+
shell: bash
176+
177+
- name: Display Python installation location
178+
run: python -c "import sys; print(sys.executable)"
179+
shell: bash
180+
181+
- name: Install Ninja
182+
run: |
183+
wget https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip
184+
unzip ninja-linux.zip -d ./ninja
185+
echo "$(pwd)/ninja" >> $GITHUB_PATH
186+
ninja --version
187+
shell: bash
188+
189+
- name: Install CMake
190+
run: |
191+
curl -SL -o cmake.tar.gz https://github.com/Kitware/CMake/releases/download/v${{ inputs.cmake-version }}/cmake-${{ inputs.cmake-version }}-Linux-x86_64.tar.gz
192+
tar -xzf cmake.tar.gz
193+
ls
194+
mv cmake-${{ inputs.cmake-version }}-linux-x86_64 ./cmake-${{ inputs.cmake-version }}
195+
echo "$(pwd)/cmake-${{ inputs.cmake-version }}/bin" >> $GITHUB_PATH
196+
shell: bash
197+
198+
- name: Prepare Qt folder
199+
run: mkdir -p "${{ github.workspace }}/deps"
200+
shell: bash
201+
202+
- name: Install Qt ${{ inputs.qt-version }}
203+
if: ${{ inputs.vfx-platform == 'CY2023' }}
204+
uses: jurplel/install-qt-action@v4
205+
with:
206+
version: '${{ inputs.qt-version }}'
207+
host: 'linux'
208+
target: 'desktop'
209+
arch: 'gcc_64'
210+
dir: "${{ github.workspace }}/deps"
211+
install-deps: 'false'
212+
modules: ${{ inputs.qt5_modules }}
213+
archives: ${{ inputs.rocky_qt5_archives }}
214+
cache: false
215+
setup-python: 'false'
216+
tools: 'tools_qtcreator'
217+
set-env: 'true'
218+
tools-only: 'false'
219+
aqtversion: '==3.2.*'
220+
221+
- name: Install Qt ${{ inputs.qt-version }}
222+
if: ${{ inputs.vfx-platform == 'CY2024' }}
223+
uses: jurplel/install-qt-action@v4
224+
with:
225+
version: '${{ inputs.qt-version }}'
226+
host: 'linux'
227+
target: 'desktop'
228+
arch: 'gcc_64'
229+
dir: "${{ github.workspace }}/deps"
230+
install-deps: 'false'
231+
modules: ${{ inputs.rocky_qt6_modules }}
232+
archives: ${{ inputs.rocky_qt6_archives }}
233+
cache: false
234+
setup-python: 'false'
235+
tools: 'tools_qtcreator'
236+
set-env: 'true'
237+
tools-only: 'false'
238+
aqtversion: '==3.2.*'
239+
240+
- name: Install Python dependencies
241+
run: python3 -m pip install --user --upgrade -r requirements.txt
242+
shell: bash
243+
244+
- name: Install Rust toolchain
245+
run: |
246+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
247+
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
248+
shell: bash
249+
250+
- name: Verify Rust installation
251+
run: |
252+
source $HOME/.cargo/env
253+
rustc --version
254+
cargo --version
255+
shell: bash
256+
257+
# For Rocky Linux, we can't cache the dependencies of Open RV
258+
# because there is not enough room in the cache (10GB limit) for all platforms.
259+
# ccache is used for this run only (not stored in Actions cache) to speed up the build.
260+
# Let's revisit this if we can reduce the size of the cache (potentially dropping x86_64 for Mac)
261+
# or by moving some dependencies from the cache to Conan or another dependency manager.
262+
- name: Install ccache
263+
run: dnf install -y ccache
264+
shell: bash
265+
266+
# Uncomment to persist ccache in Actions cache (uses repo 10GB limit).
267+
# - name: Setup ccache
268+
# uses: hendrikmuhs/ccache-action@5ebbd400eff9e74630f759d94ddd7b6c26299639
269+
# with:
270+
# key: rocky-${{ inputs.rocky-version }}-${{ inputs.vfx-platform }}-${{ inputs.build-type }}
271+
# max-size: 2G
272+
# verbose: 2
273+
274+
- name: Configure OpenRV
275+
if: ${{ inputs.vfx-platform == 'CY2023' }}
276+
run: |
277+
cmake --version
278+
export QT_HOME=/__w/OpenRV/OpenRV/deps/Qt/${{ inputs.qt-version}}/gcc_64
279+
cmake -B _build -G "Ninja" -DCMAKE_BUILD_TYPE=${{ inputs.build-type }} -DRV_DEPS_QT_LOCATION=$QT_HOME -DRV_VFX_PLATFORM=${{ inputs.vfx-platform }}
280+
shell: bash
281+
282+
- name: Configure OpenRV
283+
if: ${{ inputs.vfx-platform == 'CY2024' }}
284+
run: |
285+
cmake --version
286+
export QT_HOME=/__w/OpenRV/OpenRV/deps/Qt/${{ inputs.qt-version}}/gcc_64
287+
cmake -B _build -G "Ninja" -DCMAKE_BUILD_TYPE=${{ inputs.build-type }} -DRV_DEPS_QT_LOCATION=$QT_HOME -DRV_VFX_PLATFORM=${{ inputs.vfx-platform }}
288+
shell: bash
289+
290+
- name: Build OpenRV dependencies
291+
run: cmake --build _build --config ${{ inputs.build-type }} --target dependencies --parallel=$(python -c 'import os; print(os.cpu_count())') -v
292+
shell: bash
293+
294+
- name: Build OpenRV main executable
295+
run: cmake --build _build --config ${{ inputs.build-type }} --target main_executable --parallel=$(python -c 'import os; print(os.cpu_count())') -v
296+
shell: bash
297+
298+
- name: Tests
299+
run: ctest --test-dir _build -C ${{ inputs.build-type }} --extra-verbose
300+
shell: bash
301+
302+
- name: Free some spaces for the install steps
303+
run: |
304+
rm -rf _build/RV_DEPS_*
305+
rm -rf _build/_deps
306+
shell: bash
307+
308+
- name: Install OpenRV
309+
run: cmake --install _build --prefix $(pwd)/_install --config ${{ inputs.build-type }}
310+
shell: bash

0 commit comments

Comments
 (0)