feat(ci): Enable parallel builds for Ubuntu 20.04 & 24.04 base images - #13964

Merged

hunsche merged 44 commits into

feat/multi-ubuntu-builds

Oct 9, 2025

hunsche commented Sep 8, 2025 •

edited

Loading

Contributor

Part of: b/441792502

Description

This PR modifies the existing base image testing pipeline (trial_build.py) to support the Ubuntu 24.04 migration. It enables the parallel build and testing of base images for latest, ubuntu-20-04, and ubuntu-24-04.

Instead of introducing a new, separate CI pipeline, this change integrates the multi-version logic directly into the established trial_build.py workflow. This approach ensures that both local development builds and remote CI builds are consistent and leverage the existing, robust testing infrastructure.

Changes Made

build_and_push_test_images.py Refactored:
- A BASE_IMAGE_VERSIONS list now serves as the single source of truth for all target Ubuntu versions.
- The GCB build function (gcb_build_and_push_images) was updated to generate a single Google Cloud Build trigger with parallel steps for each image and version combination.
- The local build function (build_and_push_images) was also updated to build all versions in parallel using multiprocessing, ensuring local and remote build parity.
Dynamic Dockerfile Selection: The script now intelligently selects the correct Dockerfile (e.g., base-image/ubuntu-24-04.Dockerfile) based on the target version, skipping any that do not exist.
Consistent Versioned Tagging: Helper functions were updated to apply the correct tags to the images (e.g., gcr.io/oss-fuzz-base/base-image:ubuntu-24-04-testing).

How to Verify

Trigger the trial_build.py script against a set of projects.
Observe the Google Cloud Build logs. A single build should be triggered containing a large number of parallel steps, one for each image/version combination (e.g., base-image:latest, base-image:ubuntu-20-04, etc.).
Verify that the build completes successfully and that the test images are pushed to the Artifact Registry with the correct versioned tags.

vitaliset approved these changes

View reviewed changes

DavidKorczynski previously requested changes

View reviewed changes

infra/base-images/all.sh Outdated

DavidKorczynski commented Sep 9, 2025

Collaborator

rebasing this on top should solve the AFL issue

hunsche commented

View reviewed changes

infra/build/functions/build_and_push_test_images.py Outdated

hunsche commented

View reviewed changes

infra/build/functions/trial_build.py Outdated

hunsche force-pushed the feat/multi-ubuntu-builds branch 7 times, most recently from 2955804 to 3207b66 Compare

September 16, 2025 14:20

hunsche requested a review from DavidKorczynski

September 16, 2025 19:29

hunsche force-pushed the feat/multi-ubuntu-builds branch 2 times, most recently from 33593e5 to ddf13fb Compare

September 22, 2025 21:53

maflcko commented Sep 24, 2025

Contributor

I think the new images are still using the old llvm:

$ git grep OUR_LLVM_REVISION= 2d2bd6976f9197e812e5af1098c667b3b1085472 
2d2bd6976f9197e812e5af1098c667b3b1085472:infra/base-images/base-clang/checkout_build_install_llvm.sh:OUR_LLVM_REVISION=cb2f0d0a5f14
2d2bd6976f9197e812e5af1098c667b3b1085472:infra/base-images/base-clang/checkout_build_install_llvm_ubuntu_20_04.sh:OUR_LLVM_REVISION=llvmorg-18.1.8
2d2bd6976f9197e812e5af1098c667b3b1085472:infra/base-images/base-clang/checkout_build_install_llvm_ubuntu_24_04.sh:OUR_LLVM_REVISION=llvmorg-18.1.8

Not sure if this intentional, but this means the new images will be using the old rust version and projects would have difficutly switching from the "legacy" image (with newer versions) to the new images (with older toolchain versions).

evverx mentioned this pull request

[systemd] trim contacts #14056

Merged

hunsche force-pushed the feat/multi-ubuntu-builds branch 2 times, most recently from f04a5d9 to 0b00121 Compare

September 25, 2025 20:27

hunsche commented Sep 26, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

2 similar comments

hunsche commented Sep 26, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

hunsche commented Sep 26, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

hunsche force-pushed the feat/multi-ubuntu-builds branch 4 times, most recently from 97f2133 to 8e7f8eb Compare

September 26, 2025 22:11

hunsche commented Sep 26, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

3 similar comments

hunsche commented Sep 26, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

hunsche commented Sep 26, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

hunsche commented Sep 27, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

hunsche added 2 commits

October 8, 2025 18:52


          refactor(trial_build): Include build type in build IDs and failure lo…

99a39b7

…gs for better traceability


          fix(base-images): Remove default automake and install specific versio…

…n required by libarchive

hunsche commented Oct 8, 2025

Contributor Author

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

hunsche added 2 commits

October 8, 2025 19:04


          fix(report_generator): Update separator line in final summary for con…

45b2fc8

…sistency


          fix(install_deps): Update apt configuration for non-interactive insta…

c65cdcb

…lls and downgrades

fix(trial_build): Implement exponential backoff for build status checks with retry logic

hunsche commented Oct 8, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address --force-build

hunsche commented Oct 8, 2025

Contributor Author

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

hunsche and others added 2 commits

October 8, 2025 20:19


          Merge branch 'master' into feat/multi-ubuntu-builds

9708bc5


          style: Format code with yapf

0c13e95

hunsche commented Oct 8, 2025

Contributor Author

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address --force-build

hunsche commented Oct 8, 2025

Contributor Author

/gcbrun trial_build.py all --force-build

hunsche commented Oct 8, 2025

Contributor Author

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

hunsche commented Oct 9, 2025

Contributor Author

/gcbrun trial_build.py all --force-build


          fix(build): Update get_build_steps return values for clarity in error…

d2e5e31

… handling

hunsche commented Oct 9, 2025

Contributor Author

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build


          style: Adjust indentation for better readability in build_project_tes…

…t.py

hunsche commented Oct 9, 2025

Contributor Author

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

hunsche merged commit 4b541c7 into master

30 of 31 checks passed

hunsche deleted the feat/multi-ubuntu-builds branch

October 9, 2025 17:11

evverx mentioned this pull request

Update ossfuzz base image to newer Ubuntu (22.04 or 24.04) ? #11835

Open

jsoref mentioned this pull request

The ubuntu_version_sync.yml workflow fails with "Invalid workflow file" #14185

Closed

hunsche mentioned this pull request

feat(abseil-cpp): Migrate abseil-cpp to Ubuntu 24.04 #14216

Closed

DavidKorczynski added a commit that referenced this pull request


          infra: remove -pull from all.sh

021787b

in #13964 `--pull` was added to `docker build`. However, this causes issues when developing the images locally, as we always end up pulling upstream images causing issues when building changes that carry through several base images. Removing `--pull` from this.

DavidKorczynski mentioned this pull request

infra: remove -pull from all.sh #14335

Merged

DavidKorczynski reviewed

View reviewed changes

infra/base-images/all.sh

+                fi
+                echo "Building ${tag} from ${dockerfile}..."
+                docker build --pull -t "${tag}" -f "${dockerfile}" "${image_dir}"

DavidKorczynski Nov 25, 2025 •

edited

Loading

Collaborator

@hunsche Is there a specific reason we added --pull here? asking due to #14335

It's causing issues when updating the base images where changes depend on several base images. We should force to use the local version to be consistent with the original functionality IMO

hunsche Nov 25, 2025

Contributor Author

No specific reason, feel free to change this.

AdamKorcz pushed a commit that referenced this pull request


          infra: remove -pull from all.sh (#14335)

ecddcea

in #13964 `--pull` was added to
`docker build`. However, this causes issues when developing the images
locally, as we always end up pulling upstream images causing issues when
building changes that carry through several base images. Removing
`--pull` from this.

Was running into this when debugging
#14315

evverx mentioned this pull request

RUSTUP_TOOLCHAIN in base-builder-rust:ubuntu-24-04 is older than in base-builder-rust #14220

Open

jonathanmetzman mentioned this pull request

Fix regressions in trial builds and index builds #14883

Merged

jonathanmetzman added a commit that referenced this pull request


          Fix regressions in trial builds and index builds (#14883)

9638e6d

This fixes a few regressions in the build infra.

Put indexer builds in /out fixing uploading regression (#14086)

Fixes regressions in trial builds (#13964):
* Projects no longer build in mismatched images (e.g. Ubuntu 20 projects
in Ubuntu 24 images) that cause build failures leading tested PRs to
appear broken.
* Undoes accidental deletion of index build support in trial builds.
This restores needed functionality.
* Removes legacy builds which are redundant with Ubuntu 20.
* Corrects language to reflect bug fix in #14440.
* Removes emojis and other nonascii characters causing rendering issues.

masumi-ryugo mentioned this pull request

base-builder-rust: bump RUSTUP_TOOLCHAIN to nightly-2025-09-05 on ubuntu-24-04 #15461

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet