feat(core): Add support for building and running on manylinux_2_28; Add manylinux_2_28 dependency container images. #1041
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| FROM quay.io/pypa/manylinux_2_28_aarch64 | ||
|
|
||
| WORKDIR /root | ||
|
|
||
| RUN mkdir -p ./tools/scripts/lib_install | ||
| COPY ./tools/scripts/lib_install ./tools/scripts/lib_install | ||
|
|
||
| RUN ./tools/scripts/lib_install/manylinux_2_28/install-all.sh | ||
|
|
||
| # Remove cached files | ||
| RUN dnf clean all && rm -rf /var/cache/dnf /tmp/* /var/tmp/* | ||
|
|
||
|
Comment on lines
+10
to
+12
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Redundant cache purge
🤖 Prompt for AI Agents |
||
| # NOTE: Don't flatten the image or else we'll lose any environment modifications from the base | ||
| # image. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -eu | ||
| set -o pipefail | ||
|
|
||
| script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" | ||
| component_root="${script_dir}/../../../" | ||
|
|
||
| build_cmd=( | ||
| docker buildx build | ||
| --platform linux/arm64 | ||
| --tag clp-core-dependencies-aarch64-manylinux_2_28:dev | ||
| "$component_root" | ||
| --file "${script_dir}/Dockerfile" | ||
| --load | ||
| ) | ||
|
Comment on lines
+9
to
+16
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Allow caller-supplied tag/version Hard-coding 🤖 Prompt for AI Agents |
||
|
|
||
| if command -v git >/dev/null && git -C "$script_dir" rev-parse --is-inside-work-tree >/dev/null ; | ||
| then | ||
| build_cmd+=( | ||
| --label "org.opencontainers.image.revision=$(git -C "$script_dir" rev-parse HEAD)" | ||
| --label "org.opencontainers.image.source=$(git -C "$script_dir" remote get-url origin)" | ||
| ) | ||
| fi | ||
|
Comment on lines
+18
to
+24
Contributor
There was a problem hiding this comment. 🛠️ Refactor suggestion Git labels: handle detached HEAD & shallow clones
🤖 Prompt for AI Agents |
||
|
|
||
| echo "Running: ${build_cmd[*]}" | ||
| "${build_cmd[@]}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| FROM quay.io/pypa/manylinux_2_28_x86_64 | ||
|
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Pin base image with an immutable digest for repeatable builds
🧰 Tools🪛 Checkov (3.2.334)[LOW] 1-1: Ensure the base image uses a non latest version tag (CKV_DOCKER_7) [LOW] 1-14: Ensure that HEALTHCHECK instructions have been added to container images (CKV_DOCKER_2) [LOW] 1-14: Ensure that a user for the container has been created (CKV_DOCKER_3) 🪛 Hadolint (2.12.0)[warning] 1-1: Always tag the version of an image explicitly (DL3006) 🤖 Prompt for AI Agents |
||
|
|
||
| WORKDIR /root | ||
|
|
||
| RUN mkdir -p ./tools/scripts/lib_install | ||
| COPY ./tools/scripts/lib_install ./tools/scripts/lib_install | ||
|
|
||
| RUN ./tools/scripts/lib_install/manylinux_2_28/install-all.sh | ||
|
|
||
| # Remove cached files | ||
| RUN dnf clean all && rm -rf /var/cache/dnf /tmp/* /var/tmp/* | ||
|
|
||
|
Comment on lines
+10
to
+12
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Clean up yum metadata before final layer
🤖 Prompt for AI Agents |
||
| # NOTE: Don't flatten the image or else we'll lose any environment modifications from the base | ||
| # image. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,27 @@ | ||||||||||||||||||||||||||||||
| #!/usr/bin/env bash | ||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||
| set -eu | ||||||||||||||||||||||||||||||
| set -o pipefail | ||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||
| script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" | ||||||||||||||||||||||||||||||
| component_root="${script_dir}/../../../" | ||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||
|
Comment on lines
+6
to
+8
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Make path variables Both -script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
-component_root="${script_dir}/../../../"
+readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+readonly component_root="${script_dir}/../../../"📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||
| build_cmd=( | ||||||||||||||||||||||||||||||
| docker buildx build | ||||||||||||||||||||||||||||||
| --platform linux/amd64 | ||||||||||||||||||||||||||||||
| --tag clp-core-dependencies-x86-manylinux_2_28:dev | ||||||||||||||||||||||||||||||
| "$component_root" | ||||||||||||||||||||||||||||||
| --file "${script_dir}/Dockerfile" | ||||||||||||||||||||||||||||||
| --load | ||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||
| if command -v git >/dev/null && git -C "$script_dir" rev-parse --is-inside-work-tree >/dev/null ; | ||||||||||||||||||||||||||||||
| then | ||||||||||||||||||||||||||||||
| build_cmd+=( | ||||||||||||||||||||||||||||||
| --label "org.opencontainers.image.revision=$(git -C "$script_dir" rev-parse HEAD)" | ||||||||||||||||||||||||||||||
| --label "org.opencontainers.image.source=$(git -C "$script_dir" remote get-url origin)" | ||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||||||
|
Comment on lines
+18
to
+24
Contributor
There was a problem hiding this comment. 🛠️ Refactor suggestion Use repository root for Git metadata labels When the script lives in a sub-directory, - --label "org.opencontainers.image.revision=$(git -C "$script_dir" rev-parse HEAD)"
- --label "org.opencontainers.image.source=$(git -C "$script_dir" remote get-url origin)"
+ --label "org.opencontainers.image.revision=$(git -C "$component_root" rev-parse HEAD)"
+ --label "org.opencontainers.image.source=$(git -C "$component_root" remote get-url origin)"📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||
| echo "Running: ${build_cmd[*]}" | ||||||||||||||||||||||||||||||
| "${build_cmd[@]}" | ||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -eu | ||
| set -o pipefail | ||
|
|
||
| script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" | ||
|
|
||
|
Comment on lines
+3
to
+7
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Declare The value of -script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"🤖 Prompt for AI Agents |
||
| "${script_dir}/install-prebuilt-packages.sh" | ||
| "${script_dir}/install-packages-from-source.sh" | ||
|
|
||
| # TODO: https://github.com/y-scope/clp/issues/795 | ||
| "${script_dir}/../check-cmake-version.sh" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,22 @@ | ||||||||||||||||||||||
| #!/usr/bin/env bash | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| set -eu | ||||||||||||||||||||||
| set -o pipefail | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" | ||||||||||||||||||||||
| lib_install_scripts_dir="${script_dir}/.." | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
|
Comment on lines
+3
to
+8
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Declare constants as
-script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
-lib_install_scripts_dir="${script_dir}/.."
+readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+readonly lib_install_scripts_dir="${script_dir}/.."📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||
| # NOTE: The remaining installation scripts depend on boost, so we install it beforehand. | ||||||||||||||||||||||
| "${lib_install_scripts_dir}/install-boost.sh" 1.87.0 | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| # NOTE: | ||||||||||||||||||||||
| # 1. libarchive may statically link with LZMA, LZ4, and Zstandard, so we install them beforehand. | ||||||||||||||||||||||
| # 2. The versions of libarchive, LZMA, LZ4, and Zstandard available in manylinux_2_28's package | ||||||||||||||||||||||
| # repositories are either dated or don't include static libraries, so we install more recent | ||||||||||||||||||||||
| # versions from source. | ||||||||||||||||||||||
| "${lib_install_scripts_dir}/liblzma.sh" 5.8.1 | ||||||||||||||||||||||
| "${lib_install_scripts_dir}/lz4.sh" 1.10.0 | ||||||||||||||||||||||
| "${lib_install_scripts_dir}/zstandard.sh" 1.5.7 | ||||||||||||||||||||||
| "${lib_install_scripts_dir}/libarchive.sh" 3.8.0 | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| "${lib_install_scripts_dir}/msgpack.sh" 7.0.0 | ||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,45 @@ | ||||||||||||||||||||||||||||||||||||||
| #!/usr/bin/env bash | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| set -eu | ||||||||||||||||||||||||||||||||||||||
| set -o pipefail | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| dnf install -y \ | ||||||||||||||||||||||||||||||||||||||
| gcc-c++ \ | ||||||||||||||||||||||||||||||||||||||
| java-11-openjdk \ | ||||||||||||||||||||||||||||||||||||||
| jq \ | ||||||||||||||||||||||||||||||||||||||
| libcurl-devel \ | ||||||||||||||||||||||||||||||||||||||
| mariadb-connector-c-devel \ | ||||||||||||||||||||||||||||||||||||||
| openssl-devel \ | ||||||||||||||||||||||||||||||||||||||
| zlib-devel \ | ||||||||||||||||||||||||||||||||||||||
| zlib-static | ||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+6
to
+14
Contributor
There was a problem hiding this comment. 🧹 Nitpick (assertive) Trim the image by suppressing weak dependencies
-dnf install -y \
+dnf install -y --setopt=install_weak_deps=False \📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| # Determine architecture for `task` release to install | ||||||||||||||||||||||||||||||||||||||
| rpm_arch=$(rpm --eval "%{_arch}") | ||||||||||||||||||||||||||||||||||||||
| case "$rpm_arch" in | ||||||||||||||||||||||||||||||||||||||
| "x86_64") | ||||||||||||||||||||||||||||||||||||||
| task_pkg_arch="amd64" | ||||||||||||||||||||||||||||||||||||||
| ;; | ||||||||||||||||||||||||||||||||||||||
| "aarch64") | ||||||||||||||||||||||||||||||||||||||
| task_pkg_arch="arm64" | ||||||||||||||||||||||||||||||||||||||
| ;; | ||||||||||||||||||||||||||||||||||||||
| *) | ||||||||||||||||||||||||||||||||||||||
| echo "Error: Unsupported architecture - $rpm_arch" | ||||||||||||||||||||||||||||||||||||||
| exit 1 | ||||||||||||||||||||||||||||||||||||||
| ;; | ||||||||||||||||||||||||||||||||||||||
| esac | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| # Install `task` | ||||||||||||||||||||||||||||||||||||||
| # NOTE: We lock `task` to a version < 3.43 to avoid https://github.com/y-scope/clp/issues/872 | ||||||||||||||||||||||||||||||||||||||
| task_pkg_path=$(mktemp -t --suffix ".rpm" task-pkg.XXXXXXXXXX) || exit 1 | ||||||||||||||||||||||||||||||||||||||
| curl \ | ||||||||||||||||||||||||||||||||||||||
| --fail \ | ||||||||||||||||||||||||||||||||||||||
| --location \ | ||||||||||||||||||||||||||||||||||||||
| --output "$task_pkg_path" \ | ||||||||||||||||||||||||||||||||||||||
| --show-error \ | ||||||||||||||||||||||||||||||||||||||
| "https://github.com/go-task/task/releases/download/v3.42.1/task_linux_${task_pkg_arch}.rpm" | ||||||||||||||||||||||||||||||||||||||
| dnf install --assumeyes "$task_pkg_path" | ||||||||||||||||||||||||||||||||||||||
| rm "$task_pkg_path" | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| # Downgrade to CMake v3 to work around https://github.com/y-scope/clp/issues/795 | ||||||||||||||||||||||||||||||||||||||
| pipx uninstall cmake | ||||||||||||||||||||||||||||||||||||||
| pipx install cmake~=3.31 | ||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,38 +1,115 @@ | ||
| # Containers | ||
|
|
||
| We maintain several Docker container images that are useful for building and running CLP. All images | ||
| can be built and used locally, but some are available to download from | ||
| [GitHub Packages][gh-packages]. | ||
|
|
||
| To build an image locally, run the `build.sh` script in the image's directory. | ||
|
|
||
| ## clp-core-dependencies-<arch>-manylinux_2_28 | ||
|
|
||
| Images containing the dependencies necessary to build CLP core in a [manylinux_2_28][manylinux_2_28] | ||
| environment (aarch64 or x86). | ||
|
|
||
| Binaries built on manylinux_2_28 (based on AlmaLinux 8) are expected to be compatible with other | ||
| distros using glibc 2.28+, including: | ||
|
|
||
| * CentOS/RHEL 8+ | ||
| * Debian 10+ | ||
| * Fedora 29+ | ||
| * Ubuntu 18.10+ | ||
|
|
||
| ### clp-core-dependencies-aarch64-manylinux_2_28 | ||
|
|
||
| * Path: | ||
|
|
||
| ```text | ||
| components/core/tools/docker-images/clp-env-base-manylinux_2_28-aarch64 | ||
| ``` | ||
|
|
||
| ### clp-core-dependencies-x86-manylinux_2_28 | ||
|
|
||
| * Path: | ||
|
|
||
| ```text | ||
| components/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64 | ||
| ``` | ||
|
|
||
| ## clp-core-dependencies-x86-centos-stream-9 | ||
|
|
||
| An image containing the dependencies necessary to build CLP core in a CentOS Stream 9 x86 | ||
| environment. | ||
|
|
||
| * [GitHub Packages page][core-deps-centos-stream-9] | ||
| * Pull command: | ||
|
|
||
| ```bash | ||
| docker pull ghcr.io/y-scope/clp/clp-core-dependencies-x86-centos-stream-9:main | ||
| ``` | ||
|
|
||
| * Path: | ||
|
|
||
| ```text | ||
| components/core/tools/docker-images/clp-env-base-centos-stream-9 | ||
| ``` | ||
|
|
||
| ## clp-core-dependencies-x86-ubuntu-jammy | ||
|
|
||
| An image containing the dependencies necessary to build CLP core in an Ubuntu Jammy x86 | ||
| environment. | ||
|
|
||
| * [GitHub Packages page][core-deps-ubuntu-jammy] | ||
| * Pull command: | ||
|
|
||
| ```bash | ||
| docker pull ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy:main | ||
| ``` | ||
|
|
||
| * Path: | ||
|
|
||
| ```text | ||
| components/core/tools/docker-images/clp-env-base-ubuntu-jammy | ||
| ``` | ||
|
|
||
| ## clp-core-x86-ubuntu-jammy | ||
|
|
||
| An image containing the CLP core binaries (`clg`, `clp`, `clp-s`, `glt`, etc.) built in an Ubuntu | ||
| Jammy x86 environment. | ||
|
|
||
| * [GitHub Packages page][core-ubuntu-jammy] | ||
| * Pull command: | ||
|
|
||
| ```bash | ||
| docker pull ghcr.io/y-scope/clp/clp-core-x86-ubuntu-jammy:main | ||
| ``` | ||
|
|
||
| * Path: | ||
|
|
||
| ```text | ||
| components/core/tools/docker-images/clp-core-ubuntu-jammy | ||
| ``` | ||
|
|
||
| ## clp-execution-x86-ubuntu-jammy | ||
|
|
||
| An image containing the dependencies necessary to run the CLP package in an Ubuntu Jammy x86 | ||
| environment. | ||
|
|
||
| * [GitHub Packages page][exe-ubuntu-jammy] | ||
| * Pull command: | ||
|
|
||
| ```bash | ||
| docker pull ghcr.io/y-scope/clp/clp-execution-x86-ubuntu-jammy:main | ||
| ``` | ||
|
|
||
| * Path: | ||
|
|
||
| ```text | ||
| tools/docker-images/clp-execution-base-ubuntu-jammy | ||
| ``` | ||
|
|
||
| [core-deps-centos-stream-9]: https://github.com/y-scope/clp/pkgs/container/clp%2Fclp-core-dependencies-x86-centos-stream-9 | ||
| [core-deps-ubuntu-jammy]: https://github.com/y-scope/clp/pkgs/container/clp%2Fclp-core-dependencies-x86-ubuntu-jammy | ||
| [core-ubuntu-jammy]: https://github.com/y-scope/clp/pkgs/container/clp%2Fclp-core-x86-ubuntu-jammy | ||
| [exe-ubuntu-jammy]: https://github.com/y-scope/clp/pkgs/container/clp%2Fclp-execution-x86-ubuntu-jammy | ||
| [gh-packages]: https://github.com/orgs/y-scope/packages?repo_name=clp | ||
| [manylinux_2_28]: https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux_2_28-almalinux-8-based |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧹 Nitpick (assertive)
Base image tag should be immutable
Same comment as the x86_64 Dockerfile: pin the manylinux base image by digest to ensure deterministic builds.
🧰 Tools
🪛 Checkov (3.2.334)
[LOW] 1-1: Ensure the base image uses a non latest version tag
(CKV_DOCKER_7)
[LOW] 1-14: Ensure that HEALTHCHECK instructions have been added to container images
(CKV_DOCKER_2)
[LOW] 1-14: Ensure that a user for the container has been created
(CKV_DOCKER_3)
🪛 Hadolint (2.12.0)
[warning] 1-1: Always tag the version of an image explicitly
(DL3006)
🤖 Prompt for AI Agents