Skip to content

Optimize and Enable Runtime Datascience Images for IBM Z[s390x] (Python 3.11 & 3.12) #1513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Optimize and Enable Runtime Datascience Images for IBM Z[s390x] (Python 3.11 & 3.12) #1513

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
12c97e9
Enable s390x support for runtime-datascience (Python 3.11 & 3.12)
Nash-123 Jul 25, 2025
c589816
Update Pipfile.lock for Python 3.12
github-actions[bot] Aug 8, 2025
5108080
Update Pipfile.lock for Python 3.11
github-actions[bot] Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ci/cached-builds/gen_gha_matrix_jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
"runtime-minimal-ubi9-python-3.12",
"jupyter-minimal-ubi9-python-3.11",
"jupyter-minimal-ubi9-python-3.12",
"runtime-datascience-ubi9-python-3.11",
"runtime-datascience-ubi9-python-3.12",
# add more here
}

Expand Down
38 changes: 19 additions & 19 deletions jupyter/trustyai/ubi9-python-3.12/Pipfile.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 19 additions & 19 deletions jupyter/trustyai/ubi9-python-3.12/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -932,9 +932,9 @@ httpcore==1.0.9; python_version >= '3.8' \
httpx==0.28.1; python_version >= '3.8' \
--hash=sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc \
--hash=sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad
huggingface-hub==0.34.3; python_full_version >= '3.8.0' \
--hash=sha256:5444550099e2d86e68b2898b09e85878fbd788fc2957b506c6a79ce060e39492 \
--hash=sha256:d58130fd5aa7408480681475491c0abd7e835442082fbc3ef4d45b6c39f83853
huggingface-hub==0.34.4; python_full_version >= '3.8.0' \
--hash=sha256:9b365d781739c93ff90c359844221beef048403f1bc1f1c123c191257c3c890a \
--hash=sha256:a4228daa6fb001be3f4f4bdaf9a0db00e1739235702848df00885c9b5742c85c
idna==3.10; python_version >= '3.6' \
--hash=sha256:12f65c9b470abda6dc35cf8e63cc574b1c52b11df2c86030af0ac09b01b13ea9 \
--hash=sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3
Expand Down Expand Up @@ -2832,22 +2832,22 @@ ruamel.yaml.clib==0.2.12; python_version >= '3.9' \
s3transfer==0.11.5; python_version >= '3.8' \
--hash=sha256:757af0f2ac150d3c75bc4177a32355c3862a98d20447b69a0161812992fe0bd4 \
--hash=sha256:8c8aad92784779ab8688a61aefff3e28e9ebdce43142808eaa3f0b0f402f68b7
safetensors==0.6.1; python_version >= '3.9' \
--hash=sha256:01b51af8cb7a3870203f2735e3c7c24d1a65fb2846e75613c8cf9d284271eccc \
--hash=sha256:1b62eab84e2c69918b598272504c5d2ebfe64da6c16fdf8682054eec9572534d \
--hash=sha256:294040ff20ebe079a2b4976cfa9a5be0202f56ca4f7f190b4e52009e8c026ceb \
--hash=sha256:4720957052d57c5ac48912c3f6e07e9a334d9632758c9b0c054afba477fcbe2d \
--hash=sha256:5dd969a01c738104f707fa0e306b757f5beb3ebdcd682fe0724170a0bf1c21fb \
--hash=sha256:64a733886d79e726899b9d9643813e48a2eec49f3ef0fdb8cd4b8152046101c3 \
--hash=sha256:6f16289e2af54affd591dd78ed12b5465e4dc5823f818beaeddd49a010cf3ba7 \
--hash=sha256:75693208b492a026b926edeebbae888cc644433bee4993573ead2dc44810b519 \
--hash=sha256:7c3d8d34d01673d1a917445c9437ee73a9d48bc6af10352b84bbd46c5da93ca5 \
--hash=sha256:81ed1b69d6f8acd7e759a71197ce3a69da4b7e9faa9dbb005eb06a83b1a4e52d \
--hash=sha256:a766ba6e19b198eff09be05f24cd89eda1670ed404ae828e2aa3fc09816ba8d8 \
--hash=sha256:a8687b71ac67a0b3f8ce87df9e8024edf087e94c34ef46eaaad694dce8d2f83f \
--hash=sha256:d498363746555dccffc02a47dfe1dee70f7784f3f37f1d66b408366c5d3a989e \
--hash=sha256:eed2079dca3ca948d7b0d7120396e776bbc6680637cf199d393e157fde25c937 \
--hash=sha256:f233dc3b12fb641b36724844754b6bb41349615a0e258087560968d6da92add5
safetensors==0.6.2; python_version >= '3.9' \
--hash=sha256:1d2d2b3ce1e2509c68932ca03ab8f20570920cd9754b05063d4368ee52833ecd \
--hash=sha256:43ff2aa0e6fa2dc3ea5524ac7ad93a9839256b8703761e76e2d0b2a3fa4f15d9 \
--hash=sha256:8045db2c872db8f4cbe3faa0495932d89c38c899c603f21e9b6486951a5ecb8f \
--hash=sha256:81e67e8bab9878bb568cffbc5f5e655adb38d2418351dc0859ccac158f753e19 \
--hash=sha256:89a89b505f335640f9120fac65ddeb83e40f1fd081cb8ed88b505bdccec8d0a1 \
--hash=sha256:93de35a18f46b0f5a6a1f9e26d91b442094f2df02e9fd7acf224cfec4238821a \
--hash=sha256:9c85ede8ec58f120bad982ec47746981e210492a6db876882aa021446af8ffba \
--hash=sha256:b0e4d029ab0a0e0e4fdf142b194514695b1d7d3735503ba700cf36d0fc7136ce \
--hash=sha256:c7b214870df923cbc1593c3faee16bec59ea462758699bd3fee399d00aac072c \
--hash=sha256:cab75ca7c064d3911411461151cb69380c9225798a20e712b102edda2542ddb1 \
--hash=sha256:d6675cf4b39c98dbd7d940598028f3742e0375a6b4d4277e76beb0c35f4b843b \
--hash=sha256:d83c20c12c2d2f465997c51b7ecb00e407e5f94d7dec3ea0cc11d86f60d3fde5 \
--hash=sha256:d944cea65fad0ead848b6ec2c37cc0b197194bec228f8020054742190e9312ac \
--hash=sha256:fa48268185c52bfe8771e46325a1e21d317207bcabcb72e65c6e28e9ffeb29c7 \
--hash=sha256:fc4d0d0b937e04bdf2ae6f70cd3ad51328635fe0e6214aa1fc811f3b576b3bda
scikit-learn==1.7.1; python_version >= '3.10' \
--hash=sha256:10679f7f125fe7ecd5fad37dd1aa2daae7e3ad8df7f3eefa08901b8254b3e12c \
--hash=sha256:13679981fdaebc10cc4c13c43344416a86fcbc61449cb3e6517e1df9d12c8309 \
Expand Down
124 changes: 117 additions & 7 deletions runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,40 @@ WORKDIR /opt/app-root/bin
# OS Packages needs to be installed as root
USER 0

# Install useful OS packages
RUN dnf install -y mesa-libGL skopeo && dnf clean all && rm -rf /var/cache/yum
ARG TARGETARCH

# Install useful OS packages (and dev tools for s390x only)
RUN --mount=type=cache,target=/var/cache/dnf \
echo "Building for architecture: ${TARGETARCH}" && \
if [ "$TARGETARCH" = "s390x" ]; then \
PACKAGES="mesa-libGL skopeo gcc gcc-c++ make openssl-devel autoconf automake libtool cmake python3-devel pybind11-devel openblas-devel unixODBC-devel openssl zlib-devel"; \
else \
PACKAGES="mesa-libGL skopeo"; \
fi && \
echo "Installing: $PACKAGES" && \
dnf install -y --nogpgcheck --allowerasing --nobest $PACKAGES && \
dnf clean all && rm -rf /var/cache/yum

# Install Rust and set environment variables (s390x only)
RUN if [ "$TARGETARCH" = "s390x" ]; then \
mkdir -p /opt/.cargo && \
export HOME=/root && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o rustup-init.sh && \
chmod +x rustup-init.sh && \
CARGO_HOME=/opt/.cargo ./rustup-init.sh -y --no-modify-path && \
rm -f rustup-init.sh && \
chown -R 1001:0 /opt/.cargo && \
echo 'export PATH=/opt/.cargo/bin:$PATH' >> /etc/profile.d/cargo.sh && \
echo 'export CARGO_HOME=/opt/.cargo' >> /etc/profile.d/cargo.sh && \
echo 'export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1' >> /etc/profile.d/cargo.sh; \
fi
Comment on lines +25 to +37
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Execute remote scripts only with explicit checksums
rustup-init.sh is downloaded and executed without verifying a checksum or signature. This is a classic supply-chain attack vector and violates the project’s own #1241 guidance on binary verification.

-    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o rustup-init.sh && \
+    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o rustup-init.sh && \
+    echo "<SHA256  expected>  rustup-init.sh" | sha256sum -c - && \

Consider pinning to the official static install script hash (updated per release) or install the rust-toolset RPM once it appears for s390x.

🧰 Tools
🪛 GitHub Actions: Code static analysis

[warning] 26-26: Hadolint SC2016 info: Expressions don't expand in single quotes, use double quotes for that.

🤖 Prompt for AI Agents 
In runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu around lines 25 to 37,
the rustup-init.sh script is downloaded and executed without verifying its
checksum, which is a security risk. To fix this, add a step to download the
official checksum for the rustup-init.sh script and verify it matches the
downloaded file before execution. Alternatively, consider installing Rust via a
trusted package like the rust-toolset RPM for s390x once available, to avoid
executing remote scripts directly.


# Set python alternatives for s390x only
RUN if [ "$TARGETARCH" = "s390x" ]; then \
alternatives --install /usr/bin/python python /usr/bin/python3.11 1 && \
alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 && \
python --version && python3 --version; \
fi

# Other apps and tools installed as default user
USER 1001
Expand All @@ -25,11 +57,62 @@ RUN curl -L https://mirror.openshift.com/pub/openshift-v4/$(uname -m)/clients/oc
rm -f /tmp/openshift-client-linux.tar.gz
# Install the oc client end

##############################
# wheel-builder stage #
##############################
FROM base AS s390x-builder

USER 0
WORKDIR /tmp/build-wheels
ARG TARGETARCH

RUN echo "s390x-builder stage TARGETARCH: ${TARGETARCH}"

RUN --mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/root/.cache/dnf \
if [ "$TARGETARCH" = "s390x" ]; then \
echo "Building pyarrow wheel for s390x..." && \
dnf install -y cmake make gcc-c++ pybind11-devel wget && \
dnf clean all && \
git clone --depth 1 https://github.com/apache/arrow.git && \
cd arrow/cpp && \
mkdir release && cd release && \
Comment on lines +75 to +79
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Add git to build-deps to prevent clone failure.

-        dnf install -y cmake make gcc-c++ pybind11-devel wget && \
+        dnf install -y cmake make gcc-c++ git pybind11-devel wget && \
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
dnf install -y cmake make gcc-c++ pybind11-devel wget && \
dnf clean all && \
git clone --depth 1 https://github.com/apache/arrow.git && \
cd arrow/cpp && \
mkdir release && cd release && \
dnf install -y cmake make gcc-c++ git pybind11-devel wget && \
dnf clean all && \
git clone --depth 1 https://github.com/apache/arrow.git && \
cd arrow/cpp && \
mkdir release && cd release && \
🤖 Prompt for AI Agents 
In runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu around lines 74 to 78,
the build dependencies do not include git, which can cause the git clone command
to fail. Add git to the list of packages installed by dnf before running the git
clone command to ensure git is available during the build process.

cmake -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DARROW_PYTHON=ON \
-DARROW_PARQUET=ON \
-DARROW_ORC=ON \
-DARROW_FILESYSTEM=ON \
-DARROW_JSON=ON \
-DARROW_CSV=ON \
-DARROW_DATASET=ON \
-DARROW_DEPENDENCY_SOURCE=BUNDLED \
-DCMAKE_CXX_FLAGS="-O3 -march=z14 -mtune=z14" \
-DCMAKE_C_FLAGS="-O3 -march=z14 -mtune=z14" \
.. && \
make -j$(nproc) && \
make install && \
cd ../../python && \
pip install --no-cache-dir -U pip wheel setuptools && \
pip install --no-cache-dir -r requirements-build.txt && \
export PYARROW_PARALLEL=$(nproc) && \
export ARROW_BUILD_TYPE=release && \
CFLAGS="-O3 -march=z14 -mtune=z14" \
CXXFLAGS="-O3 -march=z14 -mtune=z14" \
python setup.py build_ext --build-type=release --bundle-arrow-cpp bdist_wheel && \
mkdir -p /tmp/wheels && \
cp dist/pyarrow-*.whl /tmp/wheels/ && \
ls -la /tmp/wheels/; \
else \
echo "Not s390x, skipping wheel build" && mkdir -p /tmp/wheels; \
fi

Comment on lines +71 to +109
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Unpinned Arrow git clone hurts reproducibility
git clone --depth 1 https://github.com/apache/arrow.git fetches whatever HEAD happens to be at build time, making builds non-reproducible and brittle.

-        git clone --depth 1 https://github.com/apache/arrow.git && \
+        ARROW_TAG=apache-arrow-16.1.0 && \
+        git clone --branch ${ARROW_TAG} --depth 1 https://github.com/apache/arrow.git && \

Pin to a known tag/revision and add a comment when updating to a newer release.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
RUN --mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/root/.cache/dnf \
if [ "$TARGETARCH" = "s390x" ]; then \
echo "Building pyarrow wheel for s390x..." && \
dnf install -y cmake make gcc-c++ pybind11-devel wget && \
dnf clean all && \
git clone --depth 1 https://github.com/apache/arrow.git && \
cd arrow/cpp && \
mkdir release && cd release && \
cmake -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DARROW_PYTHON=ON \
-DARROW_PARQUET=ON \
-DARROW_ORC=ON \
-DARROW_FILESYSTEM=ON \
-DARROW_JSON=ON \
-DARROW_CSV=ON \
-DARROW_DATASET=ON \
-DARROW_DEPENDENCY_SOURCE=BUNDLED \
-DCMAKE_CXX_FLAGS="-O3 -march=z14 -mtune=z14" \
-DCMAKE_C_FLAGS="-O3 -march=z14 -mtune=z14" \
.. && \
make -j$(nproc) && \
make install && \
cd ../../python && \
pip install --no-cache-dir -U pip wheel setuptools && \
pip install --no-cache-dir -r requirements-build.txt && \
export PYARROW_PARALLEL=$(nproc) && \
export ARROW_BUILD_TYPE=release && \
CFLAGS="-O3 -march=z14 -mtune=z14" \
CXXFLAGS="-O3 -march=z14 -mtune=z14" \
python setup.py build_ext --build-type=release --bundle-arrow-cpp bdist_wheel && \
mkdir -p /tmp/wheels && \
cp dist/pyarrow-*.whl /tmp/wheels/ && \
ls -la /tmp/wheels/; \
else \
echo "Not s390x, skipping wheel build" && mkdir -p /tmp/wheels; \
fi
dnf clean all && \
- git clone --depth 1 https://github.com/apache/arrow.git && \
+ ARROW_TAG=apache-arrow-16.1.0 && \
+ git clone --branch ${ARROW_TAG} --depth 1 https://github.com/apache/arrow.git && \
cd arrow/cpp && \
mkdir release && cd release && \
cmake -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DARROW_PYTHON=ON \
-DARROW_PARQUET=ON \
-DARROW_ORC=ON \
-DARROW_FILESYSTEM=ON \
-DARROW_JSON=ON \
-DARROW_CSV=ON \
-DARROW_DATASET=ON \
-DARROW_DEPENDENCY_SOURCE=BUNDLED \
-DCMAKE_CXX_FLAGS="-O3 -march=z14 -mtune=z14" \
-DCMAKE_C_FLAGS="-O3 -march=z14 -mtune=z14" \
.. && \
make -j$(nproc) && \
make install && \
cd ../../python && \
pip install --no-cache-dir -U pip wheel setuptools && \
pip install --no-cache-dir -r requirements-build.txt && \
export PYARROW_PARALLEL=$(nproc) && \
export ARROW_BUILD_TYPE=release && \
CFLAGS="-O3 -march=z14 -mtune=z14" \
CXXFLAGS="-O3 -march=z14 -mtune=z14" \
python setup.py build_ext --build-type=release --bundle-arrow-cpp bdist_wheel && \
mkdir -p /tmp/wheels && \
cp dist/pyarrow-*.whl /tmp/wheels/ && \
ls -la /tmp/wheels/; \
🤖 Prompt for AI Agents 
In runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu around lines 69 to 107,
the git clone command for Apache Arrow uses the default branch HEAD, which makes
builds non-reproducible and brittle. To fix this, pin the clone to a specific
known tag or commit hash by adding a checkout command after cloning or cloning
directly with a tag reference. Also, add a comment explaining the pinned version
and instructions for updating it to maintain clarity and reproducibility.

#######################
# runtime-datascience #
#######################
FROM base AS runtime-datascience

ARG TARGETARCH
ARG DATASCIENCE_SOURCE_CODE=runtimes/datascience/ubi9-python-3.11

LABEL name="odh-notebook-runtime-datascience-ubi9-python-3.11" \
Expand All @@ -43,17 +126,44 @@ LABEL name="odh-notebook-runtime-datascience-ubi9-python-3.11" \
io.openshift.build.image="quay.io/opendatahub/workbench-images:runtime-datascience-ubi9-python-3.11"

WORKDIR /opt/app-root/bin
USER 0

# Install s390x-built wheels if available
COPY --from=s390x-builder /tmp/wheels /tmp/wheels
RUN if [ "$TARGETARCH" = "s390x" ]; then \
echo "Installing s390x wheels..." && \
WHEELS=$(find /tmp/wheels/ -name "pyarrow-*.whl") && \
if [ -n "$WHEELS" ]; then \
pip install --no-cache-dir $WHEELS && \
echo "Wheel install complete"; \
else \
echo "No pyarrow wheel found!" && exit 1; \
fi && rm -rf /tmp/wheels; \
else \
echo "Skipping wheel install on non-s390x (${TARGETARCH})"; \
fi

# Install Python packages from requirements.txt
COPY ${DATASCIENCE_SOURCE_CODE}/requirements.txt ./
# Copy Elyra dependencies for air-gapped enviroment
COPY ${DATASCIENCE_SOURCE_CODE}/utils ./utils/

RUN echo "Installing softwares and packages" && \
# This may have to download and compile some dependencies, and as we don't lock requirements from `build-system.requires`,
# we often don't know the correct hashes and `--require-hashes` would therefore fail on non amd64, where building is common.
uv pip install --strict --no-deps --no-cache --no-config --no-progress --verify-hashes --compile-bytecode --index-strategy=unsafe-best-match --requirements=./requirements.txt --build-constraints=./requirements.txt && \
# Fix permissions to support pip in Openshift environments \
RUN --mount=type=cache,target=/root/.cache/pip \
echo "Installing softwares and packages" && \
if [ "$TARGETARCH" = "s390x" ]; then \
# For s390x, we need special flags and environment variables for building packages
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 \
CFLAGS="-O3" CXXFLAGS="-O3" \
uv pip install --strict --no-deps --no-cache --no-config --no-progress \
--verify-hashes --compile-bytecode --index-strategy=unsafe-best-match \
--requirements=./requirements.txt --build-constraints=./requirements.txt; \
else \
# This may have to download and compile some dependencies, and as we don't lock requirements from `build-system.requires`,
# we often don't know the correct hashes and `--require-hashes` would therefore fail on non amd64, where building is common.
uv pip install --strict --no-deps --no-cache --no-config --no-progress \
--verify-hashes --compile-bytecode --index-strategy=unsafe-best-match \
--requirements=./requirements.txt --build-constraints=./requirements.txt; \
fi && \
chmod -R g+w /opt/app-root/lib/python3.11/site-packages && \
fix-permissions /opt/app-root -P

Expand Down
7 changes: 6 additions & 1 deletion runtimes/datascience/ubi9-python-3.11/Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,12 @@ scikit-learn = "~=1.6.1"
scipy = "~=1.15.2"
skl2onnx = "~=1.18.0"
onnxconverter-common = "~=1.13.0" # Required for skl2onnx, as upgraded version is not compatible with protobuf
codeflare-sdk = "~=0.30.0"

# Exclude these packages on s390x architecture due to compatibility
codeflare-sdk = {version = "~=0.30.0", markers = "platform_machine != 's390x'"}
py-spy = {version = "~=0.4.0", markers = "platform_machine != 's390x'"}
ray = {version = "~=2.47.1", markers = "platform_machine != 's390x'", extras = ["data", "default"]}
pyarrow = {version = "~=21.0.0", markers = "platform_machine != 's390x'"}
Comment on lines +21 to +25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Version coherence between architectures may diverge – tag the Arrow checkout to match ~=21.0.0.

pyarrow is excluded for s390x in the Pipfile but is re-installed later from a wheel you build from the Arrow main branch.
That wheel is almost certainly not 21.0.0 (today main is already 16+). The result is:

  • x86_64/arm64 → pyarrow==21.0.0 (from lock)
  • s390x → pyarrow==<current-HEAD> (from wheel)

Down-stream notebooks that rely on consistent feature / ABI levels can now break only on s390x.

Recommend pinning the build to the same version tag that the Pipfile declares, e.g.:

- git clone --depth 1 https://github.com/apache/arrow.git
+ git clone --depth 1 --branch apache-arrow-21.0.0 https://github.com/apache/arrow.git

and keep the Pipfile/lock in sync whenever you upgrade.

🏁 Script executed:

#!/bin/bash
set -e

echo "1) Listing Dockerfiles under runtimes/datascience/ubi9-python-3.11:"
fd -t f Dockerfile runtimes/datascience/ubi9-python-3.11 || echo "No Dockerfile found in that path"

echo
echo "2) Searching for 'arrow' references in those Dockerfiles:"
for df in $(fd -t f Dockerfile runtimes/datascience/ubi9-python-3.11); do
  echo "---- $df ----"
  grep -n "arrow" "$df" || echo "  (no 'arrow' references)"
done

echo
echo "3) Searching entire repo for 'git clone' commands cloning Arrow:"
rg "git clone.*arrow" -n . || echo "No git clone arrow found in repo"

Length of output: 1602

Pin Arrow checkout to version 21.0.0 in the s390x build

The s390x Docker build (runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu:76) currently does:

git clone --depth 1 https://github.com/apache/arrow.git

This pulls main (now ≫21.0.0), so:

  • x86_64/arm64 → pyarrow==21.0.0 (from Pipfile.lock)
  • s390x → pyarrow== (built wheel)

downstream notebooks expecting a consistent ABI/featureset may break only on s390x.

Please update the clone to pin the tag matching your Pipfile:

- git clone --depth 1 https://github.com/apache/arrow.git
+ git clone --depth 1 --branch apache-arrow-21.0.0 https://github.com/apache/arrow.git

• Apply the same change in the 3.12 Dockerfile (runtimes/datascience/ubi9-python-3.12/Dockerfile.cpu:80) when bumping pyarrow.
• Keep the Pipfile/lock and Dockerfile tag in sync on future upgrades.

🤖 Prompt for AI Agents 
In runtimes/datascience/ubi9-python-3.11/Pipfile lines 21 to 25, the pyarrow
package is pinned to version 21.0.0, but the s390x Docker build clones the
Apache Arrow repository without specifying a tag, causing it to use the latest
main branch instead of version 21.0.0. To fix this, update the s390x Dockerfile
(runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu line 76) to clone the
Apache Arrow repository at the tag corresponding to version 21.0.0, ensuring
consistency between the Pipfile and the built wheel. Also, apply the same change
to the 3.12 Dockerfile when upgrading pyarrow in the future.


# DB connectors
pymongo = "~=4.11.2"
Expand Down
Loading
Loading