Skip to content

Test : Enable s390x support for runtime datascience 3.11/3.12 #1355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: rhoai-2.23-multi-arch-poc
Choose a base branch
from

Conversation

Nash-123
Copy link

@Nash-123 Nash-123 commented Aug 5, 2025

Summary

This PR introduces s390x (IBM Z) support and build optimizations for the runtime-datascience notebook images for Python 3.11 and 3.12.

  • Reduced build time for s390x: ~2–2.3 hours (down from 4 hours on building natively)
  • No impact on other platforms (x86_64, arm64, etc.)
  • Elyra-compatible: validated for notebook execution

Key Changes

Dockerfile.cpu (Python 3.11 & 3.12)

  • Introduced ARG TARGETARCH for architecture-specific builds
  • Multi-stage build using s390x-builder to build pyarrow from source on IBM Z
  • Enabled cache mounts option for dnf and pip
  • Optimizations for s390x:
    • Installs Rust & Cargo
    • Uses compiler flags (CFLAGS="-O3")
    • Installs development dependencies required for pyarrow compilation
  • Package installation and build logic gated via TARGETARCH

Pipfile (Python 3.11 & 3.12)

Architecture-specific markers added to exclude incompatible packages on s390x:

codeflare-sdk = { version = "~=0.30.0", markers = "platform_machine != 's390x'" }
py-spy        = { version = "~=0.4.0", markers = "platform_machine != 's390x'" }
ray           = { version = "~=2.47.1", markers = "platform_machine != 's390x'", extras = ["data", "default"] }
pyarrow       = { version = "~=21.0.0", markers = "platform_machine != 's390x'" }

Regenerated Pipfile.locks using Renewal Action

Image seems to be building fine on Konflux:

image

@Nash-123 Nash-123 self-assigned this Aug 5, 2025
@openshift-ci openshift-ci bot requested review from atheo89 and dibryant August 5, 2025 19:42
Copy link

openshift-ci bot commented Aug 5, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andyatmiami for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Nishan Acharya <[email protected]>
@Nash-123 Nash-123 requested a review from wznoinsk August 6, 2025 04:55
@wznoinsk
Copy link

wznoinsk commented Aug 6, 2025

the s390x, x86 and arm konflux builds succeeded, the ppc64 failed:
https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/ns/rhoai-tenant/applications/external-rhoai-v2-23-multi-arch-poc/pipelineruns/odh-pipeline-runtime-ds-cpu-py311-ubi9-n-v2-23-multi-arch-ss6q6/logs?task=build-images

this code is good for s390x, it's ok to merge this code to the main branch and continue working on ppc64 in a separate pull request - or add ppc64 support to this one - up to you

@Nash-123
Copy link
Author

Nash-123 commented Aug 6, 2025

  • Build Notebooks (pr) / build (runtime-datascience-ubi9-python-3.11, 3.11, linux/s390x, false) / build (pull_request)

Hi @wznoinsk, Sure. Thanks!! . We've raised the PR for this on the upstream :- opendatahub-io#1513 . currently working with @jiridanek @atheo89 to get it merged

Once that gets merged upstream, the changes should sync downstream, and I believe we’ll be good on the s390x front.

Regarding the ppc64 arch, @npanpaliya. would you like to add here anything ?

@Nash-123 Nash-123 requested a review from jiridanek August 6, 2025 07:13
@npanpaliya
Copy link

@Nash-123 - We will add Power support in a separate PR. Earlier we had created a separate dockerfile but as per the latest discussion, we need to put back our changes in the same Dockerfile, which may take time. So, don't want to hold off your PR for that. @AaruniAggarwal will raise a new PR.

@Nash-123
Copy link
Author

Nash-123 commented Aug 6, 2025

@Nash-123 - We will add Power support in a separate PR. Earlier we had created a separate dockerfile but as per the latest discussion, we need to put back our changes in the same Dockerfile, which may take time. So, don't want to hold off your PR for that. @AaruniAggarwal will raise a new PR.

Sure, Sounds good. Thanks @npanpaliya !!
fyi: @wznoinsk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants