RHOAIENG-31406: enable Runtime Datascience Images for IBM Z[s390x] (Python 3.12) #1513

Nash-123 · 2025-07-25T21:39:35Z

Summary

This PR introduces s390x (IBM Z) support and build optimizations for the runtime-datascience notebook images for Python 3.12.

Reduced build time for s390x: ~2–2.3 hours (down from 4 hours on building natively)
No impact on other platforms (x86_64, arm64, etc.)
Elyra-compatible: validated for notebook execution

Key Changes

Dockerfile.cpu (Python 3.12)

Introduced ARG TARGETARCH for architecture-specific builds
Multi-stage build using s390x-builder to build pyarrow from source on IBM Z
Enabled cache mounts option for dnf and pip
Optimizations for s390x:
- Installs Rust & Cargo
- Uses compiler flags (CFLAGS="-O3")
- Installs development dependencies required for pyarrow compilation
Package installation and build logic gated via TARGETARCH

Pipfile (Python 3.12)

Architecture-specific markers added to exclude incompatible packages on s390x:

codeflare-sdk = { version = "~=0.30.0", markers = "platform_machine != 's390x'" }
py-spy        = { version = "~=0.4.0", markers = "platform_machine != 's390x'" }
ray           = { version = "~=2.47.1", markers = "platform_machine != 's390x'", extras = ["data", "default"] }
pyarrow       = { version = "~=21.0.0", markers = "platform_machine != 's390x'" }

Regenerated Pipfile.locks using Renewal Action

Validation & Testing

Tested on Downstream Konflux, Image seems to be building fine for Z : Test:Enabled Runtime DataScience for Z red-hat-data-services/notebooks#1512
Builds completed successfully on s390x LPAR for Python 3.12
Build time reduced to ~2 hours
Verified compatibility by deploying on an local openshift cluster on s390x lpar

Sample Notebook Validation :-

Build & Push:

Validate Runtime Image(version 3.11):

+ bin/kubectl exec runtime-runtime-pod -- /bin/sh -c 'curl https://raw.githubusercontent.com/opendatahub-io/elyra/refs/heads/main/etc/generic/requirements-elyra.txt --output req.txt && 				python3 -m pip install -r req.txt > /dev/null && 				curl https://raw.githubusercontent.com/nteract/papermill/main/papermill/tests/notebooks/simple_execute.ipynb --output simple_execute.ipynb && 				python3 -m papermill simple_execute.ipynb output.ipynb > /dev/null'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1368  100  1368    0     0   7907      0 --:--:-- --:--:-- --:--:--  7907

[notice] A new release of pip is available: 24.2 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   910  100   910    0     0   3791      0 --:--:-- --:--:-- --:--:--  3791
Input Notebook:  simple_execute.ipynb
Output Notebook: output.ipynb
Executing:   0%|          | 0/3 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing DESC: 100%|██████████| 3/3 [00:01<00:00,  2.04cell/s]
+ '[' 0 -eq 1 ']'
+ echo '=> Container image runtime-datascience-ubi9-python-3.11 is a suitable Elyra runtime image'
=> Container image runtime-datascience-ubi9-python-3.11 is a suitable Elyra runtime image

Validate Runtime Image(version 3.12):

=> Checking container image runtime-datascience-ubi9-python-3.12 for python3...
+ bin/kubectl exec runtime-pod which python3
+ '[' python3 == python3 ']'
+ echo '=> Checking notebook execution...'
=> Checking notebook execution...
+ bin/kubectl exec runtime-pod -- /bin/sh -c 'curl https://raw.githubusercontent.com/opendatahub-io/elyra/refs/heads/main/etc/generic/requirements-elyra.txt --output req.txt && 				python3 -m pip install -r req.txt > /dev/null && 				curl https://raw.githubusercontent.com/nteract/papermill/main/papermill/tests/notebooks/simple_execute.ipynb --output simple_execute.ipynb && 				python3 -m papermill simple_execute.ipynb output.ipynb > /dev/null'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1368  100  1368    0     0   6705      0 --:--:-- --:--:-- --:--:--  6738

[notice] A new release of pip is available: 24.2 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   910  100   910    0     0   3939      0 --:--:-- --:--:-- --:--:--  3922
Input Notebook:  simple_execute.ipynb
Output Notebook: output.ipynb
Executing:   0%|          | 0/3 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing DESC: 100%|██████████| 3/3 [00:01<00:00,  2.09cell/s]
+ '[' 0 -eq 1 ']'
+ echo '=> Container image runtime-datascience-ubi9-python-3.12 is a suitable Elyra runtime image'
=> Container image runtime-datascience-ubi9-python-3.12 is a suitable Elyra runtime image

c.c: @jiridanek

Summary by CodeRabbit

New Features
- s390x-aware datascience runtime: conditional toolchain steps, prebuilt pyarrow wheel support and s390x-specific Python toolchain handling.
- Architecture- and Python-version gating to avoid installing incompatible packages on s390x.
Chores
- CI/pipeline matrix extended to include s390x for relevant datascience runtime builds.

openshift-ci · 2025-07-25T21:39:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andyatmiami for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2025-07-25T21:39:44Z

Walkthrough

Adds TARGETARCH-driven architecture branching to the UBI9 Python 3.12 datascience Dockerfile (introducing an s390x build stage, s390x-specific toolchain/install logic, and conditional wheel handling), expands CI s390x-compatible targets, and adds platform/python-version markers to pyproject/pylock to gate many packages for s390x.

Changes

Cohort / File(s)	Summary
Architecture-aware Docker build & s390x wheel stage `runtimes/datascience/ubi9-python-3.12/Dockerfile.cpu`	Adds `ARG TARGETARCH` and `ARG DATASCIENCE_SOURCE_CODE`; prints TARGETARCH; conditions package installs, Rust and Python toolchain setup, and pip/build flags for `s390x`; introduces `s390x-builder` stage to build a `pyarrow` wheel into `/tmp/wheels`; runtime stage copies/installs wheels conditionally and uses cache-backed pip installs; applies permission fixes.
CI matrix s390x enablement `ci/cached-builds/gen_gha_matrix_jobs.py`	Adds `"runtime-datascience-ubi9-python-3.11"` and `"runtime-datascience-ubi9-python-3.12"` to the `S390X_COMPATIBLE` set to include these targets for s390x builds.
Dependency selector change (pyproject) `runtimes/datascience/ubi9-python-3.12/pyproject.toml`	Adds platform selector to `codeflare-sdk` dependency to exclude it on `s390x` (`platform_machine != "s390x"`).
Lockfile: platform & Python-version markers `runtimes/datascience/ubi9-python-3.12/pylock.toml`	Adds/updates many `marker` entries (using `platform_machine` and `python_full_version`) to gate package distributions/wheels for `s390x` and specific Python versions; package versions/URLs largely unchanged.
Pipeline & PR generation changes `.tekton/odh-pipeline-runtime-datascience-cpu-py312-ubi9-pull-request.yaml`, `scripts/generate_pull_request_pipelineruns.py`	Adds `linux/s390x` to pipeline run build-platforms; refactors PR pipeline generation to extend build_platforms per-component (adds archs such as linux/s390x, linux/ppc64le, linux/arm64 for specific components).

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

Refactor: Use Docker build args for architecture detection in minimal Dockerfile.cpu #1160 — Dockerfile changes introduce and use ARG TARGETARCH instead of uname-based detection and add arch-specific installs/stages, which matches the issue's scope.

✨ Finishing Touches

📝 Generate Docstrings

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

Visit our Status Page to check the current availability of CodeRabbit.
Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

openshift-ci · 2025-07-25T21:39:48Z

Hi @Nash-123. Thanks for your PR.

I'm waiting for a opendatahub-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

runtimes/datascience/ubi9-python-3.12/requirements.txt

runtimes/datascience/ubi9-python-3.12/Dockerfile.cpu

coderabbitai

Actionable comments posted: 13

🔭 Outside diff range comments (1)

runtimes/datascience/ubi9-python-3.12/requirements.txt (1)
438-465: Upgrade cryptography to ≥ 44.0.1 – CVE-2024-12797
Trivy flags 43.0.3 as vulnerable (RFC7250 handshake bypass). The fix is available in 44.0.1.
-cryptography==43.0.3; platform_machine != 's390x' \
+cryptography==44.0.1; platform_machine != 's390x' \

🧹 Nitpick comments (6)

runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu (2)

26-37: Rust bootstrap script executed without checksum or sig-verification.

curl https://sh.rustup.rs | sh is convenient but opaque. To harden the image:

Download the script and verify SHA256 (Rust publishes checksums).

Pin to a known version with RUSTUP_INIT env or --default-toolchain flag.

At minimum, store the expected digest in the Dockerfile and fail the build on mismatch.

130-143: Wheel copy step fails the build for non-s390x runs.

COPY --from=s390x-builder /tmp/wheels /tmp/wheels copies even when the directory is empty (non-s390x build).
That is fine, but the subsequent find will echo “No pyarrow wheel found!” only when $WHEELS is empty inside the if branch. Because the if branch runs only on s390x, non-s390x builds silently keep the empty directory around.

Add rm -rf /tmp/wheels also in the non-s390x path to avoid leaving junk layers.

jupyter/trustyai/ubi9-python-3.12/requirements.txt (1)

918-920: Confirm huggingface-hub 0.34.1 doesn’t outrun pinned transformers
transformers==4.52.4 currently requires huggingface-hub<1.0. The new 0.34.x branch is still compatible, but keep an eye on future minor bumps—to 1.0 the constraint will tighten.
runtimes/datascience/ubi9-python-3.12/Dockerfile.cpu (3)
26-40: Persist Rust tooling in PATH via ENV, not login-shell profiles.

/etc/profile.d/cargo.sh is sourced only for login shells, so non-interactive processes (e.g. pip building wheels later in the Dockerfile) won’t see /opt/.cargo/bin.
Add an explicit ENV so every layer inherits the correct PATH/CARGO_HOME.
 RUN if [ "$TARGETARCH" = "s390x" ]; then \
@@
     echo 'export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1' >> /etc/profile.d/cargo.sh; \
 fi
+# Ensure non-login shells see Rust binaries
+ENV PATH=/opt/.cargo/bin:$PATH \
+    CARGO_HOME=/opt/.cargo \
+    GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
52-55: pip cache mount points should be writable by the current user.

The cache target is /root/.cache/pip while the active user is 1001; this will be owned by root and may become read-only for the non-root user, defeating the cache and occasionally failing writes.
Consider either switching back to USER 0 just for this instruction or changing the mount target to $HOME/.cache/pip.

153-160: Consider running micropipenv install as non-root to avoid FS permission fixes.

Because USER 0 is still active here, the subsequent chmod -R g+w dance is required.
If you adopt the previous suggestion to reset to USER 1001 before this RUN step, you can drop the manual permission fix and reduce layer size/time.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9d2bac1 and 8053c7e.

⛔ Files ignored due to path filters (12)

codeserver/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
jupyter/datascience/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
jupyter/minimal/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
jupyter/pytorch/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
jupyter/rocm/pytorch/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
jupyter/tensorflow/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
jupyter/trustyai/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
runtimes/datascience/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
runtimes/minimal/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
runtimes/pytorch/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
runtimes/rocm-pytorch/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock
runtimes/tensorflow/ubi9-python-3.12/Pipfile.lock is excluded by !**/*.lock

📒 Files selected for processing (16)

codeserver/ubi9-python-3.12/requirements.txt (2 hunks)
jupyter/datascience/ubi9-python-3.12/requirements.txt (4 hunks)
jupyter/minimal/ubi9-python-3.12/requirements.txt (2 hunks)
jupyter/pytorch/ubi9-python-3.12/requirements.txt (4 hunks)
jupyter/rocm/pytorch/ubi9-python-3.12/requirements.txt (4 hunks)
jupyter/tensorflow/ubi9-python-3.12/requirements.txt (5 hunks)
jupyter/trustyai/ubi9-python-3.12/requirements.txt (4 hunks)
runtimes/datascience/ubi9-python-3.11/Dockerfile.cpu (2 hunks)
runtimes/datascience/ubi9-python-3.11/Pipfile (1 hunks)
runtimes/datascience/ubi9-python-3.12/Dockerfile.cpu (2 hunks)
runtimes/datascience/ubi9-python-3.12/Pipfile (1 hunks)
runtimes/datascience/ubi9-python-3.12/requirements.txt (24 hunks)
runtimes/minimal/ubi9-python-3.12/requirements.txt (1 hunks)
runtimes/pytorch/ubi9-python-3.12/requirements.txt (3 hunks)
runtimes/rocm-pytorch/ubi9-python-3.12/requirements.txt (3 hunks)
runtimes/tensorflow/ubi9-python-3.12/requirements.txt (4 hunks)

🧰 Additional context used

🧠 Learnings (17)

📓 Common learnings

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1333
File: runtimes/rocm-tensorflow/ubi9-python-3.12/Dockerfile.rocm:50-50
Timestamp: 2025-07-08T19:30:01.738Z
Learning: jiridanek requested GitHub issue creation for multi-architecture support in ROCm TensorFlow image during PR #1333 review. Issue #1346 was created with comprehensive problem description covering hardcoded x86_64 architecture breaking multi-arch support, detailed impact analysis, three solution options (runtime detection, BuildKit TARGETARCH integration, hybrid approach) with pros/cons analysis, comprehensive acceptance criteria covering core requirements and testing, phased implementation guidance, related files identification, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1320
File: jupyter/pytorch/ubi9-python-3.12/Dockerfile.cuda:65-66
Timestamp: 2025-07-09T12:31:02.033Z
Learning: jiridanek requested GitHub issue creation for MSSQL repo file hardcoding problem during PR #1320 review. Issue #1363 was created and updated with comprehensive problem description covering hardcoded x86_64 MSSQL repo files breaking multi-architecture builds across 10 affected Dockerfiles (including datascience, CUDA, ROCm, and TrustyAI variants), detailed root cause analysis, three solution options with code examples, clear acceptance criteria for all image types, implementation guidance following established multi-architecture patterns, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1333
File: runtimes/tensorflow/ubi9-python-3.12/Dockerfile.cuda:38-38
Timestamp: 2025-07-08T19:30:20.513Z
Learning: jiridanek requested GitHub issue creation for multi-architecture support in TensorFlow CUDA runtime image during PR #1333 review. Issue was created with comprehensive problem description covering hardcoded NVARCH limitation, multiple solution options using TARGETARCH build argument with architecture mapping, acceptance criteria for multi-architecture builds, implementation guidance with code examples, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: atheo89
PR: opendatahub-io/notebooks#1258
File: codeserver/ubi9-python-3.11/Dockerfile.cpu:32-32
Timestamp: 2025-07-07T11:08:48.524Z
Learning: atheo89 requested GitHub issue creation for multi-architecture Dockerfile improvements during PR #1258 review, specifically for enhancing structural consistency across Docker stages, replacing $(uname -m) with ${TARGETARCH} for cross-architecture builds, and adding OCI-compliant metadata labels. Issue #1332 was created with comprehensive problem description, phased implementation approach, detailed acceptance criteria, implementation guidance with code examples, and proper context linking, continuing the established pattern of systematic code quality improvements.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1306
File: jupyter/trustyai/ubi9-python-3.12/kustomize/base/kustomization.yaml:8-12
Timestamp: 2025-07-08T19:09:48.746Z
Learning: jiridanek requested GitHub issue creation for misleading CUDA prefix in TrustyAI image tags during PR #1306 review, affecting both Python 3.11 and 3.12 versions. Issue #1338 was created with comprehensive problem description covering both affected images, repository pattern analysis comparing correct vs incorrect naming conventions, clear solution with code examples, detailed acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1320
File: jupyter/pytorch/ubi9-python-3.12/Dockerfile.cuda:42-52
Timestamp: 2025-07-09T12:29:56.162Z
Learning: jiridanek requested GitHub issue creation for OpenShift client architecture mapping problem affecting 29 Dockerfiles during PR #1320 review. Issue was created with comprehensive analysis covering all affected files using $(uname -m) returning 'aarch64' but OpenShift mirror expecting 'arm64', systematic solution using BuildKit TARGETARCH mapping with proper amd64→x86_64 and arm64→arm64 conversion, detailed acceptance criteria, and implementation guidance, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1333
File: runtimes/pytorch/ubi9-python-3.12/utils/bootstrapper.py:619-626
Timestamp: 2025-07-08T19:33:14.340Z
Learning: jiridanek requested GitHub issue creation for Python 3.12 version check bug in bootstrapper.py during PR #1333 review. Issue #1348 was created with comprehensive problem description covering version check exclusion affecting all Python 3.12 runtime images, detailed impact analysis of bootstrapper execution failures, clear solution with code examples, affected files list including all 6 runtime bootstrapper copies, acceptance criteria for testing and verification, implementation notes about code duplication and upstream reporting, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1320
File: scripts/buildinputs/dockerfile.go:35-35
Timestamp: 2025-07-10T15:02:13.228Z
Learning: jiridanek requested GitHub issue creation for improving architecture detection in buildinputs tool during PR #1320 review. The automated issue creation failed despite appearing successful, so comprehensive issue content was provided for manual creation covering qemu-user s390x build failures where runtime.GOARCH returns host architecture instead of target architecture, detailed impact analysis, four solution options (environment variable detection, configuration parameter, platform detection logic, hybrid approach), clear acceptance criteria for multi-architecture support, phased implementation guidance, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1306
File: jupyter/trustyai/ubi9-python-3.12/kustomize/base/kustomization.yaml:8-12
Timestamp: 2025-07-08T19:09:48.746Z
Learning: jiridanek requested GitHub issue creation for misleading CUDA prefix in TrustyAI image tags during PR #1306 review. Issue was created with comprehensive problem description covering both Python 3.11 and 3.12 versions, repository pattern analysis showing correct vs incorrect naming, clear solution with code examples, detailed acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1333
File: runtimes/rocm-tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-09T08:07:24.937Z
Learning: jiridanek requested GitHub issue creation for tensorflow_rocm Python 3.12 compatibility problem during PR #1333 review. Issue #1354 was successfully created with comprehensive problem description covering missing cp312 wheels causing build failures, three solution options (upstream TensorFlow, Python 3.11 only, custom build), clear acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1333
File: runtimes/rocm-pytorch/ubi9-python-3.12/de-vendor-torch.sh:9-12
Timestamp: 2025-07-08T19:28:15.791Z
Learning: jiridanek requested GitHub issue creation for runtime detection improvement of Python site-packages paths during PR #1333 review. Issue #1344 was created with comprehensive problem description covering hard-coded path fragility across UBI9 distributions, extensive affected files analysis including 4 de-vendor scripts, 30+ Dockerfiles with chmod operations, and 12+ pip.conf configurations, runtime detection solution using sysconfig module, implementation areas breakdown, benefits analysis, clear acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1396
File: runtimes/tensorflow/ubi9-python-3.12/Dockerfile.cuda:124-127
Timestamp: 2025-07-20T20:47:36.509Z
Learning: jiridanek identified that ARM64 wheels for h5py 3.14.0 are available on PyPI but being ignored due to AMD64-only dependency locking with --platform=linux/amd64. This causes unnecessary hdf5-devel package installation in ARM64 TensorFlow images when the ARM64 wheel h5py-3.14.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl could be used instead. The Pipfile.lock only contains 2 hashes for h5py, confirming limited platform consideration during lock generation.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-01T10:41:56.419Z
Learning: jiridanek's team uses containerized dependency locking for cross-platform compatibility in opendatahub-io/notebooks. They run `pipenv lock` inside UBI9 containers with specific platform arguments (`--platform=linux/amd64 --python-version 3.12`) to avoid host OS dependency conflicts when generating Pipfile.lock files.

Learnt from: grdryn
PR: opendatahub-io/notebooks#1396
File: jupyter/tensorflow/ubi9-python-3.12/Pipfile:13-14
Timestamp: 2025-07-16T00:17:10.313Z
Learning: When checking PyPI wheel availability for aarch64 architecture, the actual wheel filenames use "manylinux2014_aarch64" pattern, not "linux_aarch64". This caused incorrect analysis in PR #1396 where CodeRabbit falsely claimed CUDA companion packages lacked aarch64 wheels when searching with the wrong pattern. The correct approach is to search for "manylinux2014_aarch64" or use a more generic "aarch64" pattern when verifying wheel availability for ARM64 compatibility.

Learnt from: grdryn
PR: opendatahub-io/notebooks#1396
File: jupyter/tensorflow/ubi9-python-3.12/Pipfile:13-14
Timestamp: 2025-07-16T00:17:10.313Z
Learning: grdryn corrected CodeRabbit's false assessment about CUDA companion package wheel availability during PR #1396 review. The original analysis incorrectly checked all package releases instead of the specific versions that would be installed with tensorflow[and-cuda]~=2.19.0. The actual versions (nvidia-cudnn-cu12/9.3.0.75, nvidia-cuda-runtime-cu12/12.5.82, nvidia-cublas-cu12/12.5.3.2) do have aarch64 wheels available on PyPI, making the and-cuda extra compatible with arm64 builds.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1333
File: runtimes/rocm-pytorch/ubi9-python-3.12/de-vendor-torch.sh:9-12
Timestamp: 2025-07-08T19:28:15.791Z
Learning: jiridanek requested GitHub issue creation for runtime detection improvement of Python site-packages paths during PR #1333 review. Issue #1340 was created with comprehensive problem description covering hard-coded path fragility across UBI9 distributions, affected files analysis including de-vendor scripts and pip.conf configurations, runtime detection solution using sysconfig module, implementation areas breakdown, benefits analysis, clear acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-03T07:03:45.020Z
Learning: The Python 3.11 infrastructure for ROCm TensorFlow images in opendatahub-io/notebooks is already properly configured in the Makefile with both BASE_DIRS entries for Pipfile lock renewals and all-images targets for CI builds, requiring only commenting out the corresponding Python 3.12 entries when downgrading due to wheel availability issues.

runtimes/minimal/ubi9-python-3.12/requirements.txt (3)

Learnt from: jiridanek
PR: #1333
File: runtimes/rocm-pytorch/ubi9-python-3.12/de-vendor-torch.sh:9-12
Timestamp: 2025-07-08T19:28:15.791Z
Learning: jiridanek requested GitHub issue creation for runtime detection improvement of Python site-packages paths during PR #1333 review. Issue #1344 was created with comprehensive problem description covering hard-coded path fragility across UBI9 distributions, extensive affected files analysis including 4 de-vendor scripts, 30+ Dockerfiles with chmod operations, and 12+ pip.conf configurations, runtime detection solution using sysconfig module, implementation areas breakdown, benefits analysis, clear acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1333
File: runtimes/rocm-pytorch/ubi9-python-3.12/de-vendor-torch.sh:9-12
Timestamp: 2025-07-08T19:28:15.791Z
Learning: jiridanek requested GitHub issue creation for runtime detection improvement of Python site-packages paths during PR #1333 review. Issue #1340 was created with comprehensive problem description covering hard-coded path fragility across UBI9 distributions, affected files analysis including de-vendor scripts and pip.conf configurations, runtime detection solution using sysconfig module, implementation areas breakdown, benefits analysis, clear acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-24T12:01:45.188Z
Learning: jiridanek requested verification of 2024.2 tag package versions against corresponding Pipfiles during PR #1496 review. The analysis revealed that 2024.2 tags contain intentionally frozen older versions (N-1 behavior) while the current PR correctly updates only the N (latest) tags to match Python 3.11 Pipfiles, demonstrating proper understanding of the repository's N-1 version freezing policy.

runtimes/datascience/ubi9-python-3.12/Dockerfile.cpu (17)

Learnt from: grdryn
PR: #1320
File: rstudio/rhel9-python-3.11/Dockerfile.cuda:40-42
Timestamp: 2025-07-04T17:08:02.399Z
Learning: In the opendatahub-io/notebooks repository, when using multi-architecture Dockerfiles with BuildKit, the implicit build argument TARGETARCH is automatically available in the global scope for FROM instructions without explicit declaration. However, if TARGETARCH is used within a build stage, it must be declared explicitly within that stage. The current placement pattern (declaring ARG TARGETARCH after FROM instructions that use it) is correct for modern Docker/Podman/Buildah environments and does not require compatibility with older Docker versions.

Learnt from: atheo89
PR: #1258
File: codeserver/ubi9-python-3.11/Dockerfile.cpu:32-32
Timestamp: 2025-07-07T11:08:48.524Z
Learning: atheo89 requested GitHub issue creation for multi-architecture Dockerfile improvements during PR #1258 review, specifically for enhancing structural consistency across Docker stages, replacing $(uname -m) with ${TARGETARCH} for cross-architecture builds, and adding OCI-compliant metadata labels. Issue #1332 was created with comprehensive problem description, phased implementation approach, detailed acceptance criteria, implementation guidance with code examples, and proper context linking, continuing the established pattern of systematic code quality improvements.

Learnt from: grdryn
PR: #1320
File: jupyter/minimal/ubi9-python-3.11/Dockerfile.cuda:29-38
Timestamp: 2025-07-04T17:07:52.656Z
Learning: In the opendatahub-io/notebooks repository, modern Docker with BuildKit automatically provides build arguments like TARGETARCH in the global scope for FROM instructions, but these arguments must be explicitly declared with ARG statements inside build stages where they will be used. The ARG declaration should be placed within the stage that uses it, not moved to the global scope, as this is the correct pattern for modern Docker/Podman/Buildah environments.

Learnt from: jiridanek
PR: #1320
File: jupyter/pytorch/ubi9-python-3.12/Dockerfile.cuda:42-52
Timestamp: 2025-07-09T12:29:56.162Z
Learning: jiridanek requested GitHub issue creation for OpenShift client architecture mapping problem affecting 29 Dockerfiles during PR #1320 review. Issue was created with comprehensive analysis covering all affected files using $(uname -m) returning 'aarch64' but OpenShift mirror expecting 'arm64', systematic solution using BuildKit TARGETARCH mapping with proper amd64→x86_64 and arm64→arm64 conversion, detailed acceptance criteria, and implementation guidance, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1333
File: runtimes/tensorflow/ubi9-python-3.12/Dockerfile.cuda:38-38
Timestamp: 2025-07-08T19:30:20.513Z
Learning: jiridanek requested GitHub issue creation for multi-architecture support in TensorFlow CUDA runtime image during PR #1333 review. Issue was created with comprehensive problem description covering hardcoded NVARCH limitation, multiple solution options using TARGETARCH build argument with architecture mapping, acceptance criteria for multi-architecture builds, implementation guidance with code examples, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1320
File: scripts/buildinputs/dockerfile.go:35-35
Timestamp: 2025-07-10T15:02:13.228Z
Learning: jiridanek requested GitHub issue creation for improving architecture detection in buildinputs tool during PR #1320 review. The automated issue creation failed despite appearing successful, so comprehensive issue content was provided for manual creation covering qemu-user s390x build failures where runtime.GOARCH returns host architecture instead of target architecture, detailed impact analysis, four solution options (environment variable detection, configuration parameter, platform detection logic, hybrid approach), clear acceptance criteria for multi-architecture support, phased implementation guidance, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1333
File: runtimes/rocm-tensorflow/ubi9-python-3.12/Dockerfile.rocm:50-50
Timestamp: 2025-07-08T19:30:01.738Z
Learning: jiridanek requested GitHub issue creation for multi-architecture support in ROCm TensorFlow image during PR #1333 review. Issue #1346 was created with comprehensive problem description covering hardcoded x86_64 architecture breaking multi-arch support, detailed impact analysis, three solution options (runtime detection, BuildKit TARGETARCH integration, hybrid approach) with pros/cons analysis, comprehensive acceptance criteria covering core requirements and testing, phased implementation guidance, related files identification, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1320
File: scripts/buildinputs/dockerfile.go:35-35
Timestamp: 2025-07-10T15:02:13.228Z
Learning: jiridanek requested GitHub issue creation for improving architecture detection in buildinputs tool during PR #1320 review. Issue #1364 was created with comprehensive problem description covering qemu-user s390x build failures where runtime.GOARCH returns host architecture instead of target architecture, detailed impact analysis, four solution options (environment variable detection, configuration parameter, platform detection logic, hybrid approach), clear acceptance criteria for multi-architecture support, phased implementation guidance, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1320
File: jupyter/pytorch/ubi9-python-3.12/Dockerfile.cuda:65-66
Timestamp: 2025-07-09T12:31:02.033Z
Learning: jiridanek requested GitHub issue creation for MSSQL repo file hardcoding problem during PR #1320 review. Issue #1363 was created and updated with comprehensive problem description covering hardcoded x86_64 MSSQL repo files breaking multi-architecture builds across 10 affected Dockerfiles (including datascience, CUDA, ROCm, and TrustyAI variants), detailed root cause analysis, three solution options with code examples, clear acceptance criteria for all image types, implementation guidance following established multi-architecture patterns, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1320
File: scripts/buildinputs/dockerfile.go:35-35
Timestamp: 2025-07-10T15:02:13.228Z
Learning: jiridanek requested GitHub issue creation for improving architecture detection in buildinputs tool during PR #1320 review. Issue was created with comprehensive problem description covering qemu-user s390x build failures where runtime.GOARCH returns host architecture instead of target architecture, detailed impact analysis, four solution options (environment variable detection, configuration parameter, platform detection logic, hybrid approach), clear acceptance criteria for multi-architecture support, phased implementation guidance, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-03T07:03:45.020Z
Learning: The Python 3.11 infrastructure for ROCm TensorFlow images in opendatahub-io/notebooks is already properly configured in the Makefile with both BASE_DIRS entries for Pipfile lock renewals and all-images targets for CI builds, requiring only commenting out the corresponding Python 3.12 entries when downgrading due to wheel availability issues.

Learnt from: grdryn
PR: #1320
File: rstudio/rhel9-python-3.11/Dockerfile.cuda:34-35
Timestamp: 2025-07-04T10:41:13.061Z
Learning: In the opendatahub-io/notebooks repository, when adapting NVIDIA CUDA Dockerfiles, the project intentionally maintains consistency with upstream NVIDIA patterns even when it might involve potential risks like empty variable expansions in package installation commands. This is considered acceptable because the containers only run on RHEL 9 with known yum/dnf behavior, and upstream consistency is prioritized over defensive coding practices.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-01T10:41:56.419Z
Learning: jiridanek's team uses containerized dependency locking for cross-platform compatibility in opendatahub-io/notebooks. They run pipenv lock inside UBI9 containers with specific platform arguments (--platform=linux/amd64 --python-version 3.12) to avoid host OS dependency conflicts when generating Pipfile.lock files.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-11T11:54:28.202Z
Learning: In opendatahub-io/notebooks, Python 3.12-based images (e.g., runtime-cuda-pytorch-ubi9-python-3.12) may fail container runtime tests with "libcrypt.so.1 => not found" for MySQL SASL2 plugin libraries if libxcrypt-compat is missing. The solution is to install libxcrypt-compat in the Dockerfile.

Learnt from: jiridanek
PR: #1396
File: jupyter/tensorflow/ubi9-python-3.12/Dockerfile.cuda:192-195
Timestamp: 2025-07-18T19:01:39.811Z
Learning: In the opendatahub-io/notebooks repository, mixing CentOS packages with UBI base images is bad practice that removes supportability and creates "Frankenstein" images according to Red Hat guidance. However, using EPEL packages is acceptable, though it may require extra work with AIPCC for internal Red Hat builds. The official reference is at developers.redhat.com/articles/ubi-faq.

Learnt from: jiridanek
PR: #1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Dockerfile.rocm:34-37
Timestamp: 2025-07-02T18:19:23.024Z
Learning: In the opendatahub-io/notebooks repository, issue #1241 "Security: Add checksum verification for downloaded binaries in Python 3.12 images" covers the security concern about verifying downloaded binaries including the oc client. Red Hat does not provide signed RPM packages for the OpenShift CLI (oc) in UBI9 official repositories, making manual binary download with checksum verification the recommended approach.

Learnt from: jiridanek
PR: #1333
File: runtimes/pytorch/ubi9-python-3.12/Dockerfile.cuda:17-25
Timestamp: 2025-07-09T08:07:30.628Z
Learning: jiridanek requested GitHub issue creation for oc client installation permission problem in PyTorch CUDA runtime Dockerfile during PR #1333 review. Issue #1356 was created with comprehensive problem description covering USER 1001 permission conflicts with root-owned /opt/app-root/bin directory, detailed impact analysis of build failures and non-executable binaries, current problematic code snippet, complete solution with user switching approach, clear acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

codeserver/ubi9-python-3.12/requirements.txt (10)

Learnt from: grdryn
PR: #1396
File: jupyter/tensorflow/ubi9-python-3.12/Pipfile:13-14
Timestamp: 2025-07-16T00:17:10.313Z
Learning: grdryn corrected CodeRabbit's false assessment about CUDA companion package wheel availability during PR #1396 review. The original analysis incorrectly checked all package releases instead of the specific versions that would be installed with tensorflow[and-cuda]~=2.19.0. The actual versions (nvidia-cudnn-cu12/9.3.0.75, nvidia-cuda-runtime-cu12/12.5.82, nvidia-cublas-cu12/12.5.3.2) do have aarch64 wheels available on PyPI, making the and-cuda extra compatible with arm64 builds.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-24T12:01:45.188Z
Learning: jiridanek requested verification of 2024.2 tag package versions against corresponding Pipfiles during PR #1496 review. The analysis revealed that 2024.2 tags contain intentionally frozen older versions (N-1 behavior) while the current PR correctly updates only the N (latest) tags to match Python 3.11 Pipfiles, demonstrating proper understanding of the repository's N-1 version freezing policy.

Learnt from: jiridanek
PR: #1333
File: runtimes/minimal/ubi9-python-3.12/Pipfile:23-27
Timestamp: 2025-07-09T08:12:05.822Z
Learning: jiridanek corrected CodeRabbit's false positive about urllib3/requests version incompatibility in PR #1333 review. Investigation revealed that requests 2.32.3 actually requires urllib3<3,>=1.21.1 (not urllib3<2 as incorrectly stated), making urllib3 2.5.0 perfectly compatible. This demonstrates the importance of verifying technical assessments against actual build results and PyPI metadata before creating issues.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:15:41.168Z
Learning: TrustyAI's jupyter-bokeh was pinned to 3.0.5 due to compatibility requirements with TrustyAI's visualization components, but the actual deployed version in requirements.txt shows 3.0.7, indicating incremental testing. The upgrade to 4.0.5 in this PR represents the completion of a gradual migration strategy from the 3.x series after confirming compatibility with Bokeh 3.7.3.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-23T16:18:42.922Z
Learning: The TensorFlow ROCm Python 3.12 compatibility issue in opendatahub-io/notebooks PR #1259 was caused by using tensorflow-rocm==2.14.0.600 in Pipfile.lock which lacks Python 3.12 wheels, while the Pipfile specifies tensorflow_rocm=~=2.18.1. The solution requires updating Pipfile sources to include https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4/ repository which contains tensorflow_rocm-2.18.1-cp312-cp312-manylinux_2_28_x86_64.whl and regenerating Pipfile.lock using the piplock-refresh GitHub Action.

Learnt from: jiridanek
PR: #1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-03T07:03:45.020Z
Learning: The Python 3.11 infrastructure for ROCm TensorFlow images in opendatahub-io/notebooks is already properly configured in the Makefile with both BASE_DIRS entries for Pipfile lock renewals and all-images targets for CI builds, requiring only commenting out the corresponding Python 3.12 entries when downgrading due to wheel availability issues.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-01T10:41:56.419Z
Learning: jiridanek's team uses containerized dependency locking for cross-platform compatibility in opendatahub-io/notebooks. They run pipenv lock inside UBI9 containers with specific platform arguments (--platform=linux/amd64 --python-version 3.12) to avoid host OS dependency conflicts when generating Pipfile.lock files.

Learnt from: jiridanek
PR: #1396
File: runtimes/tensorflow/ubi9-python-3.12/Dockerfile.cuda:124-127
Timestamp: 2025-07-20T20:47:36.509Z
Learning: jiridanek identified that ARM64 wheels for h5py 3.14.0 are available on PyPI but being ignored due to AMD64-only dependency locking with --platform=linux/amd64. This causes unnecessary hdf5-devel package installation in ARM64 TensorFlow images when the ARM64 wheel h5py-3.14.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl could be used instead. The Pipfile.lock only contains 2 hashes for h5py, confirming limited platform consideration during lock generation.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:21:09.429Z
Learning: TrustyAI 0.6.1 (latest version as of June 2025) has a hard dependency constraint on jupyter-bokeh~=3.0.5, preventing upgrades to jupyter-bokeh 4.x in notebook images that include TrustyAI. This requires either waiting for TrustyAI to update their dependency or excluding TrustyAI from jupyter-bokeh upgrades.

jupyter/trustyai/ubi9-python-3.12/requirements.txt (15)

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:15:41.168Z
Learning: TrustyAI's jupyter-bokeh was pinned to 3.0.5 due to compatibility requirements with TrustyAI's visualization components, but the actual deployed version in requirements.txt shows 3.0.7, indicating incremental testing. The upgrade to 4.0.5 in this PR represents the completion of a gradual migration strategy from the 3.x series after confirming compatibility with Bokeh 3.7.3.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:21:09.429Z
Learning: TrustyAI explicitly declares jupyter-bokeh~=3.0.5 as a hard dependency in both their requirements.txt and pyproject.toml files, with no open issues tracking jupyter-bokeh 4.x compatibility. This creates an unresolvable pip dependency conflict when trying to upgrade jupyter-bokeh to 4.x in notebook images that include TrustyAI.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:15:41.168Z
Learning: The jupyter-bokeh package was previously pinned to version 3.0.5 in the TrustyAI notebook image due to compatibility requirements with TrustyAI components, as indicated by the comment "Should be pinned down to this version in order to be compatible with trustyai" that was removed in this update.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:21:09.429Z
Learning: TrustyAI 0.6.1 (latest version as of June 2025) has a hard dependency constraint on jupyter-bokeh~=3.0.5, preventing upgrades to jupyter-bokeh 4.x in notebook images that include TrustyAI. This requires either waiting for TrustyAI to update their dependency or excluding TrustyAI from jupyter-bokeh upgrades.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:21:09.429Z
Learning: The jupyter-bokeh pinning to 3.0.5 in TrustyAI notebook image was not due to TrustyAI code compatibility issues, but because the trustyai package itself explicitly declares jupyter-bokeh~=3.0.5 as a hard dependency, causing pip dependency resolution conflicts when trying to upgrade to jupyter-bokeh 4.x.

Learnt from: grdryn
PR: #1396
File: jupyter/tensorflow/ubi9-python-3.12/Pipfile:13-14
Timestamp: 2025-07-16T00:17:10.313Z
Learning: grdryn corrected CodeRabbit's false assessment about CUDA companion package wheel availability during PR #1396 review. The original analysis incorrectly checked all package releases instead of the specific versions that would be installed with tensorflow[and-cuda]~=2.19.0. The actual versions (nvidia-cudnn-cu12/9.3.0.75, nvidia-cuda-runtime-cu12/12.5.82, nvidia-cublas-cu12/12.5.3.2) do have aarch64 wheels available on PyPI, making the and-cuda extra compatible with arm64 builds.

Learnt from: jiridanek
PR: #1306
File: jupyter/trustyai/ubi9-python-3.12/kustomize/base/kustomization.yaml:8-12
Timestamp: 2025-07-08T19:09:48.746Z
Learning: jiridanek requested GitHub issue creation for misleading CUDA prefix in TrustyAI image tags during PR #1306 review, affecting both Python 3.11 and 3.12 versions. Issue #1338 was created with comprehensive problem description covering both affected images, repository pattern analysis comparing correct vs incorrect naming conventions, clear solution with code examples, detailed acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-24T12:01:45.188Z
Learning: jiridanek requested verification of 2024.2 tag package versions against corresponding Pipfiles during PR #1496 review. The analysis revealed that 2024.2 tags contain intentionally frozen older versions (N-1 behavior) while the current PR correctly updates only the N (latest) tags to match Python 3.11 Pipfiles, demonstrating proper understanding of the repository's N-1 version freezing policy.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-23T16:18:42.922Z
Learning: The TensorFlow ROCm Python 3.12 compatibility issue in opendatahub-io/notebooks PR #1259 was caused by using tensorflow-rocm==2.14.0.600 in Pipfile.lock which lacks Python 3.12 wheels, while the Pipfile specifies tensorflow_rocm=~=2.18.1. The solution requires updating Pipfile sources to include https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4/ repository which contains tensorflow_rocm-2.18.1-cp312-cp312-manylinux_2_28_x86_64.whl and regenerating Pipfile.lock using the piplock-refresh GitHub Action.

Learnt from: jiridanek
PR: #1333
File: runtimes/datascience/ubi9-python-3.12/Pipfile:33-34
Timestamp: 2025-07-08T19:26:17.140Z
Learning: jiridanek requested GitHub issue creation for jupyter-client dependency pinning inconsistency during PR #1333 review, specifically asking to note the implications of breaking changes in 9.x versions. Issue #1343 was created with comprehensive problem description covering inconsistent pinning style across all Python 3.12 runtime images, detailed breaking changes analysis (kernel protocol, session management, connection security, API changes, async/await modifications), reproducibility and security impact assessment, multiple solution options with code examples, phased acceptance criteria, implementation guidance, testing approach, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-03T07:03:45.020Z
Learning: The Python 3.11 infrastructure for ROCm TensorFlow images in opendatahub-io/notebooks is already properly configured in the Makefile with both BASE_DIRS entries for Pipfile lock renewals and all-images targets for CI builds, requiring only commenting out the corresponding Python 3.12 entries when downgrading due to wheel availability issues.

Learnt from: jiridanek
PR: #1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-03T07:03:45.020Z
Learning: tensorflow_rocm package versions 2.12.1.570 through 2.14.0.600 do not provide Python 3.12 wheels (cp312) on PyPI, causing Pipfile lock failures when attempting to create Python 3.12-based ROCm TensorFlow notebook images in opendatahub-io/notebooks.

Learnt from: jiridanek
PR: #1396
File: runtimes/tensorflow/ubi9-python-3.12/Dockerfile.cuda:124-127
Timestamp: 2025-07-20T20:47:36.509Z
Learning: jiridanek identified that ARM64 wheels for h5py 3.14.0 are available on PyPI but being ignored due to AMD64-only dependency locking with --platform=linux/amd64. This causes unnecessary hdf5-devel package installation in ARM64 TensorFlow images when the ARM64 wheel h5py-3.14.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl could be used instead. The Pipfile.lock only contains 2 hashes for h5py, confirming limited platform consideration during lock generation.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-01T10:41:56.419Z
Learning: jiridanek's team uses containerized dependency locking for cross-platform compatibility in opendatahub-io/notebooks. They run pipenv lock inside UBI9 containers with specific platform arguments (--platform=linux/amd64 --python-version 3.12) to avoid host OS dependency conflicts when generating Pipfile.lock files.

Learnt from: jiridanek
PR: #1154
File: manifests/base/jupyter-pytorch-notebook-imagestream.yaml:0-0
Timestamp: 2025-06-16T11:06:33.139Z
Learning: In the opendatahub-io/notebooks repository, N-1 versions of images in manifest files (like imagestream.yaml files) should not be updated regularly. The versions of packages like codeflare-sdk in N-1 images are frozen to what was released when the image was moved from N to N-1 version. N-1 images are only updated for security vulnerabilities of packages, not for regular version bumps. This is why the version of packages in N-1 images may be quite old compared to the latest N version.

jupyter/datascience/ubi9-python-3.12/requirements.txt (18)

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:15:41.168Z
Learning: TrustyAI's jupyter-bokeh was pinned to 3.0.5 due to compatibility requirements with TrustyAI's visualization components, but the actual deployed version in requirements.txt shows 3.0.7, indicating incremental testing. The upgrade to 4.0.5 in this PR represents the completion of a gradual migration strategy from the 3.x series after confirming compatibility with Bokeh 3.7.3.

Learnt from: jiridanek
PR: #1333
File: runtimes/datascience/ubi9-python-3.12/Pipfile:33-34
Timestamp: 2025-07-08T19:26:17.140Z
Learning: jiridanek requested GitHub issue creation for jupyter-client dependency pinning inconsistency during PR #1333 review, specifically asking to note the implications of breaking changes in 9.x versions. Issue #1343 was created with comprehensive problem description covering inconsistent pinning style across all Python 3.12 runtime images, detailed breaking changes analysis (kernel protocol, session management, connection security, API changes, async/await modifications), reproducibility and security impact assessment, multiple solution options with code examples, phased acceptance criteria, implementation guidance, testing approach, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: grdryn
PR: #1396
File: jupyter/tensorflow/ubi9-python-3.12/Pipfile:13-14
Timestamp: 2025-07-16T00:17:10.313Z
Learning: grdryn corrected CodeRabbit's false assessment about CUDA companion package wheel availability during PR #1396 review. The original analysis incorrectly checked all package releases instead of the specific versions that would be installed with tensorflow[and-cuda]~=2.19.0. The actual versions (nvidia-cudnn-cu12/9.3.0.75, nvidia-cuda-runtime-cu12/12.5.82, nvidia-cublas-cu12/12.5.3.2) do have aarch64 wheels available on PyPI, making the and-cuda extra compatible with arm64 builds.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-24T12:01:45.188Z
Learning: jiridanek requested verification of 2024.2 tag package versions against corresponding Pipfiles during PR #1496 review. The analysis revealed that 2024.2 tags contain intentionally frozen older versions (N-1 behavior) while the current PR correctly updates only the N (latest) tags to match Python 3.11 Pipfiles, demonstrating proper understanding of the repository's N-1 version freezing policy.

Learnt from: jiridanek
PR: #1333
File: runtimes/minimal/ubi9-python-3.12/Pipfile:23-27
Timestamp: 2025-07-09T08:12:05.822Z
Learning: jiridanek corrected CodeRabbit's false positive about urllib3/requests version incompatibility in PR #1333 review. Investigation revealed that requests 2.32.3 actually requires urllib3<3,>=1.21.1 (not urllib3<2 as incorrectly stated), making urllib3 2.5.0 perfectly compatible. This demonstrates the importance of verifying technical assessments against actual build results and PyPI metadata before creating issues.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-23T16:18:42.922Z
Learning: The TensorFlow ROCm Python 3.12 compatibility issue in opendatahub-io/notebooks PR #1259 was caused by using tensorflow-rocm==2.14.0.600 in Pipfile.lock which lacks Python 3.12 wheels, while the Pipfile specifies tensorflow_rocm=~=2.18.1. The solution requires updating Pipfile sources to include https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4/ repository which contains tensorflow_rocm-2.18.1-cp312-cp312-manylinux_2_28_x86_64.whl and regenerating Pipfile.lock using the piplock-refresh GitHub Action.

Learnt from: jiridanek
PR: #1306
File: jupyter/trustyai/ubi9-python-3.12/kustomize/base/kustomization.yaml:8-12
Timestamp: 2025-07-08T19:09:48.746Z
Learning: jiridanek requested GitHub issue creation for misleading CUDA prefix in TrustyAI image tags during PR #1306 review, affecting both Python 3.11 and 3.12 versions. Issue #1338 was created with comprehensive problem description covering both affected images, repository pattern analysis comparing correct vs incorrect naming conventions, clear solution with code examples, detailed acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1333
File: runtimes/pytorch/ubi9-python-3.12/utils/bootstrapper.py:619-626
Timestamp: 2025-07-08T19:33:14.340Z
Learning: jiridanek requested GitHub issue creation for Python 3.12 version check bug in bootstrapper.py during PR #1333 review. Issue #1348 was created with comprehensive problem description covering version check exclusion affecting all Python 3.12 runtime images, detailed impact analysis of bootstrapper execution failures, clear solution with code examples, affected files list including all 6 runtime bootstrapper copies, acceptance criteria for testing and verification, implementation notes about code duplication and upstream reporting, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1379
File: .tekton/odh-workbench-jupyter-datascience-cpu-py312-ubi9-push.yaml:14-17
Timestamp: 2025-07-11T11:15:47.424Z
Learning: jiridanek requested GitHub issue creation for CEL filter problem in datascience workbench Tekton pipelines during PR #1379 review. Issue #1383 was successfully created with comprehensive problem description covering both Python 3.11 and 3.12 pipelines incorrectly watching jupyter/minimal directories instead of jupyter/datascience directories, detailed impact analysis of pipeline execution failures, complete solution with before/after code examples, thorough acceptance criteria for path updates and pipeline triggering verification, implementation notes about repository structure alignment, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

Learnt from: jiridanek
PR: #1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-03T07:03:45.020Z
Learning: The Python 3.11 infrastructure for ROCm TensorFlow images in opendatahub-io/notebooks is already properly configured in the Makefile with both BASE_DIRS entries for Pipfile lock renewals and all-images targets for CI builds, requiring only commenting out the corresponding Python 3.12 entries when downgrading due to wheel availability issues.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:21:09.429Z
Learning: TrustyAI explicitly declares jupyter-bokeh~=3.0.5 as a hard dependency in both their requirements.txt and pyproject.toml files, with no open issues tracking jupyter-bokeh 4.x compatibility. This creates an unresolvable pip dependency conflict when trying to upgrade jupyter-bokeh to 4.x in notebook images that include TrustyAI.

Learnt from: jiridanek
PR: #1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-03T07:03:45.020Z
Learning: tensorflow_rocm package versions 2.12.1.570 through 2.14.0.600 do not provide Python 3.12 wheels (cp312) on PyPI, causing Pipfile lock failures when attempting to create Python 3.12-based ROCm TensorFlow notebook images in opendatahub-io/notebooks.

Learnt from: jiridanek
PR: #1218
File: jupyter/trustyai/ubi9-python-3.11/Pipfile:49-49
Timestamp: 2025-06-28T14:21:09.429Z
Learning: TrustyAI 0.6.1 (latest version as of June 2025) has a hard dependency constraint on jupyter-bokeh~=3.0.5, preventing upgrades to jupyter-bokeh 4.x in notebook images that include TrustyAI. This requires either waiting for TrustyAI to update their dependency or excluding TrustyAI from jupyter-bokeh upgrades.

Learnt from: jiridanek
PR: #1396
File: runtimes/tensorflow/ubi9-python-3.12/Dockerfile.cuda:124-127
Timestamp: 2025-07-20T20:47:36.509Z
Learning: jiridanek identified that ARM64 wheels for h5py 3.14.0 are available on PyPI but being ignored due to AMD64-only dependency locking with --platform=linux/amd64. This causes unnecessary hdf5-devel package installation in ARM64 TensorFlow images when the ARM64 wheel h5py-3.14.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl could be used instead. The Pipfile.lock only contains 2 hashes for h5py, confirming limited platform consideration during lock generation.

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-01T10:41:56.419Z
Learning: jiridanek's team uses containerized dependency locking for cross-platform compatibility in opendatahub-io/notebooks. They run pipenv lock inside UBI9 containers with specific platform arguments (--platform=linux/amd64 --python-version 3.12) to avoid host OS dependency conflicts when generating Pipfile.lock files.

Learnt from: jiridanek
PR: #1154
File: manifests/base/jupyter-pytorch-notebook-imagestream.yaml:0-0
Timestamp: 2025-06-16T11:06:33.139Z
Learning: In the opendatahub-io/notebooks repository, N-1 versions of images in manifest files (like imagestream.yaml files) should not be updated regularly. The versions of packages like codeflare-sdk in N-1 images are frozen to what was released when the image was moved from N to N-1 version. N-1 images are only updated for security vulnerabilities of packages, not for regular version bumps. This is why the version of packages in N-1 images may be quite old compared to the latest N version.

Learnt from: jiridanek
PR: #1218
File: jupyter/datascience/ubi9-python-3.11/Pipfile:34-36
Timestamp: 2025-06-28T14:13:27.890Z
Learning: In the opendatahub-io/notebooks repository, the dependency pinning strategy follows a deliberate pattern: core jupyterlab package uses exact pinning (==) across all notebook images to ensure UI consistency, while JupyterLab extensions and all server components (jupyter-server, jupyter-server-proxy, jupyter-server-terminals) use compatible release (~=) pinning to allow automatic security updates and bug fixes while maintaining API compatibility.

jupyter/minimal/ubi9-python-3.12/requirements.txt (13)