RHOAIENG-18582: Post ODH Elyra release (v4.3.0) #2555
Conversation
|
Skipping CI for Draft Pull Request. |
WalkthroughBumps odh-elyra from 4.2.4 to 4.3.1 across multiple Jupyter images. Removes local bootstrapper.py from several runtime images. Adds a Dockerfile step to fetch bootstrapper.py at build time. Updates setup-elyra.sh to copy the bootstrapper and adjust/add environment variables. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
github-actions
bot
added
the
review-requested
| # Copy Elyra dependencies for air-gapped enviroment | ||
| COPY ${DATASCIENCE_SOURCE_CODE}/utils ./utils/ | ||
| # Download Elyra boostrapper.py | ||
| RUN curl -fL https://raw.githubusercontent.com/opendatahub-io/elyra/refs/heads/main/elyra/kfp/bootstrapper.py \ |
There was a problem hiding this comment.
Are we sure that this will work in disconnected envs?
There was a problem hiding this comment.
Hello @atheo89 I thought that it would work because this would be done during the image build time. The idea is now that Notebook images don't need a custom bootstrapper.py we could download it from Elyra and use the new system property we introduced: ELYRA_INSTALL_PACKAGES to prevent Elyra from trying to download packages - this new flag should do that
There was a problem hiding this comment.
Can we curl that file from a fixed tag (associated with the Elyra version in the image) instead of main?
EDIT: nvm, I saw the other comment below.
| # Copy Elyra dependencies for air-gapped enviroment | ||
| COPY ${TENSORFLOW_SOURCE_CODE}/utils ./utils/ | ||
| # Download Elyra boostrapper.py | ||
| RUN curl -fL https://raw.githubusercontent.com/opendatahub-io/elyra/refs/tags/v4.3.0/elyra/kfp/bootstrapper.py \ |
There was a problem hiding this comment.
isn't this bootstrapper installed together with the odh-elyra package? Can't we grab it from python's site-packages? And if it is not installed, can't we install it that way?
There was a problem hiding this comment.
that's a great suggestion, it should be with the elyra package, it would be better to copy it from there - I will check and update the Dockerfile accordingly
There was a problem hiding this comment.
@jiridanek I found the local bootstrapper.py and copy it to utils on script setup-elyra.sh - would it be the best place to do it?
Thanks!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (23)
jupyter/datascience/ubi9-python-3.12/pylock.toml(1 hunks)jupyter/datascience/ubi9-python-3.12/pyproject.toml(1 hunks)jupyter/datascience/ubi9-python-3.12/setup-elyra.sh(2 hunks)jupyter/pytorch+llmcompressor/ubi9-python-3.12/pylock.toml(1 hunks)jupyter/pytorch+llmcompressor/ubi9-python-3.12/pyproject.toml(1 hunks)jupyter/pytorch/ubi9-python-3.12/pylock.toml(1 hunks)jupyter/pytorch/ubi9-python-3.12/pyproject.toml(1 hunks)jupyter/rocm/pytorch/ubi9-python-3.12/pylock.toml(1 hunks)jupyter/rocm/pytorch/ubi9-python-3.12/pyproject.toml(1 hunks)jupyter/rocm/tensorflow/ubi9-python-3.12/pylock.toml(1 hunks)jupyter/rocm/tensorflow/ubi9-python-3.12/pyproject.toml(1 hunks)jupyter/tensorflow/ubi9-python-3.12/pylock.toml(1 hunks)jupyter/tensorflow/ubi9-python-3.12/pyproject.toml(1 hunks)jupyter/trustyai/ubi9-python-3.12/pylock.toml(1 hunks)jupyter/trustyai/ubi9-python-3.12/pyproject.toml(1 hunks)runtimes/datascience/ubi9-python-3.12/utils/bootstrapper.py(0 hunks)runtimes/minimal/ubi9-python-3.12/Dockerfile.cpu(1 hunks)runtimes/minimal/ubi9-python-3.12/utils/bootstrapper.py(0 hunks)runtimes/pytorch+llmcompressor/ubi9-python-3.12/utils/bootstrapper.py(0 hunks)runtimes/pytorch/ubi9-python-3.12/utils/bootstrapper.py(0 hunks)runtimes/rocm-pytorch/ubi9-python-3.12/utils/bootstrapper.py(0 hunks)runtimes/rocm-tensorflow/ubi9-python-3.12/utils/bootstrapper.py(0 hunks)runtimes/tensorflow/ubi9-python-3.12/utils/bootstrapper.py(0 hunks)
💤 Files with no reviewable changes (7)
- runtimes/rocm-tensorflow/ubi9-python-3.12/utils/bootstrapper.py
- runtimes/tensorflow/ubi9-python-3.12/utils/bootstrapper.py
- runtimes/pytorch+llmcompressor/ubi9-python-3.12/utils/bootstrapper.py
- runtimes/minimal/ubi9-python-3.12/utils/bootstrapper.py
- runtimes/datascience/ubi9-python-3.12/utils/bootstrapper.py
- runtimes/rocm-pytorch/ubi9-python-3.12/utils/bootstrapper.py
- runtimes/pytorch/ubi9-python-3.12/utils/bootstrapper.py
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: jiridanek
PR: opendatahub-io/notebooks#1875
File: runtimes/pytorch/ubi9-python-3.12/pylock.toml:0-0
Timestamp: 2025-09-12T23:14:23.293Z
Learning: Issue #3325 in elyra-ai/elyra repository documents the urllib3 2.x compatibility blocker affecting jupyter/* images in opendatahub-io/notebooks. The dependency chain odh-elyra==4.2.3 → appengine-python-standard==1.1.10 → urllib3>=1.26.2,<2 prevents upgrading from urllib3=1.26.20 to 2.5.0 in images that include elyra, explaining why only codeserver/* and runtimes/* images achieved the 2.5.0 upgrade in PR #1875.
📚 Learning: 2025-09-12T23:14:23.293Z
Learnt from: jiridanek
PR: opendatahub-io/notebooks#1875
File: runtimes/pytorch/ubi9-python-3.12/pylock.toml:0-0
Timestamp: 2025-09-12T23:14:23.293Z
Learning: Issue #3325 in elyra-ai/elyra repository documents the urllib3 2.x compatibility blocker affecting jupyter/* images in opendatahub-io/notebooks. The dependency chain odh-elyra==4.2.3 → appengine-python-standard==1.1.10 → urllib3>=1.26.2,<2 prevents upgrading from urllib3=1.26.20 to 2.5.0 in images that include elyra, explaining why only codeserver/* and runtimes/* images achieved the 2.5.0 upgrade in PR #1875.
Applied to files:
jupyter/trustyai/ubi9-python-3.12/pyproject.tomljupyter/trustyai/ubi9-python-3.12/pylock.tomljupyter/tensorflow/ubi9-python-3.12/pylock.tomljupyter/datascience/ubi9-python-3.12/pylock.tomljupyter/pytorch+llmcompressor/ubi9-python-3.12/pylock.tomljupyter/tensorflow/ubi9-python-3.12/pyproject.tomljupyter/datascience/ubi9-python-3.12/pyproject.tomljupyter/pytorch/ubi9-python-3.12/pyproject.tomljupyter/pytorch+llmcompressor/ubi9-python-3.12/pyproject.toml
📚 Learning: 2025-07-23T16:18:42.922Z
Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-23T16:18:42.922Z
Learning: The TensorFlow ROCm Python 3.12 compatibility issue in opendatahub-io/notebooks PR #1259 was caused by using tensorflow-rocm==2.14.0.600 in Pipfile.lock which lacks Python 3.12 wheels, while the Pipfile specifies tensorflow_rocm=~=2.18.1. The solution requires updating Pipfile sources to include https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4/ repository which contains tensorflow_rocm-2.18.1-cp312-cp312-manylinux_2_28_x86_64.whl and regenerating Pipfile.lock using the piplock-refresh GitHub Action.
Applied to files:
jupyter/rocm/tensorflow/ubi9-python-3.12/pylock.toml
📚 Learning: 2025-08-29T12:20:28.311Z
Learnt from: jiridanek
PR: opendatahub-io/notebooks#1875
File: runtimes/pytorch+llmcompressor/ubi9-python-3.12/Pipfile:70-70
Timestamp: 2025-08-29T12:20:28.311Z
Learning: jiridanek requested GitHub issue creation for urllib3 2.x compatibility blocked by appengine-python-standard dependency during PR #1875 review. Issue #3325 was created on elyra-ai/elyra repository explaining the dependency conflict chain (odh-elyra==4.2.3 → appengine-python-standard==1.1.10 → urllib3>=1.26.2,<2) that blocks updating urllib3 to 2.5.0 for CVE security fixes across notebook runtime environments.
Applied to files:
jupyter/tensorflow/ubi9-python-3.12/pyproject.tomljupyter/datascience/ubi9-python-3.12/pyproject.toml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (14)
- GitHub Check: Red Hat Konflux / odh-workbench-jupyter-pytorch-llmcompressor-cuda-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-workbench-jupyter-tensorflow-cuda-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-pipeline-runtime-pytorch-llmcompressor-cuda-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-workbench-jupyter-tensorflow-rocm-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-pipeline-runtime-pytorch-cuda-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-pipeline-runtime-tensorflow-cuda-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-workbench-jupyter-datascience-cpu-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-workbench-jupyter-trustyai-cpu-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-pipeline-runtime-pytorch-rocm-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-pipeline-runtime-tensorflow-rocm-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-workbench-jupyter-pytorch-rocm-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-workbench-jupyter-pytorch-cuda-py312-ubi9-on-pull-request
- GitHub Check: Red Hat Konflux / odh-pipeline-runtime-minimal-cpu-py312-ubi9-on-pull-request
- GitHub Check: Generate job matrix
🔇 Additional comments (6)
jupyter/pytorch/ubi9-python-3.12/pylock.toml (1)
2595-2599: Confirmed odh-elyra 4.3.1 artifacts on PyPI
sdist and wheel are live with matching SHA256 hashes; safe to merge.jupyter/rocm/tensorflow/ubi9-python-3.12/pylock.toml (1)
2511-2513: Confirm Elyra target version.This lockfile now pins
odh-elyrato 4.3.1, while the PR description frames the update as 4.3.0. Please double-check that 4.3.1 is the intended release (and already available for the build pipeline) before merging.jupyter/rocm/pytorch/ubi9-python-3.12/pylock.toml (1)
2459-2461: Lockfile entry looks consistent with 4.3.1.Version, URLs, timestamps, and hashes follow the established pattern for odh-elyra updates. 👍
jupyter/datascience/ubi9-python-3.12/pyproject.toml (1)
30-30: Dependency bump matches Elyra 4.3.x rollout.Thanks for updating the datascience image to pin
odh-elyra==4.3.1; this keeps it aligned with the new bootstrapper flow introduced elsewhere in the PR. No further action from my side.jupyter/pytorch+llmcompressor/ubi9-python-3.12/pyproject.toml (1)
46-46: Good to see odh-elyra unified at 4.3.1.The pytorch+llmcompressor image now tracks the same Elyra release as the rest of the stack, which will simplify maintenance once 4.3.x ships. Looks solid.
jupyter/pytorch+llmcompressor/ubi9-python-3.12/pylock.toml (1)
2706-2708: Confirm odh-elyra bump target, lock consistency, and urllib3 compatibility
Lock pins 4.3.1 but PR title says 4.3.0—confirm intended version, ensure all pylock.toml files use 4.3.1, and verify that appengine-python-standard no longer enforces urllib3<2.
| # By copying this we must make sure that ELYRA_INSTALL_PACKAGES=false | ||
| cp /opt/app-root/lib/python3.12/site-packages/elyra/kfp/bootstrapper.py /opt/app-root/bin/utils/ |
There was a problem hiding this comment.
Ensure the bootstrapper copy failure aborts the script.
This cp runs before the bootstrapper exists anywhere else. Because the script isn’t running with set -e, if the path ever changes (e.g., packaging move, missing dependency), cp quietly fails and the remainder of the setup keeps running without a bootstrapper—breaking Elyra at runtime. Please make the step fail fast (for example by adding set -e/set -euo pipefail near the top or by checking cp ... || exit 1).
🤖 Prompt for AI Agents
In jupyter/datascience/ubi9-python-3.12/setup-elyra.sh around lines 4 to 5, the
cp command can fail silently because the script doesn't abort on errors; modify
the script to fail fast by enabling strict error handling (e.g., add set -euo
pipefail near the top) and/or make the cp command explicitly abort on failure
(append a check like cp ... || exit 1) so that a missing bootstrapper stops the
setup immediately.
There was a problem hiding this comment.
@jesuino yeah, like that, that is what I had in mind
@coderabbitai do you expect any problems just adding set -Eeuo pipefail?
There was a problem hiding this comment.
@jiridanek Adding set -Eeuo pipefail should work fine and is the right approach for fail-fast behavior. Here's what each flag does in this script:
-e: ✅ Exactly what you want—exits immediately if any command fails (like thecpfor bootstrapper)-E: No effect here (inherits ERR traps to functions, but there are no ERR traps defined)-u: ✅ Safe—treats unset variables as errors; all variables in the script are explicitly exported-o pipefail: No effect here (no pipelines in this script)
Minor pre-existing issue (unrelated to adding the flags):
Line 11 has an unquoted command substitution:
if [ ! -d $(jupyter --data-dir)/metadata/runtimes/ ]; thenShould ideally be:
if [ ! -d "$(jupyter --data-dir)/metadata/runtimes/" ]; thenThis prevents potential word-splitting issues if the path contains spaces, but it's a pre-existing condition and won't break with the new flags.
Recommendation: Go ahead and add set -Eeuo pipefail right after the shebang—it will make the script more robust without breaking existing functionality. 🚀
🧠 Learnings used
Learnt from: jiridanek
PR: opendatahub-io/notebooks#1269
File: codeserver/ubi9-python-3.12/run-nginx.sh:5-6
Timestamp: 2025-07-03T16:04:30.516Z
Learning: jiridanek requested GitHub issue creation for shell script strict mode improvement in codeserver/ubi9-python-3.12/run-nginx.sh during PR #1269 review. Issue #1308 was created with comprehensive problem description covering limitations of `set -e`, detailed solution with `set -euo pipefail`, benefits explanation for each flag, acceptance criteria, risk assessment, testing approach, and proper context linking, continuing the established pattern of systematic code quality improvements for shell scripts.
| RUN curl -fL https://raw.githubusercontent.com/opendatahub-io/elyra/refs/tags/v4.3.0/elyra/kfp/bootstrapper.py \ | ||
| -o ./utils/bootstrapper.py |
There was a problem hiding this comment.
Align bootstrapper version with Elyra bump and avoid network-dependent builds.
We now pull bootstrapper.py at build time from Elyra tag v4.3.0, but the rest of this PR upgrades ODH Elyra to 4.3.1. Shipping a 4.3.0 bootstrapper alongside 4.3.1 bits is a mismatch that can regress the fixes we’re picking up. In addition, downloading from GitHub during the build defeats disconnected/air-gapped builds—the very scenario the new ELYRA_INSTALL_PACKAGES=false knob is supposed to support. Please vendor the correct 4.3.1 bootstrapper (or fetch it from a pre-staged artifact inside the image) so the Docker build stays reproducible and works without external network access.
🤖 Prompt for AI Agents
In runtimes/minimal/ubi9-python-3.12/Dockerfile.cpu around lines 77-78, the
Dockerfile downloads bootstrapper.py from Elyra tag v4.3.0 at build time which
mismatches the repo bump to v4.3.1 and prevents air-gapped/disconnected builds;
replace the curl step with a reproducible, network-free approach by vendoring
the v4.3.1 bootstrapper into the build context (or copying it from a pre-staged
artifact inside the image/build context), update the Dockerfile to ADD/COPY that
local file into ./utils/bootstrapper.py instead of curling, and ensure the
vendored file version matches Elyra v4.3.1 (optionally add a checksum or comment
noting the source and tag).
There was a problem hiding this comment.
what if we do
cp /opt/app-root/lib/python3.12/site-packages/elyra/kfp/bootstrapper.py ...
There was a problem hiding this comment.
@jiridanek, that's an excellent solution! 🎯
Copying bootstrapper.py from the installed Elyra package would indeed solve both issues:
- ✅ Version alignment: Automatically uses the correct 4.3.1 version
- ✅ Air-gapped builds: No network dependency during build
However, there's an ordering consideration in the Dockerfile. Currently:
- Line 75:
COPY ${MINIMAL_SOURCE_CODE}/utils ./utils/ - Line 77-78:
curlbootstrapper.py (problematic) - Line 80:
uv pip install(this installs odh-elyra and creates site-packages)
To implement your approach, the copy would need to happen after the uv pip install step, since that's when the Elyra package gets installed to site-packages. Something like:
# Copy Elyra dependencies for air-gapped environment
COPY ${MINIMAL_SOURCE_CODE}/utils ./utils/
RUN echo "Installing softwares and packages" && \
uv pip install --strict --no-deps --no-cache ... && \
# Copy bootstrapper from installed Elyra package
cp /opt/app-root/lib/python3.12/site-packages/elyra/kfp/bootstrapper.py ./utils/bootstrapper.py && \
chmod -R g+w /opt/app-root/lib/python3.12/site-packages && \
fix-permissions /opt/app-root -PThis approach is cleaner, more maintainable, and naturally stays in sync with the Elyra version specified in pylock.toml. 👍
|
@jesuino: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Description
This is a preparation for Elyra 4.3.0 release. It is a working in progress because we need Elyra to be released. Elyra 4.3.0 should be released from main branch, hence a few environment properties changes are required in order to avoid breaking the images that use Elyra.
This PR include the following changes:
bootstrapper.pyremoving the need of having a copy hereELYRA_INSTALL_PACKAGES=false) so it will work on air-gapped environmentsHow Has This Been Tested?
We are waiting images to be available so we can make Elyra sanity checks.