Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/build-notebooks-TEMPLATE.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -319,15 +319,17 @@ jobs:

# region Pytest image tests

- run: echo "UV_PROJECT=${PWD}/tests" >> "${GITHUB_ENV}"

# https://github.com/astral-sh/setup-uv
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v5
with:
version: "latest"
python-version: "3.12"
enable-cache: true
cache-dependency-glob: "uv.lock"
pyproject-file: "pyproject.toml"
cache-dependency-glob: "tests/uv.lock"
pyproject-file: "tests/pyproject.toml"

- name: Check uv is installed correctly
run: uv version
Expand Down
9 changes: 7 additions & 2 deletions .github/workflows/code-quality.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,17 @@ jobs:
steps:
- uses: actions/checkout@v5

- run: echo "UV_PROJECT=${PWD}/tests" >> "${GITHUB_ENV}"

# https://github.com/astral-sh/setup-uv
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v5
with:
version: "latest"
python-version: "3.12"
enable-cache: true
cache-dependency-glob: "uv.lock"
pyproject-file: "pyproject.toml"
cache-dependency-glob: "tests/uv.lock"
pyproject-file: "tests/pyproject.toml"

- name: Check uv is installed correctly
run: uv version
Expand All @@ -54,6 +56,9 @@ jobs:
- name: Run pre-commit on all files
run: |
uv run pre-commit run --all-files
env:
# warning: `VIRTUAL_ENV=.venv` does not match the project environment path `tests/.venv` and will be ignored; use `--active` to target the active environment instead
VIRTUAL_ENV: tests/.venv

- run: uv run pytest

Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,17 @@ jobs:
steps:
- uses: actions/checkout@v5

- run: echo "UV_PROJECT=${PWD}/tests" >> "${GITHUB_ENV}"

# https://github.com/astral-sh/setup-uv
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v5
with:
version: "latest"
python-version: "3.12"
enable-cache: true
cache-dependency-glob: "uv.lock"
pyproject-file: "pyproject.toml"
cache-dependency-glob: "tests/uv.lock"
pyproject-file: "tests/pyproject.toml"

- name: Run the release notes script
run: |
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -477,6 +477,7 @@ refresh-pipfilelock-files:
echo "Regenerating requirements.txt files"
pushd $(ROOT_DIR)
bash $(ROOT_DIR)/scripts/sync-requirements-txt.sh
bash $(ROOT_DIR)/scripts/sync-pylock-toml.sh
popd
Comment on lines 477 to 481
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Good integration point; verify toolchain reproducibility

Hooking sync-pylock-toml.sh after requirements regeneration is sensible. Given the script auto-installs uv from PyPI, please confirm your CI runners permit networked tool bootstrapping and that the uv version is pinned to avoid churn in generated pylock.toml.

To verify determinism across lanes:


🏁 Script executed:

#!/bin/bash
# Expect identical pylock.toml across two uv versions on the same inputs (or document differences).
rg -n --glob 'jupyter/**/pylock.toml' || true

Length of output: 443


🏁 Script executed:

#!/bin/bash
# Locate and inspect the sync-pylock-toml.sh script for uv version pinning
rg -nF "sync-pylock-toml.sh" -l
sed -n '1,200p' scripts/sync-pylock-toml.sh

Length of output: 1190


Pin uv version in the bootstrap step and verify CI access

The sync-pylock-toml.sh script bootstraps uv with an unpinned pip install uv, which risks variability in the generated pylock.toml as new uv releases come out. To ensure reproducible builds:

• Confirm that your CI runners permit networked pip install uv during the sync step.
• In scripts/sync-pylock-toml.sh, pin the uv version in the install command. For example:

- uv --version || pip install uv
+ uv --version || pip install uv==<fixed-version>

Replace <fixed-version> with the vetted uv release used in your last successful run.
• (Optional) Document this pinned version in your Makefile or CI configuration, so future updates aren’t applied accidentally without review.

These changes will lock the toolchain and prevent unintended drift in your lockfiles.

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents
In Makefile around lines 477 to 481, the bootstrap step calls
scripts/sync-pylock-toml.sh which installs uv without pinning, causing
non-reproducible pylock.toml; update scripts/sync-pylock-toml.sh to replace the
unpinned pip install uv with a pinned version (use the last-vetted release as
<fixed-version>), ensure CI runners are allowed to perform networked pip
installs during this sync step (or add a job credential/mirror if not), and add
a short note in the Makefile or CI config documenting the pinned uv version so
future updates require explicit review.


# This is only for the workflow action
Expand Down
210 changes: 105 additions & 105 deletions codeserver/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

210 changes: 105 additions & 105 deletions codeserver/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

210 changes: 105 additions & 105 deletions codeserver/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

210 changes: 105 additions & 105 deletions codeserver/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/datascience/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

4,047 changes: 4,047 additions & 0 deletions jupyter/datascience/ubi9-python-3.11/pylock.toml

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/datascience/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/datascience/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

4,047 changes: 4,047 additions & 0 deletions jupyter/datascience/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/datascience/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

216 changes: 108 additions & 108 deletions jupyter/minimal/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

1,756 changes: 1,756 additions & 0 deletions jupyter/minimal/ubi9-python-3.11/pylock.toml

Large diffs are not rendered by default.

216 changes: 108 additions & 108 deletions jupyter/minimal/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

216 changes: 108 additions & 108 deletions jupyter/minimal/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

1,756 changes: 1,756 additions & 0 deletions jupyter/minimal/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

216 changes: 108 additions & 108 deletions jupyter/minimal/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

680 changes: 351 additions & 329 deletions jupyter/pytorch+llmcompressor/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

4,716 changes: 4,716 additions & 0 deletions jupyter/pytorch+llmcompressor/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

488 changes: 255 additions & 233 deletions jupyter/pytorch+llmcompressor/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/pytorch/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

4,283 changes: 4,283 additions & 0 deletions jupyter/pytorch/ubi9-python-3.11/pylock.toml

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/pytorch/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/pytorch/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

4,283 changes: 4,283 additions & 0 deletions jupyter/pytorch/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/pytorch/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/rocm/pytorch/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

4,127 changes: 4,127 additions & 0 deletions jupyter/rocm/pytorch/ubi9-python-3.11/pylock.toml

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/rocm/pytorch/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/rocm/pytorch/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

4,127 changes: 4,127 additions & 0 deletions jupyter/rocm/pytorch/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/rocm/pytorch/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/rocm/tensorflow/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

4,069 changes: 4,069 additions & 0 deletions jupyter/rocm/tensorflow/ubi9-python-3.11/pylock.toml

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/rocm/tensorflow/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

234 changes: 117 additions & 117 deletions jupyter/rocm/tensorflow/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

4,069 changes: 4,069 additions & 0 deletions jupyter/rocm/tensorflow/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

234 changes: 117 additions & 117 deletions jupyter/rocm/tensorflow/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/tensorflow/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

4,444 changes: 4,444 additions & 0 deletions jupyter/tensorflow/ubi9-python-3.11/pylock.toml

Large diffs are not rendered by default.

228 changes: 114 additions & 114 deletions jupyter/tensorflow/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

234 changes: 117 additions & 117 deletions jupyter/tensorflow/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

4,421 changes: 4,421 additions & 0 deletions jupyter/tensorflow/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

234 changes: 117 additions & 117 deletions jupyter/tensorflow/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/trustyai/ubi9-python-3.11/Pipfile.lock

Large diffs are not rendered by default.

4,553 changes: 4,553 additions & 0 deletions jupyter/trustyai/ubi9-python-3.11/pylock.toml

Large diffs are not rendered by default.

222 changes: 111 additions & 111 deletions jupyter/trustyai/ubi9-python-3.11/requirements.txt

Large diffs are not rendered by default.

234 changes: 117 additions & 117 deletions jupyter/trustyai/ubi9-python-3.12/Pipfile.lock

Large diffs are not rendered by default.

4,553 changes: 4,553 additions & 0 deletions jupyter/trustyai/ubi9-python-3.12/pylock.toml

Large diffs are not rendered by default.

234 changes: 117 additions & 117 deletions jupyter/trustyai/ubi9-python-3.12/requirements.txt

Large diffs are not rendered by default.

293 changes: 287 additions & 6 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,18 @@ name = "notebooks"
version = "2025.1"
description = "Open Data Hub / OpenShift AI Notebook / Workbench images, and tests for the same in Python."
readme = "README.md"
package-mode = false
requires-python = ">=3.12,<3.13"
requires-python = ">=3.11,<3.13"

# https://docs.astral.sh/uv/concepts/projects/dependencies/#managing-dependencies
dependencies = []
# WARNING: Do NOT attempt `uv lock --universal` (the default) resolution on this pyproject.toml.
# It would work (had the ` conflics ` section were defined), but it would run 10+ minutes and produce a huge `uv.lock` file (60MiB+).
# Instead, use the `scripts/sync-requirements-txt.sh` script which runs in seconds.

[dependency-groups]

############################
# Python Dependency Groups #
############################

dev = [
"pre-commit",
"pyright",
Expand All @@ -32,11 +37,287 @@ dev = [
"openshift-python-wrapper",
]

base = [
"wheel~=0.45.1",
"setuptools~=78.1.1",
]

jupyter-base = [
"jupyterlab==4.4.4",
"jupyter-server~=2.16.0",
"jupyter-server-proxy~=4.4.0",
"jupyter-server-terminals~=0.5.3",
"jupyterlab-git~=0.51.1",
"nbdime~=4.0.2",
"nbgitpuller~=1.2.2",
]

elyra-base = [
"odh-elyra==4.2.3",
"jupyterlab-lsp~=5.1.1",
"jupyterlab-widgets~=3.0.15",
"jupyter-resource-usage~=1.1.1",
]

elyra-preferred = [
"jupyter-bokeh~=4.0.5",
]

elyra-trustyai = [
"jupyter-bokeh~=3.0.5", # trustyai 0.6.1 depends on jupyter-bokeh~=3.0.5
]

db-connectors = [
"pymongo~=4.11.2",
"psycopg~=3.2.5",
"pyodbc~=5.2.0",
"mysql-connector-python~=9.3.0",
]

# onnxconverter-common ~=1.13.0 required for skl2onnx, as upgraded version is not compatible with protobuf
datascience-base = [
"boto3~=1.37.8",
"kafka-python-ng~=2.2.3",
"kfp~=2.12.1",
"plotly~=6.0.0",
"scipy~=1.15.2",
"skl2onnx~=1.18.0",
"onnxconverter-common~=1.13.0",
"kubeflow-training==1.9.0",
]

codeflare = [
"codeflare-sdk~=0.30.0",
]

datascience-preferred = [
"matplotlib~=3.10.1",
"numpy~=2.2.3",
"pandas~=2.2.3",
"scikit-learn~=1.6.1",
]

datascience-tensorflow = [
"matplotlib~=3.10.1",
"numpy~=1.26.4",
"pandas~=2.2.3",
"scikit-learn~=1.6.1",
]

datascience-trustyai = [
"matplotlib~=3.6.3",
"numpy~=1.24.1",
"pandas~=1.5.3",
"scikit-learn~=1.7.0"
]

tensorflowcuda= [
"tensorflow[and-cuda]~=2.18.0",
"tensorboard~=2.18.0",
"tf2onnx~=1.16.1",
]
tensorflowrocm = [
"tensorflow-rocm~=2.18.1",
"tensorboard~=2.18.0",
"tf2onnx~=1.16.1",
]
pytorchcuda = [
"tensorboard~=2.19.0",
"torch==2.6.0",
"torchvision==0.21.0",
]
pytorchrocm = [
"tensorboard~=2.18.0",
"torch==2.6.0",
"torchvision==0.21.0",
"pytorch-triton-rocm~=3.2.0",
]
llmcompressor = [
"vllm~=0.8.5",
"llmcompressor~=0.6.0",
"lm-eval~=0.4.8",
"loguru",
"pyyaml>=5.0.0",
"requests>=2.0.0",
"tqdm>=4.0.0",
"transformers>4.0,<5.0",
"datasets",
"accelerate>=0.20.3,!=1.1.0",
"pynvml",
"pillow",
"compressed-tensors",
]
trustyai = [
"torch==2.6.0",
"transformers~=4.53.0; python_version == '3.11'",
"transformers~=4.55.0; python_version == '3.12'",
"datasets~=3.4.1",
"accelerate~=1.5.2",
"trustyai~=0.6.1",
]

#########################
# Workbench Image Groups #
#########################

# https://docs.astral.sh/uv/concepts/projects/workspaces/#when-not-to-use-workspaces

jupyter-minimal-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
]

jupyter-datascience-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
{ include-group = "elyra-base" },
{ include-group = "elyra-preferred" },
{ include-group = "datascience-base" },
{ include-group = "datascience-preferred" },
{ include-group = "codeflare" },
{ include-group = "db-connectors" },
]

jupyter-tensorflow-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
{ include-group = "elyra-base" },
{ include-group = "elyra-preferred" },
{ include-group = "datascience-base" },
{ include-group = "datascience-tensorflow" },
{ include-group = "codeflare" },
{ include-group = "db-connectors" },
{ include-group = "tensorflowcuda" },
]

jupyter-tensorflow-rocm-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
{ include-group = "elyra-base" },
{ include-group = "elyra-preferred" },
{ include-group = "datascience-base" },
{ include-group = "datascience-tensorflow" },
{ include-group = "codeflare" },
{ include-group = "db-connectors" },
{ include-group = "tensorflowrocm" },
]

jupyter-pytorch-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
{ include-group = "elyra-base" },
{ include-group = "elyra-preferred" },
{ include-group = "datascience-base" },
{ include-group = "datascience-preferred" },
{ include-group = "codeflare" },
{ include-group = "db-connectors" },
{ include-group = "pytorchcuda" },
]

jupyter-pytorch-rocm-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
{ include-group = "elyra-base" },
{ include-group = "elyra-preferred" },
{ include-group = "datascience-base" },
{ include-group = "datascience-preferred" },
{ include-group = "codeflare" },
{ include-group = "db-connectors" },
{ include-group = "pytorchrocm" },
]

jupyter-pytorch-llmcompressor-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
{ include-group = "elyra-base" },
{ include-group = "elyra-preferred" },
{ include-group = "datascience-base" },
{ include-group = "datascience-tensorflow" },
{ include-group = "db-connectors" },
{ include-group = "llmcompressor" },
{ include-group = "pytorchcuda" },
]

jupyter-trustyai-image = [
{ include-group = "base" },
{ include-group = "jupyter-base" },
{ include-group = "elyra-base" },
{ include-group = "elyra-trustyai" },
{ include-group = "datascience-base" },
{ include-group = "datascience-trustyai" },
{ include-group = "codeflare" },
{ include-group = "db-connectors" },
{ include-group = "trustyai" },
{ include-group = "pytorchcuda" },
]

# https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-sources
[tool.uv.sources]

# NOTE: it is important to specify the `index` for the top-level groups, the ones used in the final resolution.
# Index values do not inherit from a lower-level group to the one where it is included.

# https://docs.astral.sh/uv/guides/integration/pytorch/#using-uv-with-pytorch
torch = [
{ index = "pytorch-cuda", group = "jupyter-pytorch-image" },
{ index = "pytorch-cuda", group = "jupyter-pytorch-llmcompressor-image" },
{ index = "pytorch-cuda", group = "jupyter-trustyai-image" },

{ index = "pytorch-rocm", group = "jupyter-pytorch-rocm-image" },
]
torchvision = [
{ index = "pytorch-cuda", group = "jupyter-pytorch-image" },
{ index = "pytorch-cuda", group = "jupyter-pytorch-llmcompressor-image" },
{ index = "pytorch-cuda", group = "jupyter-trustyai-image" },

{ index = "pytorch-rocm", group = "jupyter-pytorch-rocm-image" },
]
pytorch-triton-rocm = [
{ index = "pytorch-rocm", group = "jupyter-pytorch-rocm-image" },
]
tensorflow-rocm = [
{ url = "https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4/tensorflow_rocm-2.18.1-cp312-cp312-manylinux_2_28_x86_64.whl", group = "jupyter-tensorflow-rocm-image" },
]

# https://docs.astral.sh/uv/concepts/indexes/#package-indexes
# TODO(jdanek): explicit = false, otherwise `uv pip compile --emit-index-url` wont emit it
# also see https://github.com/astral-sh/uv/issues/10008, https://github.com/astral-sh/uv/issues/15534
[[tool.uv.index]]
name = "pytorch-cuda"
url = "https://download.pytorch.org/whl/cu126"
explicit = true

[[tool.uv.index]]
name = "pytorch-rocm"
url = "https://download.pytorch.org/whl/rocm6.2.4"
explicit = true

[[tool.uv.index]]
name = "pypi"
url = "https://pypi.org/simple/"
explicit = true

[[tool.uv.dependency-metadata]]
name = "tf2onnx"
version = "1.16.1"
requires-dist = ["protobuf"]

[[tool.uv.dependency-metadata]]
name = "vllm"
version = "0.8.5"
requires-dist = ["compressed-tensors"]

[[tool.uv.dependency-metadata]]
name = "tensorflow-rocm"
version = "2.18.1"
requires-dist = []

[tool.uv]
package = false

# https://docs.astral.sh/uv/concepts/resolution/#platform-specific-resolution
environments = [
"sys_platform == 'darwin'",
"sys_platform == 'linux'",
"sys_platform == 'linux' and implementation_name == 'cpython'",
]

# https://github.com/astral-sh/uv/issues/3957#issuecomment-2659350181
Expand Down
Loading
Loading