[DNR][deps] generating data depsets by elliot-barn · Pull Request #60616 · ray-project/ray

elliot-barn · 2026-01-30T20:22:07Z

Generating data depsets for data ci images

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

gemini-code-assist

Code Review

This pull request refactors the data CI image build process to use depsets, which is a good step towards better dependency management. It also includes several dependency version upgrades. I've found a couple of critical issues in the Wanda configuration files that will likely cause build failures. There's also a minor improvement that can be made to the new Dockerfile for clarity and best practices.

gemini-code-assist · 2026-01-30T20:24:07Z

ci/docker/data.base.build.wanda.yaml

@@ -0,0 +1,9 @@
+name: "databuild-py$PYTHON"
+froms: ["cr.ray.io/rayproject/oss-ci-base_ml-py$PYTHON"]
+dockerfile: ci/docker/data.build.Dockerfile


The dockerfile path appears to be incorrect. It's pointing to ci/docker/data.build.Dockerfile, but given the changes in this PR, it should probably point to the new ci/docker/data.base.build.Dockerfile to use the new depset-based installation logic.

dockerfile: ci/docker/data.base.build.Dockerfile

ci/docker/datamongo.build.wanda.yaml

gemini-code-assist · 2026-01-30T20:24:07Z

.buildkite/data.rayci.yml

  - name: databuild-multipy
    label: "wanda: databuild-py{{matrix}}"
-    wanda: ci/docker/data.build.wanda.yaml
+    wanda: ci/docker/data.base.build.wanda.yaml


The wanda configuration file path has been updated. Please ensure that the new file ci/docker/data.base.build.wanda.yaml correctly points to the new ci/docker/data.base.build.Dockerfile. I've left a specific comment on that file regarding a potential mismatch.

gemini-code-assist · 2026-01-30T20:24:07Z

ci/docker/data.base.build.Dockerfile

+
+SHELL ["/bin/bash", "-ice"]
+
+COPY . .


The COPY . . command seems redundant as the required lock file is already explicitly copied on line 10. This broad COPY could unintentionally bring in other files from the build context, potentially increasing image size or causing unexpected behavior. For better clarity, hermeticity, and caching, it's best to remove this line if it's not strictly necessary for other files not declared in the wanda.yaml srcs.

cursor · 2026-01-30T20:26:47Z

ci/docker/datamongo.build.wanda.yaml

 name: "datamongobuild-py$PYTHON"
 froms: ["cr.ray.io/rayproject/oss-ci-base_ml-py$PYTHON"]
-dockerfile: ci/docker/data.build.Dockerfile
+dockerfile: t


Dockerfile path set to invalid value t

High Severity

The dockerfile field in datamongo.build.wanda.yaml is changed to t, which is not a valid Dockerfile path. This appears to be an accidental edit (possibly partial deletion). The previous value was ci/docker/data.build.Dockerfile. This will cause the datamongobuild-multipy CI build step to fail when trying to locate the Dockerfile.

ci/docker/data.base.build.wanda.yaml

cursor · 2026-01-30T20:26:47Z

ci/docker/data.base.build.wanda.yaml

+srcs:
+  - python/deplocks/ci/data_base_depset_py$PYTHON.lock
+build_args:
+  - DOCKER_IMAGE_BASE_BUILD=cr.ray.io/rayproject/oss-ci-base_ml-py$PYTHON


Missing PYTHON_DEPSET build argument in wanda config

Medium Severity

The data.base.build.wanda.yaml does not pass PYTHON_DEPSET as a build argument, so the Dockerfile would use its default value. However, the default python/deplocks/ci/data_base_depset_py$PYTHON.lock references $PYTHON which is never defined as a Dockerfile ARG, causing it to expand to an empty string. The resulting path data_base_depset_py.lock (without version) won't match the actual lock file. The similar datatfds.build.wanda.yaml correctly passes PYTHON_DEPSET explicitly.

Additional Locations (1)

ci/docker/data.base.build.Dockerfile#L7-L8

cursor · 2026-01-30T20:26:47Z

ci/raydepsets/configs/ci_data.depsets.yaml

+      - python/requirements/ml/data-test-requirements.txt
+      - python/requirements/data/data-base.txt
+    constraints:
+      - /tmp/ray-deps/requirements_compiled_py3.13.txt


Hardcoded Python 3.13 constraints for 3.10/3.12 builds

Medium Severity

The constraints path hardcodes requirements_compiled_py3.13.txt but the depset builds target Python 3.10 and 3.12 (via build_arg_sets). All other depset configs in rayimg.depsets.yaml use ${PYTHON_VERSION} to match the constraints file to the target Python version. Using 3.13 constraints for 3.10/3.12 builds could result in dependency versions that are incompatible with those Python versions.

cursor · 2026-01-30T20:26:47Z

ci/docker/data.base.build.Dockerfile

+if [[ -n "$ARROW_MONGO_VERSION" ]]; then
+  # Older versions of Arrow Mongo require an older version of NumPy.
+  pip install numpy==1.23.5
+fi


Dead code conditional branches never execute

Low Severity

The conditional blocks for ARROW_MONGO_VERSION and RAY_CI_JAVA_BUILD will never execute because the corresponding data.base.build.wanda.yaml doesn't pass these build args. Both ARGs default to empty, so the if [[ -n "$ARROW_MONGO_VERSION" ]] and if [[ $RAY_CI_JAVA_BUILD == 1 ]] conditions always evaluate to false. This appears to be code copied from data.build.Dockerfile that isn't needed for this simpler depset-based build.

Additional Locations (1)

ci/docker/data.base.build.Dockerfile#L32-L37

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

cursor · 2026-01-30T22:06:02Z

ci/docker/data.base.build.Dockerfile

+
+ARG ARROW_MONGO_VERSION=
+ARG RAY_CI_JAVA_BUILD=
+ARG PYTHON_DEPSET=python/deplocks/ci/data_base_depset_py$PYTHON.lock


Missing ARG declaration causes empty variable expansion

High Severity

The PYTHON_DEPSET ARG default value uses $PYTHON, but PYTHON is never declared as an ARG in this Dockerfile. Without an ARG PYTHON declaration, the variable expands to an empty string, causing the path to resolve to python/deplocks/ci/data_base_depset_py.lock (missing the version number). The wanda file also doesn't pass PYTHON or PYTHON_DEPSET as a build argument to override this default.

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

aslonnie

let me know when this is ready for review.

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…/ray-project/ray into elliot-barn-data-ci-dep-upgrades

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Fixes PerReplica -> tf.Tensor conversion error with TF 2.20 / Keras 3 by forcing tf.keras to use the Keras 2 API via tf-keras. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pytest 8.2+ removed support for pytest.warns(None). Use warnings.catch_warnings(record=True) instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

downloads.apache.org is a CDN mirror redirector that inconsistently serves old releases like Hadoop 3.2.4, causing CI Docker builds to fail silently. archive.apache.org is the permanent canonical archive. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

With set -eo pipefail, `yes` receives SIGPIPE (exit 141) when hadoop closes the pipe after finishing. Replace with `echo "Y"` which sends one confirmation and exits cleanly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The GPU doctest target was missing this env var, causing TF 2.16+ to initialize Keras 3 which conflicts with tf_keras (Keras 2) imports in batch_inference.rst and transforming-data.rst. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

elliot-barn added 7 commits January 20, 2026 21:00

upgrading uv and replacing deprecated uv index flags

476ec03

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating the index strategy

fee06c9

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating uv darwin version

393db2a

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

merging master

720f2ba

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

reverting index strategy

c39a963

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating test

1a4f338

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

generating base depsets

b65c1c6

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

elliot-barn requested review from a team, matthewdeng and richardliaw as code owners January 30, 2026 20:22

elliot-barn added the go add ONLY when ready to merge, run all tests label Jan 30, 2026

gemini-code-assist bot reviewed Jan 30, 2026

View reviewed changes

cursor bot reviewed Jan 30, 2026

View reviewed changes

elliot-barn added 2 commits January 30, 2026 21:06

relaxing requirements

1cb24c2

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

relaxing dask

10e675c

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

cursor bot reviewed Jan 30, 2026

View reviewed changes

aslonnie marked this pull request as draft January 31, 2026 00:10

elliot-barn added 3 commits February 2, 2026 20:34

updating dependency

e94dac5

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

relaxing requirement

aba7ddb

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Merge branch 'master' into elliot-barn-data-ci-dep-upgrades

7936af1

aslonnie reviewed Feb 3, 2026

View reviewed changes

elliot-barn added 9 commits February 3, 2026 00:11

updating reqs

b4c41ef

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Merge branch 'elliot-barn-data-ci-dep-upgrades' of https://github.com…

c661a28

…/ray-project/ray into elliot-barn-data-ci-dep-upgrades

updating requirements compiled

01a8925

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating lock files

b39af16

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

pinning jaxlib

631c99e

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

upgrading pandas on 3.12

b58ef63

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

data doc depset

04a7f01

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

installing tf keras for tensorflow compatibility

ed7146d

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating lock file path

8caf5cf

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

elliot-barn and others added 18 commits February 5, 2026 20:07

adding new depsets

ce2a544

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

fixing conflicts

70812d1

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Merge branch 'uv-index-fix' into elliot-barn-data-ci-dep-upgrades

35cfff0

creating and using depsets

10fd1e3

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

removing duplicate build definition

5ddf172

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

adding pyarrow v9 depset

d1791b5

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

removing duplicate spec

1b7f098

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

renaming wanda file

6eb1836

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating deps

b02bc8b

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

creating mongo depset

3fd854e

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating deps

7fbd5f8

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

set TF_USE_LEGACY_KERAS=1 for test_tf and test_tfrecords

651cf13

Fixes PerReplica -> tf.Tensor conversion error with TF 2.20 / Keras 3 by forcing tf.keras to use the Keras 2 API via tf-keras. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix pytest.warns(None) deprecation in test_numpy_to_tensor_warning

8a705a3

pytest 8.2+ removed support for pytest.warns(None). Use warnings.catch_warnings(record=True) instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fixing hadoop install and updating deps

ad1b600

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

recompiling dependencies

98ae5c8

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DNR][deps] generating data depsets#60616

[DNR][deps] generating data depsets#60616
elliot-barn wants to merge 39 commits intomasterfrom
elliot-barn-data-ci-dep-upgrades

elliot-barn commented Jan 30, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jan 30, 2026

Uh oh!

Uh oh!

gemini-code-assist bot Jan 30, 2026

Uh oh!

gemini-code-assist bot Jan 30, 2026

Uh oh!

cursor bot Jan 30, 2026

Uh oh!

Uh oh!

cursor bot Jan 30, 2026

Uh oh!

cursor bot Jan 30, 2026

Uh oh!

cursor bot Jan 30, 2026

Uh oh!

cursor bot left a comment

Uh oh!

cursor bot Jan 30, 2026

Uh oh!

aslonnie left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		SHELL ["/bin/bash", "-ice"]

		COPY . .

Conversation

elliot-barn commented Jan 30, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist bot Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

cursor bot Jan 30, 2026

Choose a reason for hiding this comment

Dockerfile path set to invalid value t

Uh oh!

Uh oh!

cursor bot Jan 30, 2026

Choose a reason for hiding this comment

Missing PYTHON_DEPSET build argument in wanda config

Uh oh!

cursor bot Jan 30, 2026

Choose a reason for hiding this comment

Hardcoded Python 3.13 constraints for 3.10/3.12 builds

Uh oh!

cursor bot Jan 30, 2026

Choose a reason for hiding this comment

Dead code conditional branches never execute

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Jan 30, 2026

Choose a reason for hiding this comment

Missing ARG declaration causes empty variable expansion

Uh oh!

aslonnie left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Dockerfile path set to invalid value `t`

Missing `PYTHON_DEPSET` build argument in wanda config