Let users decide which cupy to use

[relativityhd]

As of right now, there are three pip install packages which one can use to install cupy: cupy which will build cupy from source and the two prebuild versions cupy-cuda12x and cupy-cuda11x.
This library only allows for cupy to be installed. This clashes with other libraries, e.g. cucim, which only allows installing with prebuild options.
Also, having only the source building cupy option available introduces the risk of build-failures on systems which would work fine with the prebuild ones and costs time when installing this library.

Therefore, I would like to present two possible solutions.

First: Remove cupy dependency completely and change the installation instructions that the user must cupy by themselves.

Second: Remove cupy dependency from the dependency list and introduce 3 new optional dependencies:

pyproject.toml:

...
dependencies = [
    "cupy",
    "xarray>=2024.02.0",
]

[project.optional-dependencies]
test = [
    "dask",
    "pytest",
]
source = [
    "cupy"
]
cuda12 = [
    "cupy-cuda12x",
]
cuda11 = [
    "cupy-cuda11x",
]

...

That way, a user can install this library with e.g. pip install cupy-xarray[cuda12]

I would welcome any feedback and other ideas on this. If you make a decision about this, I would be happy to implement it and open a pull request.

[jacobtomlinson]

This is a very good point! I would personally lean towards option 1, but we should definitely add some runtime checking and a helpful error message if cupy is missing.

[msarahan]

Extras are kind of annoying because there's no default value. https://discuss.python.org/t/adding-a-default-extra-require-environment/4898

I have no opinion on what the best way forward on this particular library might be, but there is an effort towards expanding the metadata that can be captured and used for package selection. https://discuss.python.org/t/selecting-variant-wheels-according-to-a-semi-static-specification/53446/111 is one proposal. That has some implementation in the repos at https://github.com/wheel-next. That proposal depends on the concept of index priority as a way of avoiding complicating the current solver implementations. Because of that, we've had to also try to get index priority to be better accepted. https://discuss.python.org/t/pep-766-handling-multiple-indexes-index-priority/71589

It's highly unlikely that any of this will help solve your concern in the short or even medium term, but I hope you'll get involved in the upstream issues and proposals that I have linked if you are so inclined.

[dcherian]

Like Jacob, I lean towards (1) because it's less maintenance work. We should still import cupy and raise a nice error saying that cupy must be manually installed.

[relativityhd]

I opened a PR, it would be great if someone of you could review and (if everything is fine) merge it. :)

[weiji14]

Seems like there used to be a cupy-wheel metapackage (cupy/cupy#6012, cupy/cupy#6688) to handle exactly this, but this was removed recently in cupy v13 cupy/cupy#7628 😞

There might be a smart way to get setuptools to detect the installed CUDA version and grab the correct pre-built cupy binary wheel (https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#platform-specific-dependencies), but it might require us to write a custom environment marker which can be tricky (there was some discussion of this at python/peps#581). Let me take a look at the PR at #67.

---

Note to future self: Reopen this after PEP 771 – Default Extras for Python Software Packages is implemented, and pick a default cuda version.