how to get rapids to resolve?

[geoHeil]

q1 - how to fix the channel priority issue

I somehow get an issue with channel priority:

failed to solve the conda requirements of 'research-cuda' 'linux-64'
  ╰─▶ Cannot solve the request because of: The following packages are incompatible
      └─ rapids ~=24.8 cannot be installed because there are no viable options:
         └─ rapids 24.8.0 | 24.8.0 | 24.8.0 | 24.8.0 | 24.8.0 | 24.8.0 would require
            └─ dask-cuda 24.8.*, which cannot be installed because there are no viable options:
               └─ dask-cuda 24.8.0 | 24.8.0 | 24.8.0 | 24.8.1 | 24.8.1 | 24.8.1 | 24.8.2 | 24.8.2 | 24.8.2 is excluded because due to strict channel priority not using this option from: 'https://conda.anaconda.org/rapidsai/'

my pixi file looks like:

[dependencies]
python = "3.11"
pip = ">=24.2"

[environments]
research-cuda = {features = ["research-cuda", "research-basics"], solve-group = "default"}

[feature.research-basics.dependencies]
pytorch = {version = ">=2.4.0,<2.5", channel = "pytorch"}
torchaudio = {version = ">=2.4.0,<2.5", channel = "pytorch"}
torchvision = {version = ">=0.19.0,<0.20", channel = "pytorch"}


[feature.research-cuda]
channels = [  
    {channel = "rapidsai", priority = 1},
    {channel = "nvidia", priority = 2},
    {channel = "pytorch", priority = 3},
    {channel = "conda-forge", priority = 4},
    ]
[feature.research-cuda.target.linux-64.dependencies]
pytorch-cuda = "~=12.4"

cuda-version = {version = ">=12.0,<=12.5", channel = "rapidsai"}
rapids = {version = "~=24.08", channel = "rapidsai"}

[feature.research-cuda.system-requirements]
cuda = "12"

[project]
name = "foo"
version = "1.0.0"
authors = [
  "bar"
]
platforms = ["linux-64", "osx-arm64"]
channels = ["conda-forge", "pytorch"]

how can I fix this?

Various attempts to modify channel priority have failed so far for me.

q2 how to refine the pytorch import

According to nvidia: 'pytorch=*=*cuda*' https://docs.rapids.ai/install this is way way to install pytorch - but as you can see the CPU environment is hardwiring the env to pytorch. How can this be fixed? Should I rely solely on channel priority?

[geoHeil]

Relacing cuda-version leads to - conda: https://conda.anaconda.org/conda-forge/noarch/cuda-version-12.4-h3060b56_3.conda being pulled in - but as far as I understand the right one is not in noarch but in rapidsai.

[geoHeil]

Pinning dask-cuda = {version = "~=24.08", channel = "rapidsai"} leads to the same issue for rmm. something with the channel priority must be wrong.

And assuming higher is better changing priority like channels = [
{channel = "rapidsai", priority = 4},
{channel = "nvidia", priority = 3},
{channel = "pytorch", priority = 2},
{channel = "conda-forge", priority = 1},
]

leads to same error for cupy - but not it is trying to pull from conda-forge

[geoHeil]

Something like this

[dependencies]
python = "3.11"
pip = ">=24.2"

[environments]
research-cuda = {features = ["research-cuda", ], solve-group = "default"}

[feature.research-cuda]
channels = [  
    {channel = "rapidsai", priority = 4},
    {channel = "nvidia", priority = 3},
    {channel = "pytorch", priority = 2},
    {channel = "conda-forge", priority = 5},
    ]
[feature.research-cuda.target.linux-64.dependencies]
#pytorch-cuda = "~=12.4"

cuda-version = {version = ">=12.0,<=12.5"}
rapids = {version = "~=24.08", channel = "rapidsai"}
dask-cuda = {version = "~=24.08", channel = "rapidsai"}
rmm = {version = "~=24.08", channel = "rapidsai"}
librmm = {version = "~=24.08", channel = "rapidsai"}
cupy = {version = "~=12.3.0", channel = "conda-forge"}
libcugraph = {version = "~=24.08", channel = "rapidsai"}
libxgboost = {version = "~=2.1.1", channel = "rapidsai"}
xgboost = {version = "~=2.1.1", channel = "rapidsai"}
py-xgboost = {version = "~=2.1.1", channel = "rapidsai"}


[feature.research-cuda.system-requirements]
cuda = "12"

[project]
name = "foo"
version = "1.0.0"
authors = [
  "bar"
]
platforms = ["linux-64", "osx-arm64"]
channels = ["conda-forge", "pytorch"]

seems to resolve but:

• neither this is as simple as conda create -n rapids-24.08 -c rapidsai -c conda-forge -c nvidia rapids=24.08 python=3.11 'cuda-version>=12.0,<=12.5' 'pytorch=*=*cuda*'
• nor is the channel priority retained that nvidia suggested. When trying to retain the priority even more (ucx, datashader, ...) manual pins seem to come up

[geoHeil]

The whole example would look like (see below).
It resolves at least. I do not know if it the right packages though.

Additionally - the following remains open:

According to nvidia: 'pytorch=*=cuda' https://docs.rapids.ai/install this is way way to install pytorch - but as you can see the CPU environment is hardwiring the env to pytorch. How can this be fixed? Should I rely solely on channel priority?

[dependencies]
python = "3.11"
pip = ">=24.2"

[environments]
research-cuda = {features = ["research-cuda", "research-basics"], solve-group = "default"}

[feature.research-basics.dependencies]
pytorch = {version = ">=2.4.0,<2.5", channel = "pytorch"}
torchaudio = {version = ">=2.4.0,<2.5", channel = "pytorch"}
torchvision = {version = ">=0.19.0,<0.20", channel = "pytorch"}



[feature.research-cuda]
channels = [  
    {channel = "rapidsai", priority = 4},
    {channel = "nvidia", priority = 3},
    {channel = "pytorch", priority = 2},
    {channel = "conda-forge", priority = 5},
    ]
[feature.research-cuda.target.linux-64.dependencies]
#pytorch-cuda = "~=12.4"

cuda-version = {version = ">=12.0,<=12.5"}
rapids = {version = "~=24.08", channel = "rapidsai"}
dask-cuda = {version = "~=24.08", channel = "rapidsai"}
rmm = {version = "~=24.08", channel = "rapidsai"}
librmm = {version = "~=24.08", channel = "rapidsai"}
cupy = {version = "~=12.3.0", channel = "conda-forge"}
libcugraph = {version = "~=24.08", channel = "rapidsai"}
libxgboost = {version = "~=2.1.1", channel = "rapidsai"}
xgboost = {version = "~=2.1.1", channel = "rapidsai"}
py-xgboost = {version = "~=2.1.1", channel = "rapidsai"}


[feature.research-cuda.system-requirements]
cuda = "12"

[project]
name = "foo"
version = "1.0.0"
authors = [
  "bar"
]
platforms = ["linux-64", "osx-arm64"]
channels = ["conda-forge", "pytorch"]

[ruben-arts]

He @geoHeil,

I've given the resolve a try, I've come up with this pixi.toml as a result. This solve for me locally and on first glance it looks to take the correct package from the right channels.

[dependencies]
python = "3.11"
pip = ">=24.2"

[environments]
research-cuda = {features = ["research-cuda", "research-basics"], solve-group = "default"}

[feature.research-basics.dependencies]
pytorch = {version = ">=2.4.0,<2.5", channel = "pytorch"}
torchaudio = {version = ">=2.4.0,<2.5", channel = "pytorch"}
torchvision = {version = ">=0.19.0,<0.20", channel = "pytorch"}

[feature.research-cuda]
# Take what you can from rapidsai, but if you can't find it there, try conda-forge, and then pytorch
# Nivia has moved all packages for cuda 12 to conda-forge
channels = [  
    {channel = "rapidsai", priority = 3},
    {channel = "conda-forge", priority = 2},
    {channel = "pytorch", priority = 1},
    ]
[feature.research-cuda.target.linux-64.dependencies]
pytorch-cuda = "~=12.4"
rapids = "~=24.08"

# These packages are available in rapidsai, but should be installed from conda-forge
cupy = { version = "*", channel = "conda-forge" }
ucx = { version = "*", channel = "conda-forge" }
datashader = { version = "*", channel = "conda-forge" }

[feature.research-cuda.system-requirements]
cuda = "12"

[project]
name = "foo"
platforms = ["linux-64", "osx-arm64"]
channels = ["conda-forge"]

I hope this is what you need.

Tips to concur the multi-channel problem.

• Don't use the nvidia channel anymore if you depend on cuda >12, that is the latest advice we got from the NVidia folks
• Pick channel you care most about as the first channel. Which is rapidsai for you.
• Only keep the dependencies you really required and let the solver figure out the rest for you.
• If your out of ideas, try to solve with adding channel-priority = "disabled" to the [project] table. This will allow the solver to ignore the priority and find "any" solution. USE WITH CAUTION as this might not give the result you expected.
• Check your pixi.lock after the solve to check for any packages that you didn't expect.

Reasons this was hard.

• We force channel priority, which is awesome if you have proper channels that work together and don't overwrite each others packages. But rapidsai does have some overwriting packages which are not used anymore so you do need to get them from conda-forge
• ABI compatibility is guaranteed by conda-forge but not between conda-forge and other channels. pixi (or any package manager) can not protect you from this issue that's one of the reasons we make the priority strict.
• In a perfect world there would be no rapidsai or pytorch channel. This is what Nvidia is working towards if I understood them correctly the last time we talked (SciPy 2024).

[geoHeil]

This mostly works.

2 questions:

• why is it still simpler on the plain nvidia side?
• how to handle the pytorch / rapids interop with cuda? Pytorch alone suggests: pytorch-cuda = "~=12.4" rapids suggests 'pytorch=*=*cuda*' plus cuda-version = {version = ">=12.0,<=12.5"}

[ruben-arts]

why is it still simpler on the plain Nvidia side

If you're talking about this conda command: conda create -n rapids-24.08 -c rapidsai -c conda-forge -c nvidia rapids=24.08 python=3.11 'cuda-version>=12.0,<=12.5' 'pytorch=*=*cuda*'.
Conda has one more channel-priority heuristic, which is called flexible. This will take the dependencies from the same channel when it's available. This is a good strategy for this occasion, as that is basically what we're doing by hand.

how to handle the pytorch / rapids interop with cuda?

Unfortunately, pytorch and Nvidia don't have a clear sync on how they handle the package location and names. This makes it harder for everyone. I know Nvidia is putting a big effort into improving this situation. Writing a clear manual right now will be outdated in a month. Nvidia will have to do that themselves.
Where moving everything to conda-forge is the key improvement.

pytorch is a channel that, as far as I know, also tries to work together with the main channel. Which doesn't work with conda-forge.
@baszalmstra from prefix-dev is working on getting pytorch for Windows working from conda-forge. We're also donating a monster machine to build Pytorch on as we speak.

So we've been aware of the situation, and together with the conda-forge community and Nvidia we're laying down the pavement to create a nice road to Cuda/pytorch/rapids installation paradise.