Adopt a "latest stable" `numba-cuda` pinning strategy

[brandon-b-miller]

Summarizing various discussions over the past few days here.

Historically we've needed to advance the lower bound quite often as numba-cuda features were created to serve cuDF specifically, while maintaining a tight upper bound due to the instability of those APIs as we got things working, bumping as needed. As more libraries that RAPIDS wants compatibility with have adopted numba-cuda however, it's become common for those libraries to be ahead of where rapids pins numba-cuda. This makes it difficult for those libraries to be coinstalled with the latest RAPIDS version. After discussion with the cudf python and numba-cuda teams, the best idea moving forward seem to be to approach the numba-cuda dependency as follows:

• Open the upper bound for numba-cuda for regular development, allowing RAPIDS to be worked on against the latest stable numba-cuda release
• At release time, add an upper bound restricting that RAPIDS release to, at latest, the most recent release of numba-cuda that exists

We won't bump the lower bound unless necessary and we'll respond to issues with numba-cuda that surface through rapids with patch releases, these will hopefully be relatively infrequent and addressable quickly.

To get this off the ground, let's migrate rapids to >=0.24.0,<0.25.0, the lower bound getting us a version containing the latest import warning fixes and the upper bound mimicking what we'd have if things were running automatically as this proposal envisions. After that, we'll:

• Update release scripts to figure out the latest numba-cuda version and adjust the bounds on release
• Remove the upper bound on development branches

PRs doing the former:
rapidsai/cudf#21117
rapidsai/cuml#7706
rapidsai/dask-cuda#1610
rapidsai/cuxfilter#759
rapidsai/rmm#2218
rapidsai/ucxx#573

cc @bdice @vyasr @jameslamb

[vyasr]

I think that over the course of the 26.04 development cycle if there are multiple numba-cuda minor releases and we don't find that any of them breaks us we can revisit at release time whether we need to pin or not.

[brandon-b-miller]

@vyasr do we want to get this in for the 26.02 RAPIDS release? cuDF is in code freeze, the other libraries are in burndown. It's a dependency rather than code but if we want it for this release I think we'll need to retarget PRs to 26.02 branches.

[vyasr]

It would be nice to get the 0.24 upgrade into 26.02 to help our CCCL friends, but I'll defer to you on how heavy a lift you think it is. Either way we should unpin on main (for 26.04) ASAP.

[brandon-b-miller]

It's not a ton of work, essentially just merging the open PRs in order and then a final cleanup PR to dask-cuda, but I lean towards leaving things as is for 26.02. This release began with a numba-cuda upgrade meant to close the cccl compatibility gap. It became clear over the release that churning the numba-cuda dep's narrow range to keep up with other libraries isn't sustainable with the speed those other libraries to developing numba-cuda for their own needs.

I think arriving at the above, hopefully more general solution is a pretty good outcome for this release even if we don't end it with the latest numba-cuda compatibility.

[jameslamb]

Implementation note: we just merged this series of PRs changing the spelling of the numba-cuda dependency in dependencies.yaml, mainly for the benefit of RAPIDS devcontainers / DLFW builds:

• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true rmm#2227
• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true cudf#21185
• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true cuml#7721
• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true ucxx#576

Regardless of which release we target, PRs for this issue will need to be updated (they'll have merge conflicts now). Please preserve the fallback to numba-cuda (no extra) added in those PRs.

[pentschev]

Given we're in code freeze already I agree with Brandon, I'd prefer to punt it to 26.04. Given other internal projects we've been currently working on, even though odds seem to be low everyone is currently short on time to deal with potential fallout from changes like this so close to release.

[brandon-b-miller]

Implementation note: we just merged this series of PRs changing the spelling of the numba-cuda dependency in dependencies.yaml, mainly for the benefit of RAPIDS devcontainers / DLFW builds:

• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true rmm#2227
• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true cudf#21185
• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true cuml#7721
• Fallback to numba-cuda with no extra CUDA packages if 'cuda_suffixed' isn't true ucxx#576

Regardless of which release we target, PRs for this issue will need to be updated (they'll have merge conflicts now). Please preserve the fallback to numba-cuda (no extra) added in those PRs.

@jameslamb does dask-cuda need this change also?

[brandon-b-miller]

Per the above discussion, I'll migrate PRs (currently targetting main) to remove the upper bound and handle the task of updating release scripts after those are passing and merged.