-
Notifications
You must be signed in to change notification settings - Fork 4
Description
RAPIDS is tentatively targeting 26.04 for Python 3.14. We have a few options for how we get there based on how we choose to proceed on a few open build projects, which this issue aims to describe in the context of their relevance for the new Python release.
Our typical approach for adding a new Python version is straightforward: we build new packages for each distribution mechanism (conda and wheels) and deploy them. Each new Python version adds some build and storage overhead until we drop a previous one. Since our GPU CI runners are at a premium, we typically avoid increasing the number of test jobs in PRs, instead choosing to distribute the same number of jobs more thinly across the new, expanded matrix. This approach works, but it means that each new Python version means additional work and a migration across all of RAPIDS. More importantly, it means that until this migration occurs and we can make a new release, our users cannot run our software on the new Python version. Historically we have not been particularly prompt with this, often lagging Python releases by multiple months.
Most of our dependencies tend to release new Python support significantly faster than us, so upgrading faster is typically an option. The main exception that we have been impacted by recently is nvcomp, which has added Python bindings but had not yet moved to a model of supporting new Python versions until well into the 3.13 release. Since no RAPIDS libraries currently use nvcomp Python bindings, we instead moved back to our legacy model of vendoring the nvcomp binaries directly into our packages (this was our model before nvcomp began publishing packages). Ideally we would like to stop vendoring (see #171), but not at the expense of blocking our ability to support new Python versions promptly. Our best option here would be adopting nvcomp's new packages that split out Python and C++ components completely, fully decoupling us from their Python support plans.
Another major consideration is that we have a potential path forward for supporting new Python versions without building new packages for each one. Now that we have successfully demonstrated that almost all of RAPIDS builds under the limited API, we may be able to switch to shipping a single package that works across different Python versions. However, there are significant unknowns remaining since we have yet to run anything more than the most basic of smoke test on these builds, and even that was only on the same Python version they were built with. Also, we must keep in mind that limited API builds are only relevant for extension modules. That being said, it is in our best interest to start testing limited API builds sooner rather than later so that we can smoke out any outstanding bugs in Cython's limited API support (which is where I anticipate any outstanding issues to be). There aren't many packages that have jumped on this bandwagon quite yet, and as part of even getting RAPIDS to build under the limited API I submitted multiple issues and PRs upstream. The sooner we can start doing that for runtime issues (if any) the sooner we will be able to leverage limited API builds. If that is not for the same release where we add 3.14, we could still do it in a subsequent release and benefit (even if it doesn't simplify the Python 3.14 support path).
For pure Python packages we have a simpler path forward (#43), but we also need to do some work there.
A final consideration is that we are also interested in adding support for free-threaded Python, and much of the work needed to enable that will also dovetail with the other projects that get us towards 3.14 support.
With all of this in mind, I propose that we front load as much work as possible with the aim of shipping 3.14 support using pure Python packages, limited API builds, and fully devendored nvcomp. Since all of these are projects that we want to support anyway, we might as well invest in them now. If we wind up needing to fall back to the easy path in a few months, we will still have made worthwhile progress towards our key goals. With that in mind, here is the sequence of other issues that we need to work through, in order of when they need to be done.
Approach
Pre-work: RAPIDS
For limited API/pure Python packages, we should do the following:
- Support more flexible selection of built package at test time #204
- Align conda and wheel building workflows #44
- Properly support building pure Python packages #43
- Build Python packages using the limited API #42
- Use libnvcomp C++ wheels #171
Migration
- ci-imgs
-
ci-conda(drop Python 3.10, add Python 3.14, upgrade miniforge ci-imgs#367) -
ci-wheel/citestwheel(Add Python 3.14 wheel CI images ci-imgs#359)
-
- shared-workflows
Post-migration
(copied from #120)
- Update the
ci-imgsrepo's latest configuration to use Python 3.14 - Update the
dockerrepo's build and test matrices, and references in docs - Update the build matrix in
release-scriptsso that Python 3.13 wheels are uploaded - update release selector in
docs - update all the repos to point at
mainonshared-workflowsagain - move
devcontainersPython version pin to Python 3.14 - delete the
python-3.14branch inshared-workflows - merge any remaining
rapids-reviserchanges