Skip to content

Python 3.14 support #205

@vyasr

Description

@vyasr

RAPIDS is tentatively targeting 26.04 for Python 3.14. We have a few options for how we get there based on how we choose to proceed on a few open build projects, which this issue aims to describe in the context of their relevance for the new Python release.

Our typical approach for adding a new Python version is straightforward: we build new packages for each distribution mechanism (conda and wheels) and deploy them. Each new Python version adds some build and storage overhead until we drop a previous one. Since our GPU CI runners are at a premium, we typically avoid increasing the number of test jobs in PRs, instead choosing to distribute the same number of jobs more thinly across the new, expanded matrix. This approach works, but it means that each new Python version means additional work and a migration across all of RAPIDS. More importantly, it means that until this migration occurs and we can make a new release, our users cannot run our software on the new Python version. Historically we have not been particularly prompt with this, often lagging Python releases by multiple months.

Most of our dependencies tend to release new Python support significantly faster than us, so upgrading faster is typically an option. The main exception that we have been impacted by recently is nvcomp, which has added Python bindings but had not yet moved to a model of supporting new Python versions until well into the 3.13 release. Since no RAPIDS libraries currently use nvcomp Python bindings, we instead moved back to our legacy model of vendoring the nvcomp binaries directly into our packages (this was our model before nvcomp began publishing packages). Ideally we would like to stop vendoring (see #171), but not at the expense of blocking our ability to support new Python versions promptly. Our best option here would be adopting nvcomp's new packages that split out Python and C++ components completely, fully decoupling us from their Python support plans.

Another major consideration is that we have a potential path forward for supporting new Python versions without building new packages for each one. Now that we have successfully demonstrated that almost all of RAPIDS builds under the limited API, we may be able to switch to shipping a single package that works across different Python versions. However, there are significant unknowns remaining since we have yet to run anything more than the most basic of smoke test on these builds, and even that was only on the same Python version they were built with. Also, we must keep in mind that limited API builds are only relevant for extension modules. That being said, it is in our best interest to start testing limited API builds sooner rather than later so that we can smoke out any outstanding bugs in Cython's limited API support (which is where I anticipate any outstanding issues to be). There aren't many packages that have jumped on this bandwagon quite yet, and as part of even getting RAPIDS to build under the limited API I submitted multiple issues and PRs upstream. The sooner we can start doing that for runtime issues (if any) the sooner we will be able to leverage limited API builds. If that is not for the same release where we add 3.14, we could still do it in a subsequent release and benefit (even if it doesn't simplify the Python 3.14 support path).

For pure Python packages we have a simpler path forward (#43), but we also need to do some work there.

A final consideration is that we are also interested in adding support for free-threaded Python, and much of the work needed to enable that will also dovetail with the other projects that get us towards 3.14 support.

With all of this in mind, I propose that we front load as much work as possible with the aim of shipping 3.14 support using pure Python packages, limited API builds, and fully devendored nvcomp. Since all of these are projects that we want to support anyway, we might as well invest in them now. If we wind up needing to fall back to the easy path in a few months, we will still have made worthwhile progress towards our key goals. With that in mind, here is the sequence of other issues that we need to work through, in order of when they need to be done.

Approach

Pre-work: RAPIDS

For limited API/pure Python packages, we should do the following:

Migration

Post-migration

(copied from #120)

  • Update the ci-imgs repo's latest configuration to use Python 3.14
  • Update the docker repo's build and test matrices, and references in docs
  • Update the build matrix in release-scripts so that Python 3.13 wheels are uploaded
  • update release selector in docs
  • update all the repos to point at main on shared-workflows again
  • move devcontainers Python version pin to Python 3.14
  • delete the python-3.14 branch in shared-workflows
  • merge any remaining rapids-reviser changes

Notes

Enabling #167 would also make #171 less pressing.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions