Nexus: End Support for Python < 3.10 and package versions past EOL

[brockdyer03]

Current Status

The significant breadth of Python versions and package versions supported by Nexus is nothing short of a miracle, however in the words of Matt Smith playing the 11th Doctor, "Times change, and so must I."

The current minimum supported version policy for Nexus is as follows (versions from the pyproject.toml file in #5742, dates from nexus/nexus/versions.py or PyPI):

python     >= 3.6.0   # 2016-12-23
numpy      >= 1.13.0  # 2017-06-07
scipy      >= 0.19.0  # 2017-03-09
h5py       >= 2.6.0   # 2016-04-08
matplotlib >= 2.0.0   # 2017-01-17
spglib     >= 1.12.0  # 2019-01-29
cif2cell   >= 2.1.0   # 2023-10-05
pydot      >= 1.2.4   # 2017-12-25
seekpath   >= 1.9.0   # 2019-07-25

These versions have all been verified with Python 3.6.0 using a Docker container based on Ubuntu 18.04.

The Problem

The sheer number of versions supported by Nexus creates a significant number of problems with respect to development. For instance, there are several places in Nexus that require checking the version of various packages prior to an operation being performed.

Additionally, versions.py is designed for checking dependencies and their versions, but is often not updated and generally is a roundabout way to just have a centralized version set that would be provided by a pyproject.toml file.

Version-Specific Code

(testing.py, lines 1-8)

try:
    import numpy as np
    if np.lib.NumpyVersion(np.__version__) >= '2.0.0b1':
        np.set_printoptions(legacy="1.25")
    numpy_available = True
except:
    numpy_available = False
#end try

(qdens, lines 825-832)

# Use np.trapz for backwards and forwards compatibility.
np_version = tuple(map(int, np.__version__.split('.')[:2]))
use_trapz = np_version < (2, 0)
if use_trapz:
    total_electrons = np.trapz(lmean, r)
else:
    # In NumPy >= 2.0, prefer np.trapezoid
    total_electrons = np.trapezoid(lmean, r)

(generic.py, lines 516-540)

try:
    tmp = pickle.load(fobj)
except ModuleNotFoundError:
    try:
        # Old pickles from before Nexus was packaged (PR #5700, December 20 2025)
        # won't have the correct module path. The custom unpickler will handle this by 
        # prepending "nexus." to the module path
        tmp = NexusUnpickler(fobj).load()
    except UnpicklingError:
        try:
            # NumPy pickles can use latin1 encoding
            # They will likely still fail from an underflow since they are not pickle-compliant
            tmp = NexusUnpickler(fobj).load(encoding='latin1')
        except UnpicklingError:
            # fallback for files created with protocol 5
            # in environments that only support up to protocol 4
            try:
                import pickle5
                tmp = pickle5.load(fobj)
            except ImportError:
                have_pickle5 = False
                error("Highest pickle protocol in current python version is {}, but {} is written using a higher protocol. Install pickle5, e.g. via pip, to enable protocol 5 in python <= 3.7.x".format(pickle.HIGHEST_PROTOCOL, fpath))
        #end try
    #end try
#end try

(structure.py, lines 5895-5915)

try:
    import seekpath
    from seekpath import get_explicit_k_path
    version = seekpath.__version__

    try:
        version = [int(i) for i in version.split('.')]
        if len(version) < 3:
            raise ValueError
        #end if
    except ValueError:
        raise ValueError("Unable to parse version number")
    #end try
    if tuple(version) < (1, 8, 3):
        raise ValueError("Invalid seekpath version, need >= 1.8.4")
    #end if
    del version
    del seekpath
except:
    get_explicit_k_path = unavailable('seekpath','get_explicit_k_path')
#end try

Proposed Version Support

I propose that we follow the support timeline of Python and the support timeline of NumPy. This means Python >=3.10 and NumPy >= 2.0.0.

This policy has several main motivators.

1. Non-EOL Python versions do not receive security updates, leaving them more vulnerable to outside attacks.
2. By supporting only NumPy >= 2.0.0 we will no longer need to worry about bugs that are specific to Numpy < 2.
3. Prospective contributors do not need to put excessive thought into using a recently-introduced feature.
4. We can limit the amount of version-specific documenting (e.g. Nexus: Installable as a Python package #5742 currently includes notes for pip < 10, which only comes with Python 3.6)
5. New updates now pose less of a threat to the stability of Nexus as it is far less likely that syntax has changed or functions have been deprecated.

Functions Introduced After Python 3.6

Python Functions and Utilities

• Dataclasses (Python 3.7)
• Postponed evaluation of annotations (Python 3.7)
• Dictionary union via the union operator | (Python 3.9)
• Built-in methods to remove prefixes and suffixes from strings (Python 3.9)
• Updated math.gcd to accept more than 2 arguments (Python 3.9)
• Added math.lcm (Python 3.9)
• Add pathlib.Path.readlink for resolving symlinks (Python 3.9)
• Add structural pattern matching (match statements, equivalent to switch in C/C++) (Python 3.10)
• Add support for parenthesized context managers (Python 3.10)
• Improved the __repr__ and __str__ of Enums (Python 3.10)
• Updated pprint to be able to print dataclasses (Python 3.10)

Numpy Functions and Utilities

• numpy.gcd and numpy.lcm (NumPy 1.15.0)
• Added the where keyword for np.all, np.any, np.mean, np.std, and np.var (Numpy 1.20.0)
• Add is_integer check for floats and ints (Numpy 1.22.0)
• Add support for in-place matrix multiplication (Numpy 1.25.0)
• Add numpy.isdtype (Numpy 2.0.0)
• Add support for in-place FFT operations (Numpy 2.0.0)

Non-EOL Operating System Python Defaults

Here is a list of some relevant operating systems that have not hit EOL and their default or oldest available Python versions:

• Ubuntu 22.04 LTS
  • Python 3.10
• Debian 12
  • Python 3.11
• RHEL 9 (ditto for CentOS 9)
  • Python 3.9
• RHEL 10 (ditto for CentOS 10)
  • Python 3.12
• Fedora 42
  • Python 3.13
• OpenSUSE 16
  • Python 3.13
• MacOS (Homebrew)
  • Python 3.10.19

Diagrammatic Representation

---
config:
    theme: 'dark'
    themeVariables:
        activeTaskBorderColor: '#03a200'
        activeTaskBkgColor: '#81B1DB'
        critBorderColor: '#E83737'
        todayLineColor: '#03a200'
        vertLineColor: '#ff0000'
---
gantt
dateFormat YYYY-MM-DD
axisFormat %m / %Y
title Support Window

Current Nexus Support Policy : vert, 2017-06-07, 3650d

section Python
3.6 : crit, active, 2016-12-23,2021-12-22
3.7 : crit, active, 2018-06-27,2023-06-26
3.8 : crit, active, 2019-10-14,2024-10-12
3.9 : crit, active, 2020-10-05,2025-10-04
3.10 : active, 2021-10-04,2026-10-03
3.11 : active, 2022-10-24,2027-10-23
3.12 : active, 2023-10-02,2028-09-30
3.13 : active, 2024-10-07,2029-10-06
3.14 : active, 2025-10-07,2030-10-06

section NumPy
1.13.0 : crit, active, 2017-06-07,2019-06-07
1.14.0 : crit, active, 2018-01-06,2020-01-06
1.15.0 : crit, active, 2018-07-23,2020-07-22
1.16.0 : crit, active, 2019-01-14,2021-01-13
1.17.0 : crit, active, 2019-07-26,2021-07-25
1.18.0 : crit, active, 2019-12-22,2021-12-21
1.19.0 : crit, active, 2020-06-20,2022-06-20
1.20.0 : crit, active, 2021-01-30,2023-01-30
1.21.0 : crit, active, 2021-06-22,2023-06-22
1.22.0 : crit, active, 2021-12-31,2023-12-31
1.23.0 : crit, active, 2022-06-22,2024-06-21
1.24.0 : crit, active, 2022-12-18,2024-12-17
1.25.0 : crit, active, 2023-06-17,2025-06-16
1.26.0 : crit, active, 2023-09-16,2025-09-15
2.0.0 : active, 2024-06-16,2026-06-16
2.1.0 : active, 2024-08-18,2026-08-18
2.2.0 : active, 2024-12-08,2026-12-08
2.3.0 : active, 2025-06-07,2027-06-07
2.4.0 : active, 2025-12-20,2027-12-20

section SciPy
0.19.0 : crit, active, 2017-03-09,2019-03-09
1.0.0 :  crit, active, 2017-10-25,2019-10-25
1.1.0 :  crit, active, 2018-05-05,2020-05-04
1.2.0 :  crit, active, 2018-12-18,2020-12-17
1.3.0 :  crit, active, 2019-05-17,2021-05-16
1.4.0 :  crit, active, 2019-12-16,2021-12-15
1.5.0 :  crit, active, 2020-06-21,2022-06-21
1.6.0 :  crit, active, 2020-12-31,2022-12-31
1.7.0 :  crit, active, 2021-06-20,2023-06-20
1.8.0 :  crit, active, 2022-02-05,2024-02-05
1.9.0 :  crit, active, 2022-07-29,2024-07-28
1.10.0 : crit, active, 2023-01-03,2025-01-02
1.11.0 : crit, active, 2023-06-25,2025-06-24
1.12.0 : crit, active, 2024-01-20,2026-01-19
1.13.0 : active, 2024-04-02,2026-04-02
1.14.0 : active, 2024-06-24,2026-06-24
1.15.0 : active, 2025-01-03,2027-01-03
1.16.0 : active, 2025-06-22,2027-06-22
1.17.0 : active, 2026-01-10,2028-01-10

section Matplotlib
2.0.0 :  crit, active, 2017-01-17,2019-01-17
2.1.0 :  crit, active, 2017-10-07,2019-10-07
2.2.0 :  crit, active, 2018-03-06,2020-03-05
3.0.0 :  crit, active, 2018-09-18,2020-09-17
3.1.0 :  crit, active, 2019-05-18,2021-05-17
3.2.0 :  crit, active, 2020-03-04,2022-03-04
3.3.0 :  crit, active, 2020-07-16,2022-07-16
3.4.0 :  crit, active, 2021-03-26,2023-03-26
3.5.0 :  crit, active, 2021-11-16,2023-11-16
3.6.0 :  crit, active, 2022-09-16,2024-09-15
3.7.0 :  crit, active, 2023-02-13,2025-02-12
3.8.0 :  crit, active, 2023-09-15,2025-09-14
3.9.0 : active, 2024-05-15,2026-05-15
3.10.0 : active, 2024-12-14,2026-12-14

Loading

[jtkrogel]

Brock and I have talked about this quite a bit over the last couple of days.

What we think makes the most sense is to have Nexus follow the Python support schedule.

This would mean that as of "today" we guarantee support only for Python 3.10 and newer, as well as modules that are compatible with 3.10 or newer.

[brockdyer03]

Assuming Nexus is currently at version 2.2.0 (lining up with QMCPACK's minor versions), I've assembled a table of Nexus versions assuming a 6 month release schedule. This table also includes the minimum versions of Python, Numpy, Scipy, Matplotlib, and h5py that would in theory be supported given the release date.

Release Date	Nexus	Python	NumPy	SciPy	Matplotlib	h5py
2/12/2026	2.2.0	>=3.6	1.13.0	0.19.0	2.0.0	2.6.0
8/01/2026	2.3.0	>=3.10	1.22.0	1.8.0	3.3.0	3.6.0
2/01/2027	2.4.0	>=3.10	1.22.0	1.8.0	3.3.0	3.6.0
8/01/2027	2.5.0	>=3.11	1.24.0	1.10.0	3.3.0	3.8.0
2/01/2028	2.6.0	>=3.11	1.24.0	1.10.0	3.3.0	3.8.0
8/01/2028	2.7.0	>=3.12	1.26.0	1.12.0	3.6.0	3.10.0
2/01/2029	2.8.0	>=3.12	1.26.0	1.12.0	3.6.0	3.10.0

I think it would probably be prudent to try and accelerate the Numpy support timeline so that we can more quickly move away from Numpy < 2.0 since that would let us remove any version checks. Ideally we'd deprecate Numpy < 2.0 with Nexus 2.6 or 2.7.

[ye-luo]

We should treat OS shipped python and scientific production python separately. The OS shipped python has more concern about security. The scientific production python are usually more focused on functionality and performance. In all the machines that I don't own and manage, the system python is never recommended for scientific production and newer python interpreter is readily available. So our minimal requirement should at least 3.10 which is the oldest non-EOL version or a newer version greatly benefiting nexus (better performance, less code for older python/libraries).

What are the python, also numpy if needed, versions required by NumPy, SciPy, Matplotlib, h5py? if they pursue more recent versions, we have to move on. If the common denominator is 3.10, we just stick to it.

It seems that "np_version < (2, 0)" only occurred once I don't feel the need to rush.

[brockdyer03]

The current versions of Python that the latest NumPy versions target are defined by the Scientific Python Ecosystem Coordination (SPEC) 0 proposal. That is, the next version of NumPy to be released (which will be 2.5.0 in just a few weeks I believe) will only target Python >= 3.12.

The support drop timeline in SPEC 0 suggests when package versions should be dropped.

I found that chart hard to read, but basically the idea is that you check whatever your package's release date will be, then draw a line vertically down from that and any other package's window that intersects your vertical line is a version that you should support according to SPEC 0.

[prckent]

Will try to get my 2c on this submitted tomorrow, Wednesday. To state the obvious, this appears to really be a discussion around what we want our support policy to be, what constraints need to be satisfied to reasonably support users & developers, and how to go about checking the policy in CI and during review.

[jtkrogel]

This is basically clear already but I rephrase here. The underlying motivation is to determine when (and which) "new" Python code features can be introduced into the Nexus codebase, which induces/necessitates a support schedule.

The benefits I see in the proposed support schedule is that it is non-arbitrary and follows industry standards.

It also never leaves a user without a version of Nexus that supports their Python version (since Nexus 2.2.0 works up to present, and future versions would cover all later Python versions).

[prckent]

Outsourcing the "what to support" question to SPEC is a good idea. Importantly it is easy to communicate, review, and test against. e.g. We can spin up oldest & newest version containers in CI, plus any intermediate versions if we have specific concerns. Note that unfortunately SPEC doesn't solve everything for us, e.g. h5py is an important dependency. Fortunately we have historically only run into problems with h5py when the underlying HDF5 versions have made major changes, so it is not that big a concern, just an uncovered risk.

I was all set to simply write "lets adopt the Scientific Python Ecosystem Coordination (SPEC) 0 proposal", except there is one quirk to discuss: my reading of the support Gantt chart is that they have already dropped numpy 1.x. I think this would e.g. break the ability to run Nexus out of the box on the current LTS release of Ubuntu (24.04) using only vendor provided packages. The upcoming 26.04 release fixes that, so if we were to consistently support the latest LTS release then the problem goes away in 2 months.

It is helpful and convenient to be able to run Nexus (and QMCPACK) using only vendor provided and curated dependencies, and it certainly makes things easier for the few people who can not install software directly from popular python package repositories. We have never had a formal policy to support the current release of any specific OS distribution, but it has been implicit. It has not been explicit around the related issue of QMCPACK C++ support, but it is something we have been consistently careful about, e.g. when bumping C++ language spec or minimum gcc & llvm versions. Perhaps we adopt the python version window now but wait ~3 months regarding numpy 1.x?

[jtkrogel]

What we had talked about above was to follow the Python support schedule set by the Python devs, i.e. to support Python 3.x until the Python folks themselves drop support (in this sense, referencing SPEC was misleading).

Under this, we support Python 3.10 until 9/2027 (its sunset date). Then we support the first version of a module dependency that supported Python 3.10 (e.g. numpy 1.22.0).

This gives us something like a 4 year support window as of today, and nominally a 5 year support window at the point a particular Python3.x sunsets.

[prckent]

TLDR: Lets move to Python minimum 3.10.0 and numpy 1.22.0

Details:

After more discussions offline + as listed above, I think we can straightforwardly set a couple of minimum versions (python, numpy), as a first step in a longer ranged plan.

First, setting python 3.10 and numpy 1.22.0 as the minimum versions would enable us to move forward with some new features as discussed above. It is also the least constraining of any of the options, so I don't see any downside. This needs to be documented and tested for in a few places + we can start testing against it in CI.

Second, we should start documenting (adding mini tutorials) in both the NEXUS and QMCPACK documentation on different routes for users to get access to newer versions of python and python packages. This is needed everywhere, but clearly the assumption that users know how to do this is not valid for everyone. However, despite both perlmutter and frontier having a system python v3.6, clearly people have found solutions on these machines without any documentation on our part.

Third, contracting out the decision making and support matrix to something like SPEC is clearly a good idea long term. Other codes in electronic structure and Monte Carlo have likely implicitly adopted it. If we can teach people how to access recent enough python effectively, then adopting this would be easy - the cost for users would be minimal or zero, since they will already be making and updating a python environment, or using a sufficiently up to date package manager (etc.).

Ultimately, if we can point users to recipes for how to get to new enough python + dependencies, then we can focus and use our limited resources more effectively.