Drop Python 3.7 as part of NEP 29

[matthewfeickert]

The Scikit-HEP projects follow a modified version of NEP 29. The NEP 29 drop schedule is the following

On next release, drop support for Python 3.5 (initially released on Sep 13, 2015)
On Jan 07, 2020 drop support for NumPy 1.14 (initially released on Jan 06, 2018)
On Jun 23, 2020 drop support for Python 3.6 (initially released on Dec 23, 2016)
On Jul 23, 2020 drop support for NumPy 1.15 (initially released on Jul 23, 2018)
On Jan 13, 2021 drop support for NumPy 1.16 (initially released on Jan 13, 2019)
On Jul 26, 2021 drop support for NumPy 1.17 (initially released on Jul 26, 2019)
On Dec 22, 2021 drop support for NumPy 1.18 (initially released on Dec 22, 2019)
On Dec 26, 2021 drop support for Python 3.7 (initially released on Jun 27, 2018)  # Python 3.7 dropped end of 2021
On Jun 21, 2022 drop support for NumPy 1.19 (initially released on Jun 20, 2020)
On Jan 31, 2023 drop support for NumPy 1.20 (initially released on Jan 31, 2021)  # Next NEP drop is here
On Apr 14, 2023 drop support for Python 3.8 (initially released on Oct 14, 2019)
On Jun 23, 2023 drop support for NumPy 1.21 (initially released on Jun 22, 2021)
On Jan 01, 2024 drop support for NumPy 1.22 (initially released on Dec 31, 2021)
On Apr 05, 2024 drop support for Python 3.9 (initially released on Oct 05, 2020)
On Apr 04, 2025 drop support for Python 3.10 (initially released on Oct 04, 2021)

and so NumPy, SciPy, and Matplotlib (all projects that pyhf has some dependency/extra on) have already dropped Python 3.7 support. It would be nice for pyhf to drop support for Python 3.7 in pyhf v0.8.0 as well, but before we do that we should ensure that this would not be deeply problematic for analyses that are currently using pyhf.

I assume that this isn't a problem given that the next LCG view release (LCG 103) will have pyhf v0.7.0 included with Python 3.9, given the Python version in the dev4 nightly release:

[19:59] login01.af.uchicago.edu:~ $ . /cvmfs/sft.cern.ch/lcg/views/dev4/latest/x86_64-centos7-gcc11-opt/setup.sh
[19:59] login01.af.uchicago.edu:~ $ python3 --version --version
Python 3.9.12 (main, Jun  7 2022, 16:09:12) 
[GCC 11.2.0]
[19:59] login01.af.uchicago.edu:~ $ python3 -m pip show pyhf
Name: pyhf
Version: 0.7.0
Summary: pure-Python HistFactory implementation with tensors and autodiff
Home-page: https://github.com/scikit-hep/pyhf
Author: Lukas Heinrich, Matthew Feickert, Giordon Stark
Author-email: [email protected], [email protected], [email protected]
License: Apache
Location: /cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Wed/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
Requires: click, jsonpatch, jsonschema, pyyaml, scipy, tqdm
Required-by: 
[20:00] login01.af.uchicago.edu:~ $

Do we know of any users that would be negatively impacted if the next release of pyhf (probably late 2022/early 2023) dropped Python 3.7 support?

[matthewfeickert]

I've sent this to the pyhf-announcements mailing list, but tagging known users who will be able to give feedback on their CPython runtime restrictions: @alexander-held, @masonproffitt, @phinate, @WolfgangWaltenberger, @sabinekraml, @balunas, @rjhyneman, @nhartman94, @madbaron, @ssaha1234.

(Note to future self: I'm not tagging the MadAnalysis5 maintainers here as MadAnalysis5 is already Python 3.8+. 👍 )

[alexander-held]

I've been considering the same for cabinetry and am currently not aware of any blockers.

[phinate]

I'm happy to do this for relaxed! I think this also lets me drop from __future__ import annotations :)

[masonproffitt]

I'm surprised that NumPy et al. drop support so quickly, but it's no problem for me

[matthewfeickert]

Thanks for all the feedback so far. If there isn't any further feedback in the next day I'll merge in PR #2044.

[matthewfeickert]

PR #2044 has been merged, so the next pyhf minor release v0.8.0 will drop support for Python 3.7. Any pyhf v0.7.x patch releases will still support Python 3.7.

This also allows for Issue #1772 to move forward.

[matthewfeickert]

@henryiii had some good points on this topic today in a conversation on the IRIS-HEP Slack

[Henry]
One key feature of the scientific stack is that they are pretty stable - does it really matter if numpy is a couple of versions old if you are on 3.7? Very likely not. You couldn’t upgrade to 3.8, so why would you complain about not upgrading to 1.2x, whatever it was that dropped 3.7. It’s harder for young packages; there might be a pretty big difference between one version and the last. I’d like for people to be able to use the new vector constructors, etc, even if they are stuck on 3.7 (and last I checked, a few of us were using 3.7). I’m also not very fond of NEP 29; I understand where it came from (10 year cycle on Python 2.7), but we have a uniform EOL plan upstream now and uniform Python releases so it would be nice if we could all just agree and use that. Now it makes me think “Is this a scientific package” vs. “is it a general package”, which splits the ecosystem. 🤷

CentOS 7 and CentOS 8 have Python 3.6, which is why you are seeing it at CERN, but the LCG stack ships with at least 3.8, so you shouldn’t be using 2.6 (SL 6), 2.7, or 3.6. Pretty sure 3.8 is the oldest being actively used by the experiments. Currently LCG is on 3.9. Looks like they’ve dropped the 2.7 stack altogether.

[Matthew]
Should we change the Scikit-HEP recommendations (https://scikit-hep.org/supported-python-versions) to just support the oldest Python that isn't EOL and then newer, instead of following NEP 29 drop dates?

[Henry]
IMO a package author can always keep support longer if they want to; I’d only guarantee as a whole scikit-hep at least support NEP 29 versions. I followed NEP 29 when Hist came out, and IIRC uproot-browser might follow NEP 29. I’ve just kept more foundational libraries closer to Python EoL (boost-histogram for example).

So given the Python EOL schedule

$ pipx install norwegianblue
$ eol python
| cycle |  release   | latest | latest release |    eol     |
|:------|:----------:|:-------|:--------------:|:----------:|
| 3.10  | 2021-10-04 | 3.10.8 |   2022-10-11   | 2026-10-04 |
| 3.9   | 2020-10-05 | 3.9.15 |   2022-10-11   | 2025-10-05 |
| 3.8   | 2019-10-14 | 3.8.15 |   2022-10-11   | 2024-10-14 |
| 3.7   | 2018-06-26 | 3.7.15 |   2022-10-10   | 2023-06-27 |
| 3.6   | 2016-12-22 | 3.6.15 |   2021-09-03   | 2021-12-23 |
| 3.5   | 2015-09-12 | 3.5.10 |   2020-09-05   | 2020-09-13 |
| 3.4   | 2014-03-15 | 3.4.10 |   2019-03-18   | 2019-03-18 |
| 3.3   | 2012-09-29 | 3.3.7  |   2017-09-19   | 2017-09-29 |
| 2.7   | 2010-07-03 | 2.7.18 |   2020-04-19   | 2020-01-01 |
| 2.6   | 2008-10-01 | 2.6.9  |   2013-10-29   | 2013-10-29 |

and where the experiments are I think it was still reasonable to drop Python 3.7 support, but that we don't need to be in any rush to drop Python 3.8 in 2023 or 2024.

Thanks @henryiii for your thoughts!