Skip to content

Commit a94d207

Browse files
rgommersrgommers
committed
DOC: update the NumPy Roadmap
With NumPy 2.0 RCs available, that release is feature-complete and a lot on the roadmap is outdated. So time for a large update. [skip actions] [skip cirrus] [skip azp]
1 parent 2970735 commit a94d207

File tree

1 file changed

+98
-50
lines changed

1 file changed

+98
-50
lines changed

‎doc/neps/roadmap.rst

Lines changed: 98 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -18,25 +18,19 @@ may include (among other things) interoperability protocols, better duck typing
1818
support and ndarray subclass handling.
1919

2020
The key goal is: *make it easy for code written for NumPy to also work with
21-
other NumPy-like projects.* This will enable GPU support via, e.g, CuPy or JAX,
21+
other NumPy-like projects.* This will enable GPU support via, e.g, CuPy, JAX or PyTorch,
2222
distributed array support via Dask, and writing special-purpose arrays (either
2323
from scratch, or as a ``numpy.ndarray`` subclass) that work well with SciPy,
24-
scikit-learn and other such packages.
24+
scikit-learn and other such packages. A large step forward in this area was
25+
made in NumPy 2.0, with adoption of and compliance with the array API standard
26+
(v2022.12, see :ref:`NEP47`). Future work in this direction will include
27+
support for newer versions of the array API standard, and adding features as
28+
needed based on real-world experience and needs.
2529

26-
The ``__array_ufunc__`` and ``__array_function__`` protocols are stable, but
27-
do not cover the whole API. New protocols for overriding other functionality
28-
in NumPy are needed. Work in this area aims to bring to completion one or more
29-
of the following proposals:
30-
31-
- :ref:`NEP30`
32-
- :ref:`NEP31`
33-
- :ref:`NEP35`
34-
- :ref:`NEP37`
35-
36-
In addition we aim to provide ways to make it easier for other libraries to
37-
implement a NumPy-compatible API. This may include defining consistent subsets
38-
of the API, as discussed in `this section of NEP 37
39-
<https://numpy.org/neps/nep-0037-array-module.html#requesting-restricted-subsets-of-numpy-s-api>`__.
30+
In addition, the ``__array_ufunc__`` and ``__array_function__`` protocols
31+
fulfill a role here - they are stable and used by several downstream projects.
32+
They do not cover the whole API, so use of the array API standard is preferred
33+
for new code.
4034

4135

4236
Performance
@@ -46,17 +40,25 @@ Improvements to NumPy's performance are important to many users. We have
4640
focused this effort on Universal SIMD (see :ref:`NEP38`) intrinsics which
4741
provide nice improvements across various hardware platforms via an abstraction
4842
layer. The infrastructure is in place, and we welcome follow-on PRs to add
49-
SIMD support across all relevant NumPy functions.
43+
SIMD support across relevant NumPy functionality.
44+
45+
Transitioning from C to C++, both in the SIMD infrastructure and in NumPy
46+
internals more widely, is in progress. We have also started to make use of
47+
Google Highway (see :ref:`NEP54`), and that usage is likely to expand. Work
48+
towards support for newer SIMD instruction sets, like SVE on arm64, is ongoing.
5049

5150
Other performance improvement ideas include:
5251

5352
- A better story around parallel execution.
5453
- Optimizations in individual functions.
55-
- Reducing ufunc and ``__array_function__`` overhead.
5654

5755
Furthermore we would like to improve the benchmarking system, in terms of coverage,
58-
easy of use, and publication of the results (now
59-
`here <https://pv.github.io/numpy-bench>`__) as part of the docs or website.
56+
easy of use, and publication of the results. Benchmarking PRs/branches compared
57+
to the `main` branch is a primary purpose, and required for PRs that are
58+
performance-focused (e.g., adding SIMD acceleration to a function). In
59+
addition, we'd like a performance overview like the one we had `here
60+
<https://pv.github.io/numpy-bench>`__, set up in a way that is more
61+
maintainable long-term.
6062

6163

6264
Documentation and website
@@ -68,69 +70,115 @@ documentation on many topics are missing or outdated. See :ref:`NEP44` for
6870
planned improvements. Adding more tutorials is underway in the
6971
`numpy-tutorials repo <https://github.com/numpy/numpy-tutorials>`__.
7072

71-
Our website (https://numpy.org) was completely redesigned recently. We aim to
72-
further improve it by adding translations, more case studies and other
73-
high-level content, and more (see `this tracking issue <https://github.com/numpy/numpy.org/issues/266>`__).
73+
We also intend to make all the example code in our documentation interactive -
74+
work is underway to do so via ``jupyterlite-sphinx`` and Pyodide.
75+
76+
Our website (https://numpy.org) is in good shape. Further work on expanding the
77+
number of languages that the website is translated in is desirable. As are
78+
improvements to the interactive notebook widget, through JupyterLite.
7479

7580

7681
Extensibility
7782
-------------
7883

79-
We aim to make it much easier to extend NumPy. The primary topic here is to
80-
improve the dtype system - see :ref:`NEP41` and related NEPs linked from it.
81-
Concrete goals for the dtype system rewrite are:
82-
83-
- Easier custom dtypes:
84+
We aim to continue making it easier to extend NumPy. The primary topic here is to
85+
improve the dtype system - see for example :ref:`NEP41` and related NEPs linked
86+
from it. In NumPy 2.0, a new C API for user-defined dtypes was made public. We aim
87+
to encourage its usage and improve this API further.
8488

85-
- Simplify and/or wrap the current C-API
86-
- More consistent support for dtype metadata
87-
- Support for writing a dtype in Python
89+
Ideas for new dtypes that may be developed outside of the main NumPy repository
90+
first, and that could potentially be upstreamed into NumPy later, include:
8891

89-
- Allow adding (a) new string dtype(s). This could be encoded strings with
90-
fixed-width storage (e.g., ``utf8`` or ``latin1``), and/or a variable length
91-
string dtype. The latter could share an implementation with ``dtype=object``,
92-
but be explicitly type-checked.
93-
One of these should probably be the default for text data. The current
94-
string dtype support is neither efficient nor user friendly.
92+
- A quad-precision (128-bit) dtype
93+
- A ``bfloat16`` dtype
94+
- A fixed-width string dtype which supports encodings (e.g., ``utf8`` or
95+
``latin1``)
96+
- A unit dtype
9597

9698

9799
User experience
98100
---------------
99101

100102
Type annotations
101103
````````````````
102-
NumPy 1.20 adds type annotations for most NumPy functionality, so users can use
103-
tools like `mypy`_ to type check their code and IDEs can improve their support
104+
Type annotations for most NumPy functionality is complete (although some
105+
submodules like ``numpy.ma`` are missing return types), so users can use tools
106+
like `mypy`_ to type check their code and IDEs can improve their support
104107
for NumPy. Improving those type annotations, for example to support annotating
105-
array shapes and dtypes, is ongoing.
108+
array shapes (see `gh-16544 <https://github.com/numpy/numpy/issues/16544>`__),
109+
is ongoing.
106110

107111
Platform support
108112
````````````````
109113
We aim to increase our support for different hardware architectures. This
110114
includes adding CI coverage when CI services are available, providing wheels on
111-
PyPI for POWER8/9 (``ppc64le``), providing better build and install
112-
documentation, and resolving build issues on other platforms like AIX.
115+
PyPI for platforms that are in high enough demand (e.g., we added ``musllinux``
116+
ones for NumPy 2.0), and resolving build issues on platforms that we don't test
117+
in CI (e.g., AIX).
118+
119+
We intend to write a NEP covering the support levels we provide and what is
120+
required for a platform to move to a higher tier of support, similar to
121+
`PEP 11 <https://peps.python.org/pep-0011/>`__.
122+
123+
CPython 3.13 will be the first release to offer a free-threaded build (i.e.,
124+
a CPython build with the GIL disabled). Work is in progress to support this
125+
well in NumPy. After that is stable and complete, there may be opportunities to
126+
actually make use of the potential for performance improvements from
127+
free-threaded CPython, or make it easier to do so for NumPy's users.
128+
129+
Binary size reduction
130+
`````````````````````
131+
The number of downloads of NumPy from PyPI and other platforms continues to
132+
increase - as of May 2024 we're at >200 million downloads/month from PyPI
133+
alone). Reducing the size of an installed NumPy package has many benefits:
134+
faster installs, lower disk space usage, smaller load on PyPI, less
135+
environmental impact, easier to fit more packages on top of NumPy into an AWS
136+
Lambda layer, lower latency for Pyodide users, and so on. We aim for
137+
significant reductions, as well as making it easier for end users and packagers
138+
to produce smaller custom builds (e.g., we added support for stripping tests
139+
before 2.1.0). See `gh-25737 <https://github.com/numpy/numpy/issues/25737>`__
140+
for details.
141+
142+
143+
NumPy 2.0 stabilization & downstream usage
144+
------------------------------------------
145+
146+
We made a very large amount of changes (and improvements!) in NumPy 2.0. The
147+
release process has taken a very long time, and part of the ecosystem is still
148+
catching up. We may need to slow down for a while, and possible help the rest
149+
of the ecosystem with adapting to the ABI and API changes.
150+
151+
We will need to assess the costs and benefits to NumPy itself,
152+
downstream package authors, and end users. Based on that assessment
153+
154+
155+
Security
156+
--------
157+
158+
NumPy is quite secure - we get only a limited number of reports about potential
159+
vulnerabilities, and most of those are incorrect. We have made strides with a
160+
documented security policy, a private disclosure method, and maintaining an
161+
OpenSSF scorecard (with a high score). However, we have not changed much in how
162+
we approach supply chain security in quite a while. We aim to make improvements
163+
here, for example achieving fully reproducible builds for all the build
164+
artifacts we publish - and providing full provenance information for them.
113165

114166

115167
Maintenance
116168
-----------
117169

118-
- ``MaskedArray`` needs to be improved, ideas include:
170+
- ``numpy.ma`` is still in poor shape and under-maintained. It needs to be
171+
improved, ideas include:
119172

120173
- Rewrite masked arrays to not be a ndarray subclass -- maybe in a separate project?
121174
- MaskedArray as a duck-array type, and/or
122175
- dtypes that support missing values
123176

124-
- Fortran integration via ``numpy.f2py`` requires a number of improvements, see
125-
`this tracking issue <https://github.com/numpy/numpy/issues/14938>`__.
126-
- A backend system for ``numpy.fft`` (so that e.g. ``fft-mkl`` doesn't need to monkeypatch numpy).
127177
- Write a strategy on how to deal with overlap between NumPy and SciPy for ``linalg``.
128-
- Deprecate ``np.matrix`` (very slowly).
178+
- Deprecate ``np.matrix`` (very slowly) - this is feasible ones the switch-over
179+
from sparse matrices to sparse arrays in SciPy is complete.
129180
- Add new indexing modes for "vectorized indexing" and "outer indexing" (see :ref:`NEP21`).
130181
- Make the polynomial API easier to use.
131-
- Integrate an improved text file loader.
132-
- Ufunc and gufunc improvements, see `gh-8892 <https://github.com/numpy/numpy/issues/8892>`__
133-
and `gh-11492 <https://github.com/numpy/numpy/issues/11492>`__.
134182

135183

136184
.. _`mypy`: https://mypy.readthedocs.io

0 commit comments

Comments
 (0)