numpy C API notes

The following are based on information spread across many different web pages at the time of writing. The contents here are, therefore, not guaranteed to be up-to-date.

First, my current, general understanding of application binary interfaces (ABIs) and application programmatic interfaces (APIs):

• You can maintain ABI compatibility but break API compatibility (See the note here). For instance, you can have a function that always accepts an integer and returns a float. If, however, you change what that function does for different integer arguments, you have changed the API without changing the ABI. You can build, link, and execute correctly, but your outputs will change.
• You can also change the ABI but not the API. For instance, if a function returns a numpy array, then the ABI might be different from numpy 1.X to 2.Y if the numpy array data structure has changed or is compiled differently despite the fact that the function's behavior (and therefore API) is unaltered. Empirical testing with surmise indeed suggests that this happened between 1.X and 2.Y, which is known to be an ABI-breaking major transition.
• If, however, you change the API significantly by, for example, adding a new argument to a function or a data structure or altering the type of an argument or return variable, then the ABI must also change.

Some numpy packaging details

• A given numpy version specifies the oldest Python version that it is compatible with. For instance, 2.2.2 is compatible with Python >= 3.10 despite the fact that 3.9 is still officially supported. At some time, a numpy version could become incompatible with newer versions of Python, which means that support of that numpy version must be dropped in order to include support of the newer Python version. To deal with this complexity and live in harmony with Python and numpy versioning schemes, the numpy docs say

We recommend all packages depending on NumPy to follow the recommendations in NEP 29.

• the numpy package includes both a Python and a C interface. The latter is available for direct use from C code or indirectly through the use of Cython or similar tools. If using Cython, then numpy is a build-time dependency of the package and it is important to understand potential ABI and API compatibility issues.

NumPy’s ABI is forward but not backward compatible. This means: binaries compiled against a given target version of NumPy’s C API will still run correctly with newer NumPy versions, but not with older versions.

• I suspect that the above is only true so long as the API is not altered in a way that must also alter the ABI.
• Strangely, the numpy docs make a statement that implies that the numpy C interface is backward compatible starting at 1.25. What is the benefit of having a backward compatible API without a backward compatible ABI? In such cases, I could distribute a prebuilt binary at a version and know that older versions of numpy have a matching interface, but that it won't matter because the code will fail at runtime due to incompatible binaries.

Before NumPy 1.25, the NumPy C-API was not exposed in a backwards compatible way by default. This means that when compiling with a NumPy version earlier than 1.25 you have to compile with the oldest version you wish to support.

• Based on the same section, I believe that for numpy >= 1.25

By default, NumPy will expose an API that is backwards compatible with the oldest NumPy version that supports the currently oldest compatible Python version.

• Therefore, since Python 3.9 is the oldest currently supported Python and numpy 1.19 was the first numpy version to support Python 3.9, then all numpy versions >= 1.25 that support Python 3.9 will be backward compatible with all older numpy versions back to and including 1.19. Therefore, when we increase the oldest Python version that we support, we can offer numpy support as far back as the earliest numpy version that works with the oldest Python version, but do not have to do so.
• Numpy docs explain that while numpy uses the PEP440 major.minor.bugfix version scheme, the scheme is not semantic versioning.
• numpy has a scheme for managing when and how to break backward compatibility.
• A change in major version will most likely imply ABI breakage. They acknowledge that some ABI breakage might have occurred in the past for minor versions.
• Minor versions can include changes to the interface including adding new features and removing deprecated code. However, backwards-incompatible interface changes require the use of deprecation warnings in at least the two preceding releases. Therefore, if your code breaks due to a backwards-incompatible interface change in a minor version, then you should see deprecation warnings for ~1 year, should update your code during that year to use the new interface, and should subsequently increase your minimum supported numpy version to at least the earliest version at which the new interface was officially available. This would also necessitate publishing prebuilt binaries that were built with the newer interface and that are expected to be API/ABI compatible with the numpy versions that users are allowed to install based on the content of setup.py.
• Numpy 2 docs indicate both API and ABI breakages.
• At present, wheels built with 1.X numpy will not work with 2.Y numpy installations. However, wheels built with 2.Y numpy will work with both 1.X and 2.Y versions.
• In principle, it appears that a package that uses Cython+numpy can work with numpy >=1.25 so long as they compile wheels with >=2.0.0 and the numpy functionality they use are not deprecated at a 2.Y minor release. This would require, however, that the numpy ABI be somewhat backward-compatible contrary to what the numpy documentation says. If you want to adhere 100% to numpy's statements, then you should compile wheels with 2.0.0 since that is stated to be backward compatible with 1.X and should work with >= 2.0.0 since the ABI is guaranteed to be forward ABI compatible but not necessarily backward compatible. This would work so long as the numpy API used by surmise is not changed at a 2.Y minor revision. Note that such a change might also force a change of ABI, in which case the ABI is not forward compatible either.