API: Default to hidden visibility for API tables

This adds a new ``NPY_API_SYMBOL_ATTRIBUTE`` but defaults to using
hidden visibility (always the case on windows!).

That actually makes the situation "worse" for ``eigenpy`` in some degree,
since it forces them to adapt, but it also allows them to decide to just
roll with it (by actually also exporting it on windows which).

Since it aligns windows and linux it seems like a good idea?  OTOH,
who knows how many projects get away with just not caring about windows...

I tried to reorganize the docs a bit on how to import the array API...

Author: seberg

doc/source/numpy_2_0_migration_guide.rst

@@ -221,6 +221,21 @@ using the NumPy types. You can still write cython code using the ``c.real`` and
 ``c.imag`` attributes (using the native typedefs), but you can no longer use
 in-place operators ``c.imag += 1`` in Cython's c++ mode.
 
+.. _api-table-visibility-change:
+
+Changes in table visibility (linking error)
+-------------------------------------------
+If you are experiencing linking errors related to ``PyArray_API`` or
+``PyArray_RUNTIME_VERSION``, NumPy changed the default visibility to hidden
+(which was always the case on windows).
+You can use the :c:macro:`NPY_API_SYMBOL_ATTRIBUTE` to opt-out of this change.
+However, we generally discourage linking across project boundaries because
+it breaks NumPy compatibility checks.
+
+If you are experiencing problems due to an upstream header including NumPy,
+the solution is to make sure you ``#include "numpy/ndarrayobject.h"`` before
+their header and import NumPy yourself based on  :ref:`_including-the-c-api`.
+
 
 Changes to namespaces
 =====================

doc/source/reference/c-api/array.rst

@@ -3814,13 +3814,118 @@ Other conversions
     in the *vals* array. The sequence can be smaller then *maxvals* as
     the number of converted objects is returned.
 
+.. _including-the-c-api:
 
-Miscellaneous
--------------
+Including and importing the C API
+---------------------------------
 
+To use the NumPy C-API you typically need to include the
+``numpy/ndarrayobject.h`` header and ``numpy/ufuncobject.h`` for some ufunc
+related functionality (``arrayobject.h`` is an alias for ``ndarrayobject.h``).
 
-Importing the API
-~~~~~~~~~~~~~~~~~
+These two headers export most relevant functionality.  In general any project
+which uses the NumPy API must import NumPy using one of the functions
+``PyArray_ImportNumPyAPI()`` or ``import_array()``.
+In some places, functionality which requires ``import_array()`` is not
+needed, because you only need type definitions.  In this case, it is
+sufficient to include ``numpy/ndarratypes.h``.
+
+For the typical Python project, multiple C or C++ files will be compiled into
+a single shared object (the Python C-module) and ``PyArray_ImportNumPyAPI()``
+should be called inside it's module initialization.
+
+When you have a single C-file, this will consist of::
+
+.. code-block:: c
+
+    #include "numpy/ndarrayobject.h"
+
+    PyMODINIT_FUNC PyInit_my_module(void)
+    {
+        if (PyArray_ImportNumPyAPI() < 0) {
+            return NULL;
+        }
+        /* Other initialization code. */
+    }
+
+However, most projects will have additional C files which are all
+linked together into a single Python module.
+In this case, the helper C files typically do not have a canonical place
+where ``PyArray_ImportNumPyAPI`` should be called (although it is OK and
+fast to call it often).
+
+To solve this, NumPy provides the following pattern that the the main
+file is modified to define ``PY_ARRAY_UNIQUE_SYMBOL`` before the include:
+
+.. code-block:: c
+    /* Main module file */
+    #define PY_ARRAY_UNIQUE_SYMBOL MyModule
+    #include "numpy/ndarrayobject.h"
+
+    PyMODINIT_FUNC PyInit_my_module(void)
+    {
+        if (PyArray_ImportNumPyAPI() < 0) {
+            return NULL;
+        }
+        /* Other initialization code. */
+    }
+
+while the other files use:
+
+.. code-block:: C
+
+    /* Second file without any import */
+    #define NO_IMPORT_ARRAY
+    #define PY_ARRAY_UNIQUE_SYMBOL MyModule
+    #include "numpy/ndarrayobject.h"
+
+You can of course add the defines to a local header used throughout.
+You just have to make sure that the main file does _not_ define
+``NO_IMPORT_ARRAY``.
+
+For ``numpy/ufuncobject.h`` the same logic applies, but the unique symbol
+mechanism is ``#define PY_UFUNC_UNIQUE_SYMBOL`` (both can match).
+
+Additionally, you will probably wish to add a
+``#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION``
+to avoid warnings about possible use of old API.
+
+.. note::
+    If you are experiencing access violations make sure that the NumPy API
+    was properly imported and the symbol ``PyArray_API`` is not ``NULL``.
+    When in a debugger, this symbols actual name will be
+    ``PY_ARRAY_UNIQUE_SYMBOL``+``PyArray_API``, so for example
+    ``MyModulePyArray_API`` in the above.
+    (E.g. even a ``printf("%p\n", PyArray_API);`` just before the crash.)
+
+
+Mechanism details and dynamic linking
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The main part of the mechanism is that without NumPy needs to define
+a ``void **PyArray_API`` table for you to look up all functions.
+Depending on your macro setup, this takes different routes depending on
+whether :c:macro:`NO_IMPORT_ARRAY` and  :c:macro:`PY_ARRAY_UNIQUE_SYMBOL`
+are defined:
+
+* If neither is defined, the C-API is declared to
+  ``static void **PyArray_API``, so it is only visible within the
+  compilation unit/file using ``#includes numpy/arrayobject.h``.
+* If only ``PY_ARRAY_UNIQUE_SYMBOL`` is defined (it could be empty) then
+  the it is declared to a non-static ``void **`` allowing it to be used
+  by other files which are linked.
+* If ``NO_IMPORT_ARRAY`` is defined, the table is declared as
+  ``extern void **``, meaning that it must be linked to a file which does not
+  use ``NO_IMPORT_ARRAY`.
+
+The ``PY_ARRAY_UNIQUE_SYMBOL`` mechanism additionally mangles the names to
+avoid conflicts.
+
+
+.. versionchanged::
+    NumPy 2.0 exports the headers to avoid sharing the table outside of a
+    single shared object/dll (this was always the case on Windows).
+    Please see :c:macro:`NPY_API_SYMBOL_ATTRIBUTE` for details.
 
 In order to make use of the C-API from another extension module, the
 :c:func:`import_array` function must be called. If the extension module is
@@ -3844,61 +3949,45 @@ the C-API is needed then some additional steps must be taken.
     module that will make use of the C-API. It imports the module
     where the function-pointer table is stored and points the correct
     variable to it.
+    This macro includes a ``return NULL;`` on error, so that
+    ``PyArray_ImportNumPyAPI()`` is preferable for custom error checking.
+    You may also see use of ``_import_array()`` (a function, not
+    a macro, but you may want to raise a better error if it fails) and
+    the variations ``import_array1(ret)`` which customizes the return value. 
 
 .. c:macro:: PY_ARRAY_UNIQUE_SYMBOL
 
-.. c:macro:: NO_IMPORT_ARRAY
+.. c:macro:: NPY_API_SYMBOL_ATTRIBUTE
 
-    Using these #defines you can use the C-API in multiple files for a
-    single extension module. In each file you must define
-    :c:macro:`PY_ARRAY_UNIQUE_SYMBOL` to some name that will hold the
-    C-API (*e.g.* myextension_ARRAY_API). This must be done **before**
-    including the numpy/arrayobject.h file. In the module
-    initialization routine you call :c:func:`import_array`. In addition,
-    in the files that do not have the module initialization
-    sub_routine define :c:macro:`NO_IMPORT_ARRAY` prior to including
-    numpy/arrayobject.h.
-
-    Suppose I have two files coolmodule.c and coolhelper.c which need
-    to be compiled and linked into a single extension module. Suppose
-    coolmodule.c contains the required initcool module initialization
-    function (with the import_array() function called). Then,
-    coolmodule.c would have at the top:
+    .. versionadded:: 2.0
 
-    .. code-block:: c
+    An additional symbol which can be used to share e.g. visibility beyond
+    shared object boundaries.
+    By default, NumPy adds the C visibility hidden attribute (if available):
+    ``void __attribute__((visibility("hidden"))) **PyArray_API;``.
+    You can change this by defining ``NPY_API_SYMBOL_ATTRIBUTE``, which will
+    make this:
+    ``void NPY_API_SYMBOL_ATTRIBUTE **PyArray_API;`` (with additional
+    name mangling via the unique symbol).
 
-        #define PY_ARRAY_UNIQUE_SYMBOL cool_ARRAY_API
-        #include numpy/arrayobject.h
+    Adding an empty ``#define NPY_API_SYMBOL_ATTRIBUTE`` will have the same
+    behavior as NumPy 1.x.
 
-    On the other hand, coolhelper.c would contain at the top:
+    .. note::
+        Windows never had shared visbility although you can use this macro
+        to achieve it.  We generally discourage sharing beyond shared boundary
+        lines since importing the array API includes NumPy version checks.
+
+.. c:macro:: NO_IMPORT_ARRAY
+
+    Defining ``NO_IMPORT_ARRAY`` before the ``ndarrayobject.h`` include
+    indicates that the NumPy C API import is handled in a different file
+    and the include mechanism will not be added here.
+    You must have one file without ``NO_IMPORT_ARRAY`` defined.
 
-    .. code-block:: c
 
-        #define NO_IMPORT_ARRAY
-        #define PY_ARRAY_UNIQUE_SYMBOL cool_ARRAY_API
-        #include numpy/arrayobject.h
-
-    You can also put the common two last lines into an extension-local
-    header file as long as you make sure that NO_IMPORT_ARRAY is
-    #defined before #including that file.
-
-    Internally, these #defines work as follows:
-
-    * If neither is defined, the C-API is declared to be
-      ``static void**``, so it is only visible within the
-      compilation unit that #includes numpy/arrayobject.h.
-    * If :c:macro:`PY_ARRAY_UNIQUE_SYMBOL` is #defined, but
-      :c:macro:`NO_IMPORT_ARRAY` is not, the C-API is declared to
-      be ``void**``, so that it will also be visible to other
-      compilation units.
-    * If :c:macro:`NO_IMPORT_ARRAY` is #defined, regardless of
-      whether :c:macro:`PY_ARRAY_UNIQUE_SYMBOL` is, the C-API is
-      declared to be ``extern void**``, so it is expected to
-      be defined in another compilation unit.
-    * Whenever :c:macro:`PY_ARRAY_UNIQUE_SYMBOL` is #defined, it
-      also changes the name of the variable holding the C-API, which
-      defaults to ``PyArray_API``, to whatever the macro is
-      #defined to.
+Miscellaneous
+-------------
 
 Checking the API Version
 ~~~~~~~~~~~~~~~~~~~~~~~~

doc/source/release/2.0.0-notes.rst

@@ -655,6 +655,17 @@ Please see :ref:`migration_c_descr` for more information.
 
 (`gh-25943 <https://github.com/numpy/numpy/pull/25943>`__)
 
+API symbols now hidden but customizable
+---------------------------------------
+NumPy now defaults to hide the API symbols it adds to allow all NumPy API
+usage.
+This means that by default you cannot dynamically fetch the NumPy API from
+another library (this was never possible on windows).
+Please see :ref:`api-table-visibility-change` and the new
+:c:macro:`NPY_API_SYMBOL_ATTRIBUTE` which allows to customize/revert this.
+
+(`gh-26103 <https://github.com/numpy/numpy/pull/26103>`__)
+
 
 NumPy 2.0 C API removals
 ========================

numpy/_core/code_generators/generate_numpy_api.py

@@ -33,13 +33,18 @@
         _NPY_VERSION_CONCAT_HELPER(PY_ARRAY_UNIQUE_SYMBOL)
 #endif
 
+/* By default do not export API in an .so (was never the case on windows) */
+#ifndef NPY_API_SYMBOL_ATTRIBUTE
+    #define NPY_API_SYMBOL_ATTRIBUTE NPY_VISIBILITY_HIDDEN
+#endif
+
 #if defined(NO_IMPORT) || defined(NO_IMPORT_ARRAY)
-extern void **PyArray_API;
-extern int PyArray_RUNTIME_VERSION;
+extern NPY_API_SYMBOL_ATTRIBUTE void **PyArray_API;
+extern NPY_API_SYMBOL_ATTRIBUTE int PyArray_RUNTIME_VERSION;
 #else
 #if defined(PY_ARRAY_UNIQUE_SYMBOL)
-void **PyArray_API;
-int PyArray_RUNTIME_VERSION;
+NPY_API_SYMBOL_ATTRIBUTE void **PyArray_API;
+NPY_API_SYMBOL_ATTRIBUTE int PyArray_RUNTIME_VERSION;
 #else
 static void **PyArray_API = NULL;
 static int PyArray_RUNTIME_VERSION = 0;

numpy/_core/code_generators/generate_ufunc_api.py

@@ -18,11 +18,16 @@
 #define PyUFunc_API PY_UFUNC_UNIQUE_SYMBOL
 #endif
 
+/* By default do not export API in an .so (was never the case on windows) */
+#ifndef NPY_API_SYMBOL_ATTRIBUTE
+    #define NPY_API_SYMBOL_ATTRIBUTE NPY_VISIBILITY_HIDDEN
+#endif
+
 #if defined(NO_IMPORT) || defined(NO_IMPORT_UFUNC)
-extern void **PyUFunc_API;
+extern NPY_API_SYMBOL_ATTRIBUTE void **PyUFunc_API;
 #else
 #if defined(PY_UFUNC_UNIQUE_SYMBOL)
-void **PyUFunc_API;
+NPY_API_SYMBOL_ATTRIBUTE void **PyUFunc_API;
 #else
 static void **PyUFunc_API=NULL;
 #endif