PEP 780: Update to use sys.abi_info

peps/pep-0780.rst

@@ -8,7 +8,7 @@ Status: Draft
 Type: Standards Track
 Topic: Packaging
 Created: 21-Mar-2025
-Python-Version: 3.14
+Python-Version: 3.15
 Post-History:
   `05-Aug-2024 <https://discuss.python.org/t/60007>`__,
   `26-Mar-2025 <https://discuss.python.org/t/86013>`__,
@@ -22,15 +22,16 @@ Post-History:
 Abstract
 ========
 
-This PEP defines using ABI features as environment markers for project
-dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508`
-(later moved to :ref:`packaging:dependency-specifiers`) introduced environment
-markers to specify dependencies based on rules that describe when the
-dependency should be used. This PEP extends the environment markers to allow
-specifying dependencies based on specific ABI features of the Python
-interpreter. For this, it defines a set of `ABI Features`_ and specifies how
-they are made available for `environment markers <pep-780-sys_abi_features_>`_
-as a new marker variable, ``sys_abi_features``.
+This PEP defines the use of information about the interpreter ABI as
+environment markers for project dependencies through a new ``sys_abi_info``
+environment marker variable. :pep:`508` (later moved to
+:ref:`packaging:dependency-specifiers`) introduced environment markers to
+specify dependencies based on rules that describe when the dependency should
+be used. This PEP extends environment markers to allow specifying dependencies
+based on specific information about the ABI of the Python interpreter. For
+this, it defines a set of `ABI Features`_ and specifies how they are made
+available for `environment markers <pep-780-sys_abi_info_>`_ via a new marker
+variable, ``sys_abi_info``.
 
 Motivation
 ==========
@@ -57,23 +58,19 @@ marker for free-threading`_ Discourse thread:
   wheels. Picking up the wrong Cython version is causing a lot of obscure build
   failures and runtime crashes. It would be beneficial if the metadata could
   express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_).
-- CFFI has no support for free-threading yet, and `Armin Rigo, one of the
-  maintainers, has stated
-  <https://github.com/python-cffi/cffi/pull/143#issuecomment-2580781899>`__
-  that it may be a good idea to fork ``cffi``, implement support for
-  free-threading, and only come back to the CFFI project with a single large PR
-  that adds support after the functionality "is reasonably well-tested (either
-  as tests or, better in this case, tested by being in use in various other
-  projects)". There are a lot of projects that depend on ``cffi``. They are
-  likely fine to start depending on a fork for free-threading only, however
+- CFFI has only recently gained support for free-threading, and for a
+  significant period of time, it looked likely that support would first be
+  added in a fork. There are a lot of projects that depend on ``cffi``, which
+  are likely fine to start depending on a fork for free-threading only, however
   depending on a fork for >=3.13 or for all Python versions seems like a much
-  larger ask, and more disruptive for distribution packagers.
+  larger ask, and more disruptive for distribution packagers. While the
+  situation has been resolved with CFFI 2.0.0, other projects may face similar
+  issues.
 
-While these concrete examples may be addressed later this year by Cython and
-CFFI making compatible releases, the same issue is going to repeat further up
-the stack. The free-threading rollout is expected to take several years, and an
-environment marker for free-threading will make that rollout significantly
-easier.
+While concrete examples may be addressed with compatible releases, the same
+issue is going to repeat further up the stack. The free-threading rollout is
+expected to take several years, and an environment marker for free-threading
+will make that rollout significantly easier.
 
 Another important ABI feature that is not yet covered by environment markers is
 the bitness of the interpreter. In most cases, the ``sys_platform`` or
@@ -135,7 +132,7 @@ Relation to Other PEPs
 This PEP extends environment markers with set semantics for ABI features.
 :pep:`751#additions-to-marker-expression-syntax` includes a similar extension
 for lock file specific environment markers; although the two have been
-developed indepedently, they are compatible where they overlap in terms of the
+developed independently, they are compatible where they overlap in terms of the
 new set semantics.
 
 Specification
@@ -149,57 +146,63 @@ and "**OPTIONAL**"" in this document are to be interpreted as described in
 ABI Features
 ------------
 
-ABI features are intrinsic properties of the Python interpreter, expressed as
-simple, understandable strings. However, not all features are equally
-applicable to all Python interpreters or Python versions. For example, the
-distinction between free-threaded and GIL-enabled interpreters is only relevant
-for CPython 3.13 onwards, but the bitness of the interpreter is relevant for
-all interpreters.
+ABI features are intrinsic properties of the Python interpreter.
+However, not all features are equally applicable to all Python interpreters or
+Python versions. For example, the distinction between free-threaded and
+GIL-enabled interpreters is only relevant for CPython 3.13 onwards, but the
+bitness of the interpreter, i.e. the number of bits in a pointer, is relevant
+for all interpreters.
 
 All interpreters MUST handle the following ABI features as stated. ABI features
 that are restricted to particular interpreters MUST NOT be provided by other
-interpreters. The features are subdivided into groups and for each group there
-MUST be exactly one feature present, except when the group is marked as
-optional, in which case there MUST be at most one feature present.
+interpreters.
 
-``free-threading`` or ``gil-enabled`` (only CPython)
-    If the Python interpreter is free-threaded, the ``free-threading`` feature
-    MUST be present and the ``gil-enabled`` feature MUST NOT be present.
-    Otherwise, the ``gil-enabled`` feature MUST be present and the
-    ``free-threading`` feature MUST NOT be present.
+In Python 3.15 and later, the ABI features are determined by the new
+:external+py3.15:py:data:`sys.abi_info` attribute. In earlier Python versions,
+the ABI features are the same as for Python 3.15 and given as follows:
 
-``debug`` (only CPython, optional)
-    This ABI feature is reserved for the ``--with-pydebug`` build of CPython.
-    If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities,
-    the ``debug`` feature MUST be present. On POSIX systems, this corresponds
-    to the Python expression ``"d" in sys.abiflags``.
+``pointer_bits``
+    The bitness of the interpreter, that is, the number of bits in a pointer,
+    typically ``32`` or ``64`` [#bitness]_.
 
-``32-bit`` or ``64-bit`` (optional)
-    The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit
-    build [#bitness]_. If the bitness is unknown or neither 32-bit nor 64-bit,
-    this feature MUST NOT be present.
+``free_threaded`` (only CPython)
+    If the Python interpreter is free-threaded, ``free_threaded`` MUST be
+    ``True``, otherwise it MUST be ``False``.
 
-.. _pep-780-sys_abi_features:
+``debug`` (only CPython)
+    This ABI feature corresponds to the
+    :external+py3.15:py:attr:`sys.abi_info.debug` field.
 
-The ``sys_abi_features`` Environment Marker
--------------------------------------------
+``byteorder``
+    Indicates the byte order of the interpreter. The value MUST be either
+    ``"little"`` or ``"big"``.
+
+.. _pep-780-sys_abi_info:
+
+The ``sys_abi_info`` Environment Marker
+---------------------------------------
 
 To make ABI features available in dependency specifications, a new environment
-marker variable, ``sys_abi_features``, is added to the format of dependency
-specifiers.
+marker variable, ``sys_abi_info``, is added to the format of dependency
+specifiers. It is built to reflect the value of ``sys.abi_info``, which is a
+new attribute of the ``sys`` module introduced in Python 3.15 as a simple
+namespace. To represent its fields in the environment marker variable, they are
+expressed as a frozen set of strings of the form ``f"{name}::{value}"``, where
+``name`` is the name of the field and ``value`` is the string representation of
+its value.
 
 To do this, we need to extend the grammar laid out in :pep:`508` and maintained
 in the :ref:`packaging:dependency-specifiers` and document the possible values.
 
-The grammar is extended to include the ``sys_abi_features`` marker variable by
+The grammar is extended to include the ``sys_abi_info`` marker variable by
 augmenting the definition of ``env_var`` as follows::
 
     env_var       = ('python_version' | 'python_full_version' |
                      'os_name' | 'sys_platform' | 'platform_release' |
                      'platform_system' | 'platform_version' |
                      'platform_machine' | 'platform_python_implementation' |
                      'implementation_name' | 'implementation_version' |
-                     'sys_abi_features' |
+                     'sys_abi_info' |
                      'extra' # ONLY when defined by a containing layer
                      )
 
@@ -212,10 +215,10 @@ Like the grammar, also the overview table of environment markers in
     * - Marker
       - Python equivalent
       - Sample values
-    * - ``sys_abi_features``
-      - no direct equivalent available
-      - ``{'free-threading', '64-bit'}``,
-        ``{'gil-enabled', 'debug', '32-bit'}``
+    * - ``sys_abi_info``
+      - ``sys.abi_info``
+      - ``{'free_threaded::True', 'pointer_bits::64'}``,
+        ``{'free_threaded::False', 'debug::True', 'pointer_bits::32'}``
 
 With these additions, ABI features can be used in dependency specifications via
 the ``in`` operator to test for the presence of a feature, or the ``not in``
@@ -229,22 +232,22 @@ Require Cython Pre-release for Free-Threaded Python
 To require a pre-release of Cython only for a free-threaded Python interpreter,
 the following dependency specification can be used::
 
-    cython >3.1.0a1; "free-threading" in sys_abi_features
-    cython ==3.0.*; "free-threading" not in sys_abi_features
+    cython >3.1.0a1; "free_threaded::True" in sys_abi_info
+    cython ==3.0.*; "free_threaded::False" in sys_abi_info
 
 Require SciPy Unless on 32-bit ``win32``
 ----------------------------------------
 To require SciPy unless on a 32-bit interpreter on Windows, the following
 dependency specification can be used::
 
-    scipy; platform_system != "Windows" or "32-bit" not in sys_abi_features
+    scipy; platform_system != "Windows" or "pointer_bits::32" not in sys_abi_info
 
 Require NumPy for a Free-Threaded Interpreter With Debugging Capabilities
 --------------------------------------------------------------------------
 To require NumPy only for a free-threaded interpreter with debugging
 capabilities, the following dependency can be used::
 
-    numpy; "free-threading" in sys_abi_features and "debug" in sys_abi_features
+    numpy; "free-threaded::True" in sys_abi_info and "debug::True" in sys_abi_info
 
 Backwards Compatibility
 =======================