Merge branch 'origin/main' into nogil-gc-mark-alive

Author: nascheme

.github/CODEOWNERS

@@ -96,13 +96,14 @@ Doc/library/site.rst          @FFY00
 Lib/test/test_except*.py      @iritkatriel
 Objects/exceptions.c          @iritkatriel
 
-# Hashing
-**/*hashlib*                  @gpshead @tiran
+# Hashing & cryptographic primitives
+**/*hashlib*                  @gpshead @tiran @picnixz
 **/*pyhash*                   @gpshead @tiran
-**/sha*                       @gpshead @tiran
-Modules/md5*                  @gpshead @tiran
-**/*blake*                    @gpshead @tiran
+**/sha*                       @gpshead @tiran @picnixz
+Modules/md5*                  @gpshead @tiran @picnixz
+**/*blake*                    @gpshead @tiran @picnixz
 Modules/_hacl/**              @gpshead
+**/*hmac*                     @gpshead @picnixz
 
 # logging
 **/*logging*                  @vsajip

.github/workflows/reusable-change-detection.yml

@@ -83,7 +83,22 @@ jobs:
           # into the PR branch anyway.
           #
           # https://github.com/python/core-workflow/issues/373
-          git diff --name-only "origin/$GITHUB_BASE_REF.." | grep -qvE '(\.rst$|^Doc|^Misc|^\.pre-commit-config\.yaml$|\.ruff\.toml$|\.md$|mypy\.ini$)' && echo "run-tests=true" >> "$GITHUB_OUTPUT" || true
+          grep_ignore_args=(
+            # file extensions
+              -e '\.md$'
+              -e '\.rst$'
+            # top-level folders
+              -e '^Doc/'
+              -e '^Misc/'
+            # configuration files
+              -e '^\.github/CODEOWNERS$'
+              -e '^\.pre-commit-config\.yaml$'
+              -e '\.ruff\.toml$'
+              -e 'mypy\.ini$'
+          )
+          git diff --name-only "origin/$GITHUB_BASE_REF.."        \
+            | grep -qvE "${grep_ignore_args[@]}"                  \
+            && echo "run-tests=true" >> "$GITHUB_OUTPUT" || true
         fi
 
         # Check if we should run hypothesis tests

Doc/c-api/apiabiversion.rst

@@ -6,9 +6,13 @@
 API and ABI Versioning
 ***********************
 
+
+Build-time version constants
+----------------------------
+
 CPython exposes its version number in the following macros.
-Note that these correspond to the version code is **built** with,
-not necessarily the version used at **run time**.
+Note that these correspond to the version code is **built** with.
+See :c:var:`Py_Version` for the version used at **run time**.
 
 See :ref:`stable` for a discussion of API and ABI stability across versions.
 
@@ -37,37 +41,83 @@ See :ref:`stable` for a discussion of API and ABI stability across versions.
 .. c:macro:: PY_VERSION_HEX
 
    The Python version number encoded in a single integer.
+   See :c:func:`Py_PACK_FULL_VERSION` for the encoding details.
 
-   The underlying version information can be found by treating it as a 32 bit
-   number in the following manner:
-
-   +-------+-------------------------+-------------------------+--------------------------+
-   | Bytes | Bits (big endian order) | Meaning                 | Value for ``3.4.1a2``    |
-   +=======+=========================+=========================+==========================+
-   |   1   |         1-8             |  ``PY_MAJOR_VERSION``   | ``0x03``                 |
-   +-------+-------------------------+-------------------------+--------------------------+
-   |   2   |         9-16            |  ``PY_MINOR_VERSION``   | ``0x04``                 |
-   +-------+-------------------------+-------------------------+--------------------------+
-   |   3   |         17-24           |  ``PY_MICRO_VERSION``   | ``0x01``                 |
-   +-------+-------------------------+-------------------------+--------------------------+
-   |   4   |         25-28           |  ``PY_RELEASE_LEVEL``   | ``0xA``                  |
-   +       +-------------------------+-------------------------+--------------------------+
-   |       |         29-32           |  ``PY_RELEASE_SERIAL``  | ``0x2``                  |
-   +-------+-------------------------+-------------------------+--------------------------+
+   Use this for numeric comparisons, for example,
+   ``#if PY_VERSION_HEX >= ...``.
 
-   Thus ``3.4.1a2`` is hexversion ``0x030401a2`` and ``3.10.0`` is
-   hexversion ``0x030a00f0``.
 
-   Use this for numeric comparisons, e.g. ``#if PY_VERSION_HEX >= ...``.
-
-   This version is also available via the symbol :c:var:`Py_Version`.
+Run-time version
+----------------
 
 .. c:var:: const unsigned long Py_Version
 
-   The Python runtime version number encoded in a single constant integer, with
-   the same format as the :c:macro:`PY_VERSION_HEX` macro.
+   The Python runtime version number encoded in a single constant integer.
+   See :c:func:`Py_PACK_FULL_VERSION` for the encoding details.
    This contains the Python version used at run time.
 
+   Use this for numeric comparisons, for example, ``if (Py_Version >= ...)``.
+
    .. versionadded:: 3.11
 
-All the given macros are defined in :source:`Include/patchlevel.h`.
+
+Bit-packing macros
+------------------
+
+.. c:function:: uint32_t Py_PACK_FULL_VERSION(int major, int minor, int micro, int release_level, int release_serial)
+
+   Return the given version, encoded as a single 32-bit integer with
+   the following structure:
+
+   +------------------+-------+----------------+-----------+--------------------------+
+   |                  | No.   |                |           | Example values           |
+   |                  | of    |                |           +-------------+------------+
+   | Argument         | bits  | Bit mask       | Bit shift | ``3.4.1a2`` | ``3.10.0`` |
+   +==================+=======+================+===========+=============+============+
+   | *major*          |   8   | ``0xFF000000`` | 24        | ``0x03``    | ``0x03``   |
+   +------------------+-------+----------------+-----------+-------------+------------+
+   | *minor*          |   8   | ``0x00FF0000`` | 16        | ``0x04``    | ``0x0A``   |
+   +------------------+-------+----------------+-----------+-------------+------------+
+   | *micro*          |   8   | ``0x0000FF00`` | 8         | ``0x01``    | ``0x00``   |
+   +------------------+-------+----------------+-----------+-------------+------------+
+   | *release_level*  |   4   | ``0x000000F0`` | 4         | ``0xA``     | ``0xF``    |
+   +------------------+-------+----------------+-----------+-------------+------------+
+   | *release_serial* |   4   | ``0x0000000F`` | 0         | ``0x2``     | ``0x0``    |
+   +------------------+-------+----------------+-----------+-------------+------------+
+
+   For example:
+
+   +-------------+------------------------------------+-----------------+
+   | Version     | ``Py_PACK_FULL_VERSION`` arguments | Encoded version |
+   +=============+====================================+=================+
+   | ``3.4.1a2`` | ``(3, 4, 1, 0xA, 2)``              | ``0x030401a2``  |
+   +-------------+------------------------------------+-----------------+
+   | ``3.10.0``  | ``(3, 10, 0, 0xF, 0)``             | ``0x030a00f0``  |
+   +-------------+------------------------------------+-----------------+
+
+   Out-of range bits in the arguments are ignored.
+   That is, the macro can be defined as:
+
+   .. code-block:: c
+
+      #ifndef Py_PACK_FULL_VERSION
+      #define Py_PACK_FULL_VERSION(X, Y, Z, LEVEL, SERIAL) ( \
+         (((X) & 0xff) << 24) |                              \
+         (((Y) & 0xff) << 16) |                              \
+         (((Z) & 0xff) << 8) |                               \
+         (((LEVEL) & 0xf) << 4) |                            \
+         (((SERIAL) & 0xf) << 0))
+      #endif
+
+   ``Py_PACK_FULL_VERSION`` is primarily a macro, intended for use in
+   ``#if`` directives, but it is also available as an exported function.
+
+   .. versionadded:: 3.14
+
+.. c:function:: uint32_t Py_PACK_VERSION(int major, int minor)
+
+   Equivalent to ``Py_PACK_FULL_VERSION(major, minor, 0, 0, 0)``.
+   The result does not correspond to any Python release, but is useful
+   in numeric comparisons.
+
+   .. versionadded:: 3.14

Doc/c-api/init.rst

@@ -1492,6 +1492,17 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 
    .. versionadded:: 3.8
 
+
+.. c:function:: PyObject* PyUnstable_InterpreterState_GetMainModule(PyInterpreterState *interp)
+
+   Return a :term:`strong reference` to the ``__main__`` `module object <moduleobjects>`_
+   for the given interpreter.
+
+   The caller must hold the GIL.
+
+   .. versionadded:: 3.13
+
+
 .. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag)
 
    Type of a frame evaluation function.

Doc/data/stable_abi.dat

@@ -57,6 +57,11 @@ Pending removal in Python 3.16
     In the rare case that you need the bitwise inversion of
     the underlying integer, convert to ``int`` explicitly (``~int(x)``).
 
+* :mod:`functools`:
+
+  * Calling the Python implementation of :func:`functools.reduce` with *function*
+    or *sequence* as keyword arguments has been deprecated since Python 3.14.
+
 * :mod:`shutil`:
 
   * The :class:`!ExecError` exception
@@ -79,8 +84,3 @@ Pending removal in Python 3.16
 
   * The undocumented and unused :attr:`!TarFile.tarfile` attribute
     has been deprecated since Python 3.13.
-
-* :mod:`functools`:
-
-  * Calling the Python implementation of :func:`functools.reduce` with *function*
-    or *sequence* as keyword arguments has been deprecated since Python 3.14.

Doc/deprecations/pending-removal-in-3.16.rst

@@ -246,6 +246,9 @@ Scheduling callbacks
    another thread, this function *must* be used, since :meth:`call_soon` is not
    thread-safe.
 
+   This function is safe to be called from a reentrant context or signal handler,
+   however, it is not safe or fruitful to use the returned handle in such contexts.
+
    Raises :exc:`RuntimeError` if called on a loop that's been closed.
    This can happen on a secondary thread when the main application is
    shutting down.
@@ -967,6 +970,9 @@ Watching file descriptors
    invoke *callback* with the specified arguments once *fd* is available for
    reading.
 
+   Any preexisting callback registered for *fd* is cancelled and replaced by
+   *callback*.
+
 .. method:: loop.remove_reader(fd)
 
    Stop monitoring the *fd* file descriptor for read availability. Returns
@@ -978,6 +984,9 @@ Watching file descriptors
    invoke *callback* with the specified arguments once *fd* is available for
    writing.
 
+   Any preexisting callback registered for *fd* is cancelled and replaced by
+   *callback*.
+
    Use :func:`functools.partial` :ref:`to pass keyword arguments
    <asyncio-pass-keywords>` to *callback*.
 

Doc/library/asyncio-eventloop.rst

@@ -115,11 +115,11 @@ Queue
 
    .. method:: task_done()
 
-      Indicate that a formerly enqueued task is complete.
+      Indicate that a formerly enqueued work item is complete.
 
       Used by queue consumers. For each :meth:`~Queue.get` used to
-      fetch a task, a subsequent call to :meth:`task_done` tells the
-      queue that the processing on the task is complete.
+      fetch a work item, a subsequent call to :meth:`task_done` tells the
+      queue that the processing on the work item is complete.
 
       If a :meth:`join` is currently blocking, it will resume when all
       items have been processed (meaning that a :meth:`task_done`

Doc/library/asyncio-queue.rst

@@ -870,6 +870,36 @@ invalid non-\ ``NULL`` pointers would crash Python)::
    ValueError: NULL pointer access
    >>>
 
+.. _ctypes-thread-safety:
+
+Thread safety without the GIL
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In Python 3.13, the :term:`GIL` may be disabled on :term:`experimental free threaded <free threading>` builds.
+In ctypes, reads and writes to a single object concurrently is safe, but not across multiple objects:
+
+   .. code-block:: pycon
+
+      >>> number = c_int(42)
+      >>> pointer_a = pointer(number)
+      >>> pointer_b = pointer(number)
+
+In the above, it's only safe for one object to read and write to the address at once if the GIL is disabled.
+So, ``pointer_a`` can be shared and written to across multiple threads, but only if ``pointer_b``
+is not also attempting to do the same. If this is an issue, consider using a :class:`threading.Lock`
+to synchronize access to memory:
+
+   .. code-block:: pycon
+
+      >>> import threading
+      >>> lock = threading.Lock()
+      >>> # Thread 1
+      >>> with lock:
+      ...    pointer_a.contents = 24
+      >>> # Thread 2
+      >>> with lock:
+      ...    pointer_b.contents = 42
+
 
 .. _ctypes-type-conversions:
 

Doc/library/ctypes.rst

@@ -46,9 +46,15 @@ module.  See module :mod:`glob` for pathname expansion (:mod:`glob` uses
 a period are not special for this module, and are matched by the ``*`` and ``?``
 patterns.
 
-Also note that :func:`functools.lru_cache` with the *maxsize* of 32768 is used to
-cache the compiled regex patterns in the following functions: :func:`fnmatch`,
-:func:`fnmatchcase`, :func:`.filter`.
+Unless stated otherwise, "filename string" and "pattern string" either refer to
+:class:`str` or ``ISO-8859-1`` encoded :class:`bytes` objects. Note that the
+functions documented below do not allow to mix a :class:`!bytes` pattern with
+a :class:`!str` filename, and vice-versa.
+
+Finally, note that :func:`functools.lru_cache` with a *maxsize* of 32768
+is used to cache the (typed) compiled regex patterns in the following
+functions: :func:`fnmatch`, :func:`fnmatchcase`, :func:`.filter`.
+
 
 .. function:: fnmatch(name, pat)
 
@@ -78,16 +84,16 @@ cache the compiled regex patterns in the following functions: :func:`fnmatch`,
 
 .. function:: filter(names, pat)
 
-   Construct a list from those elements of the :term:`iterable` *names*
-   that match pattern *pat*.
+   Construct a list from those elements of the :term:`iterable` of filename
+   strings *names* that match the pattern string *pat*.
    It is the same as ``[n for n in names if fnmatch(n, pat)]``,
    but implemented more efficiently.
 
 
 .. function:: translate(pat)
 
    Return the shell-style pattern *pat* converted to a regular expression for
-   using with :func:`re.match`.
+   using with :func:`re.match`. The pattern is expected to be a :class:`str`.
 
    Example: