Merge remote-tracking branch 'upstream/main' into class_stoptheworld

Author: Fidget-Spinner

.github/CODEOWNERS

@@ -42,7 +42,7 @@ Python/bytecodes.c            @markshannon
 Python/optimizer*.c           @markshannon
 Python/optimizer_analysis.c   @Fidget-Spinner
 Python/optimizer_bytecodes.c  @Fidget-Spinner
-Python/symtable.c             @JelleZijlstra @carljm 
+Python/symtable.c             @JelleZijlstra @carljm
 Lib/_pyrepl/*                 @pablogsal @lysnikolaou @ambv
 Lib/test/test_patma.py        @brandtbucher
 Lib/test/test_type_*.py       @JelleZijlstra
@@ -201,7 +201,6 @@ Doc/c-api/stable.rst          @encukou
 **/*itertools*                @rhettinger
 **/*collections*              @rhettinger
 **/*random*                   @rhettinger
-**/*queue*                    @rhettinger
 **/*bisect*                   @rhettinger
 **/*heapq*                    @rhettinger
 **/*functools*                @rhettinger

Doc/c-api/weakref.rst

@@ -96,3 +96,19 @@ as much as it can.
    This iterates through the weak references for *object* and calls callbacks
    for those references which have one. It returns when all callbacks have
    been attempted.
+
+
+.. c:function:: void PyUnstable_Object_ClearWeakRefsNoCallbacks(PyObject *object)
+
+   Clears the weakrefs for *object* without calling the callbacks.
+
+   This function is called by the :c:member:`~PyTypeObject.tp_dealloc` handler
+   for types with finalizers (i.e., :meth:`~object.__del__`).  The handler for
+   those objects first calls :c:func:`PyObject_ClearWeakRefs` to clear weakrefs
+   and call their callbacks, then the finalizer, and finally this function to
+   clear any weakrefs that may have been created by the finalizer.
+
+   In most circumstances, it's more appropriate to use
+   :c:func:`PyObject_ClearWeakRefs` to clear weakrefs instead of this function.
+
+   .. versionadded:: 3.13

Doc/data/stable_abi.dat

@@ -694,6 +694,9 @@ Glossary
 
          CPython does not consistently apply the requirement that an iterator
          define :meth:`~iterator.__iter__`.
+         And also please note that the free-threading CPython does not guarantee
+         the thread-safety of iterator operations.
+
 
    key function
       A key function or collation function is a callable that returns a value

Doc/glossary.rst

@@ -0,0 +1,254 @@
+.. highlight:: c
+
+.. _freethreading-extensions-howto:
+
+******************************************
+C API Extension Support for Free Threading
+******************************************
+
+Starting with the 3.13 release, CPython has experimental support for running
+with the :term:`global interpreter lock` (GIL) disabled in a configuration
+called :term:`free threading`.  This document describes how to adapt C API
+extensions to support free threading.
+
+
+Identifying the Free-Threaded Build in C
+========================================
+
+The CPython C API exposes the ``Py_GIL_DISABLED`` macro: in the free-threaded
+build it's defined to ``1``, and in the regular build it's not defined.
+You can use it to enable code that only runs under the free-threaded build::
+
+    #ifdef Py_GIL_DISABLED
+    /* code that only runs in the free-threaded build */
+    #endif
+
+Module Initialization
+=====================
+
+Extension modules need to explicitly indicate that they support running with
+the GIL disabled; otherwise importing the extension will raise a warning and
+enable the GIL at runtime.
+
+There are two ways to indicate that an extension module supports running with
+the GIL disabled depending on whether the extension uses multi-phase or
+single-phase initialization.
+
+Multi-Phase Initialization
+..........................
+
+Extensions that use multi-phase initialization (i.e.,
+:c:func:`PyModuleDef_Init`) should add a :c:data:`Py_mod_gil` slot in the
+module definition.  If your extension supports older versions of CPython,
+you should guard the slot with a :c:data:`PY_VERSION_HEX` check.
+
+::
+
+    static struct PyModuleDef_Slot module_slots[] = {
+        ...
+    #if PY_VERSION_HEX >= 0x030D0000
+        {Py_mod_gil, Py_MOD_GIL_NOT_USED},
+    #endif
+        {0, NULL}
+    };
+
+    static struct PyModuleDef moduledef = {
+        PyModuleDef_HEAD_INIT,
+        .m_slots = module_slots,
+        ...
+    };
+
+
+Single-Phase Initialization
+...........................
+
+Extensions that use single-phase initialization (i.e.,
+:c:func:`PyModule_Create`) should call :c:func:`PyUnstable_Module_SetGIL` to
+indicate that they support running with the GIL disabled.  The function is
+only defined in the free-threaded build, so you should guard the call with
+``#ifdef Py_GIL_DISABLED`` to avoid compilation errors in the regular build.
+
+::
+
+    static struct PyModuleDef moduledef = {
+        PyModuleDef_HEAD_INIT,
+        ...
+    };
+
+    PyMODINIT_FUNC
+    PyInit_mymodule(void)
+    {
+        PyObject *m = PyModule_Create(&moduledef);
+        if (m == NULL) {
+            return NULL;
+        }
+    #ifdef Py_GIL_DISABLED
+        PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
+    #endif
+        return m;
+    }
+
+
+General API Guidelines
+======================
+
+Most of the C API is thread-safe, but there are some exceptions.
+
+* **Struct Fields**: Accessing fields in Python C API objects or structs
+  directly is not thread-safe if the field may be concurrently modified.
+* **Macros**: Accessor macros like :c:macro:`PyList_GET_ITEM` and
+  :c:macro:`PyList_SET_ITEM` do not perform any error checking or locking.
+  These macros are not thread-safe if the container object may be modified
+  concurrently.
+* **Borrowed References**: C API functions that return
+  :term:`borrowed references <borrowed reference>` may not be thread-safe if
+  the containing object is modified concurrently.  See the section on
+  :ref:`borrowed references <borrowed-references>` for more information.
+
+
+Container Thread Safety
+.......................
+
+Containers like :c:struct:`PyListObject`,
+:c:struct:`PyDictObject`, and :c:struct:`PySetObject` perform internal locking
+in the free-threaded build.  For example, the :c:func:`PyList_Append` will
+lock the list before appending an item.
+
+
+Borrowed References
+===================
+
+.. _borrowed-references:
+
+Some C API functions return :term:`borrowed references <borrowed reference>`.
+These APIs are not thread-safe if the containing object is modified
+concurrently.  For example, it's not safe to use :c:func:`PyList_GetItem`
+if the list may be modified concurrently.
+
+The following table lists some borrowed reference APIs and their replacements
+that return :term:`strong references <strong reference>`.
+
++-----------------------------------+-----------------------------------+
+| Borrowed reference API            | Strong reference API              |
++===================================+===================================+
+| :c:func:`PyList_GetItem`          | :c:func:`PyList_GetItemRef`       |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyDict_GetItem`          | :c:func:`PyDict_GetItemRef`       |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyDict_GetItemWithError` | :c:func:`PyDict_GetItemRef`       |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyDict_GetItemString`    | :c:func:`PyDict_GetItemStringRef` |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyDict_SetDefault`       | :c:func:`PyDict_SetDefaultRef`    |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyDict_Next`             | no direct replacement             |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyWeakref_GetObject`     | :c:func:`PyWeakref_GetRef`        |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyWeakref_GET_OBJECT`    | :c:func:`PyWeakref_GetRef`        |
++-----------------------------------+-----------------------------------+
+| :c:func:`PyImport_AddModule`      | :c:func:`PyImport_AddModuleRef`   |
++-----------------------------------+-----------------------------------+
+
+Not all APIs that return borrowed references are problematic.  For
+example, :c:func:`PyTuple_GetItem` is safe because tuples are immutable.
+Similarly, not all uses of the above APIs are problematic.  For example,
+:c:func:`PyDict_GetItem` is often used for parsing keyword argument
+dictionaries in function calls; those keyword argument dictionaries are
+effectively private (not accessible by other threads), so using borrowed
+references in that context is safe.
+
+Some of these functions were added in Python 3.13.  You can use the
+`pythoncapi-compat <https://github.com/python/pythoncapi-compat>`_ package
+to provide implementations of these functions for older Python versions.
+
+
+Memory Allocation APIs
+======================
+
+Python's memory management C API provides functions in three different
+:ref:`allocation domains <allocator-domains>`: "raw", "mem", and "object".
+For thread-safety, the free-threaded build requires that only Python objects
+are allocated using the object domain, and that all Python object are
+allocated using that domain.  This differes from the prior Python versions,
+where this was only a best practice and not a hard requirement.
+
+.. note::
+
+   Search for uses of :c:func:`PyObject_Malloc` in your
+   extension and check that the allocated memory is used for Python objects.
+   Use :c:func:`PyMem_Malloc` to allocate buffers instead of
+   :c:func:`PyObject_Malloc`.
+
+
+Thread State and GIL APIs
+=========================
+
+Python provides a set of functions and macros to manage thread state and the
+GIL, such as:
+
+* :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release`
+* :c:func:`PyEval_SaveThread` and :c:func:`PyEval_RestoreThread`
+* :c:macro:`Py_BEGIN_ALLOW_THREADS` and :c:macro:`Py_END_ALLOW_THREADS`
+
+These functions should still be used in the free-threaded build to manage
+thread state even when the :term:`GIL` is disabled.  For example, if you
+create a thread outside of Python, you must call :c:func:`PyGILState_Ensure`
+before calling into the Python API to ensure that the thread has a valid
+Python thread state.
+
+You should continue to call :c:func:`PyEval_SaveThread` or
+:c:macro:`Py_BEGIN_ALLOW_THREADS` around blocking operations, such as I/O or
+lock acquisitions, to allow other threads to run the
+:term:`cyclic garbage collector <garbage collection>`.
+
+
+Protecting Internal Extension State
+===================================
+
+Your extension may have internal state that was previously protected by the
+GIL.  You may need to add locking to protect this state.  The approach will
+depend on your extension, but some common patterns include:
+
+* **Caches**: global caches are a common source of shared state.  Consider
+  using a lock to protect the cache or disabling it in the free-threaded build
+  if the cache is not critical for performance.
+* **Global State**: global state may need to be protected by a lock or moved
+  to thread local storage. C11 and C++11 provide the ``thread_local`` or
+  ``_Thread_local`` for
+  `thread-local storage <https://en.cppreference.com/w/c/language/storage_duration>`_.
+
+
+Building Extensions for the Free-Threaded Build
+===============================================
+
+C API extensions need to be built specifically for the free-threaded build.
+The wheels, shared libraries, and binaries are indicated by a ``t`` suffix.
+
+* `pypa/manylinux <https://github.com/pypa/manylinux>`_ supports the
+  free-threaded build, with the ``t`` suffix, such as ``python3.13t``.
+* `pypa/cibuildwheel <https://github.com/pypa/cibuildwheel>`_ supports the
+  free-threaded build if you set
+  `CIBW_FREE_THREADED_SUPPORT <https://cibuildwheel.pypa.io/en/stable/options/#free-threaded-support>`_.
+
+Limited C API and Stable ABI
+............................
+
+The free-threaded build does not currently support the
+:ref:`Limited C API <limited-c-api>` or the stable ABI.  If you use
+`setuptools <https://setuptools.pypa.io/en/latest/setuptools.html>`_ to build
+your extension and currently set ``py_limited_api=True`` you can use
+``py_limited_api=not sysconfig.get_config_var("Py_GIL_DISABLED")`` to opt out
+of the limited API when building with the free-threaded build.
+
+.. note::
+    You will need to build separate wheels specifically for the free-threaded
+    build.  If you currently use the stable ABI, you can continue to build a
+    single wheel for multiple non-free-threaded Python versions.
+
+
+Windows
+.......
+
+Due to a limitation of the official Windows installer, you will need to
+manually define ``Py_GIL_DISABLED=1`` when building extensions from source.

Doc/howto/free-threading-extensions.rst

@@ -32,6 +32,7 @@ Python Library Reference.
    isolating-extensions.rst
    timerfd.rst
    mro.rst
+   free-threading-extensions.rst
 
 General:
 

Doc/howto/index.rst

@@ -780,16 +780,6 @@ not have to be) the original ``STACK[-2]``.
    .. versionadded:: 3.12
 
 
-.. opcode:: BEFORE_ASYNC_WITH
-
-   Resolves ``__aenter__`` and ``__aexit__`` from ``STACK[-1]``.
-   Pushes ``__aexit__`` and result of ``__aenter__()`` to the stack::
-
-      STACK.extend((__aexit__, __aenter__())
-
-   .. versionadded:: 3.5
-
-
 
 **Miscellaneous opcodes**
 
@@ -944,18 +934,6 @@ iterations of the loop.
    Pushes :func:`!builtins.__build_class__` onto the stack.  It is later called
    to construct a class.
 
-
-.. opcode:: BEFORE_WITH
-
-   This opcode performs several operations before a with block starts.  First,
-   it loads :meth:`~object.__exit__` from the context manager and pushes it onto
-   the stack for later use by :opcode:`WITH_EXCEPT_START`.  Then,
-   :meth:`~object.__enter__` is called. Finally, the result of calling the
-   ``__enter__()`` method is pushed onto the stack.
-
-   .. versionadded:: 3.11
-
-
 .. opcode:: GET_LEN
 
    Perform ``STACK.append(len(STACK[-1]))``.
@@ -1812,6 +1790,17 @@ iterations of the loop.
    .. versionadded:: 3.12
 
 
+.. opcode:: LOAD_SPECIAL
+
+   Performs special method lookup on ``STACK[-1]``.
+   If ``type(STACK[-1]).__xxx__`` is a method, leave
+   ``type(STACK[-1]).__xxx__; STACK[-1]`` on the stack.
+   If ``type(STACK[-1]).__xxx__`` is not a method, leave
+   ``STACK[-1].__xxx__; NULL`` on the stack.
+
+   .. versionadded:: 3.14
+
+
 **Pseudo-instructions**
 
 These opcodes do not appear in Python bytecode. They are used by the compiler

Doc/library/dis.rst

@@ -504,9 +504,9 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
    are true.
 
    This, for example, is true of ``int.__add__``.  An object passing this test
-   has a :meth:`~object.__get__` method but not a :meth:`~object.__set__`
-   method, but beyond that the set of attributes varies.  A
-   :attr:`~definition.__name__` attribute is usually
+   has a :meth:`~object.__get__` method, but not a :meth:`~object.__set__`
+   method or a :meth:`~object.__delete__` method.  Beyond that, the set of
+   attributes varies.  A :attr:`~definition.__name__` attribute is usually
    sensible, and :attr:`!__doc__` often is.
 
    Methods implemented via descriptors that also pass one of the other tests
@@ -515,6 +515,11 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
    :attr:`~method.__func__` attribute (etc) when an object passes
    :func:`ismethod`.
 
+   .. versionchanged:: 3.13
+      This function no longer incorrectly reports objects with :meth:`~object.__get__`
+      and :meth:`~object.__delete__`, but not :meth:`~object.__set__`, as being method
+      descriptors (such objects are data descriptors, not method descriptors).
+
 
 .. function:: isdatadescriptor(object)