Merge branch 'main' into TracebackException-swallows-cause-context-flasey-exception

Author: YvesDup

Doc/howto/curses.rst

@@ -145,8 +145,8 @@ importing the :func:`curses.wrapper` function and using it like this::
            v = i-10
            stdscr.addstr(i, 0, '10 divided by {} is {}'.format(v, 10/v))
 
-       stdscr.refresh()
-       stdscr.getkey()
+           stdscr.refresh()
+           stdscr.getkey()
 
    wrapper(main)
 

Doc/howto/free-threading-python.rst

@@ -152,3 +152,33 @@ to re-enable it in a thread-safe way in the 3.14 release.  This overhead is
 expected to be reduced in upcoming Python release.   We are aiming for an
 overhead of 10% or less on the pyperformance suite compared to the default
 GIL-enabled build.
+
+
+Behavioral changes
+==================
+
+This section describes CPython behavioural changes with the free-threaded
+build.
+
+
+Context variables
+-----------------
+
+In the free-threaded build, the flag :data:`~sys.flags.thread_inherit_context`
+is set to true by default which causes threads created with
+:class:`threading.Thread` to start with a copy of the
+:class:`~contextvars.Context()` of the caller of
+:meth:`~threading.Thread.start`.  In the default GIL-enabled build, the flag
+defaults to false so threads start with an
+empty :class:`~contextvars.Context()`.
+
+
+Warning filters
+---------------
+
+In the free-threaded build, the flag :data:`~sys.flags.context_aware_warnings`
+is set to true by default.  In the default GIL-enabled build, the flag defaults
+to false.  If the flag is true then the :class:`warnings.catch_warnings`
+context manager uses a context variable for warning filters.  If the flag is
+false then :class:`~warnings.catch_warnings` modifies the global filters list,
+which is not thread-safe.  See the :mod:`warnings` module for more details.

Doc/includes/typestruct.h

@@ -54,11 +54,11 @@ typedef struct _typeobject {
     iternextfunc tp_iternext;
 
     /* Attribute descriptor and subclassing stuff */
-    struct PyMethodDef *tp_methods;
-    struct PyMemberDef *tp_members;
-    struct PyGetSetDef *tp_getset;
+    PyMethodDef *tp_methods;
+    PyMemberDef *tp_members;
+    PyGetSetDef *tp_getset;
     // Strong reference on a heap type, borrowed reference on a static type
-    struct _typeobject *tp_base;
+    PyTypeObject *tp_base;
     PyObject *tp_dict;
     descrgetfunc tp_descr_get;
     descrsetfunc tp_descr_set;
@@ -70,17 +70,26 @@ typedef struct _typeobject {
     inquiry tp_is_gc; /* For PyObject_IS_GC */
     PyObject *tp_bases;
     PyObject *tp_mro; /* method resolution order */
-    PyObject *tp_cache;
-    PyObject *tp_subclasses;
-    PyObject *tp_weaklist;
+    PyObject *tp_cache; /* no longer used */
+    void *tp_subclasses;  /* for static builtin types this is an index */
+    PyObject *tp_weaklist; /* not used for static builtin types */
     destructor tp_del;
 
-    /* Type attribute cache version tag. Added in version 2.6 */
+    /* Type attribute cache version tag. Added in version 2.6.
+     * If zero, the cache is invalid and must be initialized.
+     */
     unsigned int tp_version_tag;
 
     destructor tp_finalize;
     vectorcallfunc tp_vectorcall;
 
     /* bitset of which type-watchers care about this type */
     unsigned char tp_watched;
+
+    /* Number of tp_version_tag values used.
+     * Set to _Py_ATTR_CACHE_UNUSED if the attribute cache is
+     * disabled for this type (e.g. due to custom MRO entries).
+     * Otherwise, limited to MAX_VERSIONS_PER_CLASS (defined elsewhere).
+     */
+    uint16_t tp_versions_used;
 } PyTypeObject;

Doc/library/contextlib.rst

@@ -49,7 +49,7 @@ Functions and classes provided:
 
    While many objects natively support use in with statements, sometimes a
    resource needs to be managed that isn't a context manager in its own right,
-   and doesn't implement a ``close()`` method for use with ``contextlib.closing``
+   and doesn't implement a ``close()`` method for use with ``contextlib.closing``.
 
    An abstract example would be the following to ensure correct resource
    management::

Doc/library/decimal.rst

@@ -1884,13 +1884,20 @@ the current thread.
 
 If :func:`setcontext` has not been called before :func:`getcontext`, then
 :func:`getcontext` will automatically create a new context for use in the
-current thread.
-
-The new context is copied from a prototype context called *DefaultContext*. To
-control the defaults so that each thread will use the same values throughout the
-application, directly modify the *DefaultContext* object. This should be done
-*before* any threads are started so that there won't be a race condition between
-threads calling :func:`getcontext`. For example::
+current thread.  New context objects have default values set from the
+:data:`decimal.DefaultContext` object.
+
+The :data:`sys.flags.thread_inherit_context` flag affects the context for
+new threads.  If the flag is false, new threads will start with an empty
+context.  In this case, :func:`getcontext` will create a new context object
+when called and use the default values from *DefaultContext*.  If the flag
+is true, new threads will start with a copy of context from the caller of
+:meth:`threading.Thread.start`.
+
+To control the defaults so that each thread will use the same values throughout
+the application, directly modify the *DefaultContext* object. This should be
+done *before* any threads are started so that there won't be a race condition
+between threads calling :func:`getcontext`. For example::
 
    # Set applicationwide defaults for all threads about to be launched
    DefaultContext.prec = 12

Doc/library/gzip.rst

@@ -122,9 +122,7 @@ The module defines the following items:
    .. method:: peek(n)
 
       Read *n* uncompressed bytes without advancing the file position.
-      At most one single read on the compressed stream is done to satisfy
-      the call.  The number of bytes returned may be more or less than
-      requested.
+      The number of bytes returned may be more or less than requested.
 
       .. note:: While calling :meth:`peek` does not change the file position of
          the :class:`GzipFile`, it may change the position of the underlying

Doc/library/sys.rst

@@ -535,7 +535,8 @@ always available. Unless explicitly noted otherwise, all variables are read-only
 .. data:: flags
 
    The :term:`named tuple` *flags* exposes the status of command line
-   flags. The attributes are read only.
+   flags.  Flags should only be accessed only by name and not by index.  The
+   attributes are read only.
 
    .. list-table::
 
@@ -594,6 +595,18 @@ always available. Unless explicitly noted otherwise, all variables are read-only
       * - .. attribute:: flags.warn_default_encoding
         - :option:`-X warn_default_encoding <-X>`
 
+      * - .. attribute:: flags.gil
+        - :option:`-X gil <-X>` and :envvar:`PYTHON_GIL`
+
+      * - .. attribute:: flags.thread_inherit_context
+        - :option:`-X thread_inherit_context <-X>` and
+          :envvar:`PYTHON_THREAD_INHERIT_CONTEXT`
+
+      * - .. attribute:: flags.context_aware_warnings
+        - :option:`-X context_aware_warnings <-X>` and
+          :envvar:`PYTHON_CONTEXT_AWARE_WARNINGS`
+
+
    .. versionchanged:: 3.2
       Added ``quiet`` attribute for the new :option:`-q` flag.
 
@@ -620,6 +633,15 @@ always available. Unless explicitly noted otherwise, all variables are read-only
    .. versionchanged:: 3.11
       Added the ``int_max_str_digits`` attribute.
 
+   .. versionchanged:: 3.13
+      Added the ``gil`` attribute.
+
+   .. versionchanged:: 3.14
+      Added the ``thread_inherit_context`` attribute.
+
+   .. versionchanged:: 3.14
+      Added the ``context_aware_warnings`` attribute.
+
 
 .. data:: float_info
 

Doc/library/threading.rst

@@ -334,7 +334,7 @@ since it is impossible to detect the termination of alien threads.
 
 
 .. class:: Thread(group=None, target=None, name=None, args=(), kwargs={}, *, \
-                  daemon=None)
+                  daemon=None, context=None)
 
    This constructor should always be called with keyword arguments.  Arguments
    are:
@@ -359,6 +359,16 @@ since it is impossible to detect the termination of alien threads.
    If ``None`` (the default), the daemonic property is inherited from the
    current thread.
 
+   *context* is the :class:`~contextvars.Context` value to use when starting
+   the thread.  The default value is ``None`` which indicates that the
+   :data:`sys.flags.thread_inherit_context` flag controls the behaviour.  If
+   the flag is true, threads will start with a copy of the context of the
+   caller of :meth:`~Thread.start`.  If false, they will start with an empty
+   context.  To explicitly start with an empty context, pass a new instance of
+   :class:`~contextvars.Context()`.  To explicitly start with a copy of the
+   current context, pass the value from :func:`~contextvars.copy_context`. The
+   flag defaults true on free-threaded builds and false otherwise.
+
    If the subclass overrides the constructor, it must make sure to invoke the
    base class constructor (``Thread.__init__()``) before doing anything else to
    the thread.
@@ -369,6 +379,9 @@ since it is impossible to detect the termination of alien threads.
    .. versionchanged:: 3.10
       Use the *target* name if *name* argument is omitted.
 
+   .. versionchanged:: 3.14
+      Added the *context* parameter.
+
    .. method:: start()
 
       Start the thread's activity.

Doc/library/warnings.rst

@@ -324,11 +324,13 @@ the warning using the :class:`catch_warnings` context manager::
 While within the context manager all warnings will simply be ignored. This
 allows you to use known-deprecated code without having to see the warning while
 not suppressing the warning for other code that might not be aware of its use
-of deprecated code.  Note: this can only be guaranteed in a single-threaded
-application. If two or more threads use the :class:`catch_warnings` context
-manager at the same time, the behavior is undefined.
+of deprecated code.
 
+    .. note::
 
+        See :ref:`warning-concurrent-safe` for details on the
+        concurrency-safety of the :class:`catch_warnings` context manager when
+        used in programs using multiple threads or async functions.
 
 .. _warning-testing:
 
@@ -364,10 +366,13 @@ the warning has been cleared.
 Once the context manager exits, the warnings filter is restored to its state
 when the context was entered. This prevents tests from changing the warnings
 filter in unexpected ways between tests and leading to indeterminate test
-results. The :func:`showwarning` function in the module is also restored to
-its original value.  Note: this can only be guaranteed in a single-threaded
-application. If two or more threads use the :class:`catch_warnings` context
-manager at the same time, the behavior is undefined.
+results.
+
+    .. note::
+
+        See :ref:`warning-concurrent-safe` for details on the
+        concurrency-safety of the :class:`catch_warnings` context manager when
+        used in programs using multiple threads or async functions.
 
 When testing multiple operations that raise the same kind of warning, it
 is important to test them in a manner that confirms each operation is raising
@@ -615,12 +620,71 @@ Available Context Managers
 
     .. note::
 
-        The :class:`catch_warnings` manager works by replacing and
-        then later restoring the module's
-        :func:`showwarning` function and internal list of filter
-        specifications.  This means the context manager is modifying
-        global state and therefore is not thread-safe.
+        See :ref:`warning-concurrent-safe` for details on the
+        concurrency-safety of the :class:`catch_warnings` context manager when
+        used in programs using multiple threads or async functions.
+
 
     .. versionchanged:: 3.11
 
         Added the *action*, *category*, *lineno*, and *append* parameters.
+
+
+.. _warning-concurrent-safe:
+
+Concurrent safety of Context Managers
+-------------------------------------
+
+The behavior of :class:`catch_warnings` context manager depends on the
+:data:`sys.flags.context_aware_warnings` flag.  If the flag is true, the
+context manager behaves in a concurrent-safe fashion and otherwise not.
+Concurrent-safe means that it is both thread-safe and safe to use within
+:ref:`asyncio coroutines <coroutine>` and tasks.  Being thread-safe means
+that behavior is predictable in a multi-threaded program.  The flag defaults
+to true for free-threaded builds and false otherwise.
+
+If the :data:`~sys.flags.context_aware_warnings` flag is false, then
+:class:`catch_warnings` will modify the global attributes of the
+:mod:`warnings` module.  This is not safe if used within a concurrent program
+(using multiple threads or using asyncio coroutines).  For example, if two
+or more threads use the :class:`catch_warnings` class at the same time, the
+behavior is undefined.
+
+If the flag is true, :class:`catch_warnings` will not modify global
+attributes and will instead use a :class:`~contextvars.ContextVar` to
+store the newly established warning filtering state.  A context variable
+provides thread-local storage and it makes the use of :class:`catch_warnings`
+thread-safe.
+
+The *record* parameter of the context handler also behaves differently
+depending on the value of the flag.  When *record* is true and the flag is
+false, the context manager works by replacing and then later restoring the
+module's :func:`showwarning` function.  That is not concurrent-safe.
+
+When *record* is true and the flag is true, the :func:`showwarning` function
+is not replaced.  Instead, the recording status is indicated by an internal
+property in the context variable.  In this case, the :func:`showwarning`
+function will not be restored when exiting the context handler.
+
+The :data:`~sys.flags.context_aware_warnings` flag can be set the :option:`-X
+context_aware_warnings<-X>` command-line option or by the
+:envvar:`PYTHON_CONTEXT_AWARE_WARNINGS` environment variable.
+
+    .. note::
+
+        It is likely that most programs that desire thread-safe
+        behaviour of the warnings module will also want to set the
+        :data:`~sys.flags.thread_inherit_context` flag to true.  That flag
+        causes threads created by :class:`threading.Thread` to start
+        with a copy of the context variables from the thread starting
+        it.  When true, the context established by :class:`catch_warnings`
+        in one thread will also apply to new threads started by it.  If false,
+        new threads will start with an empty warnings context variable,
+        meaning that any filtering that was established by a
+        :class:`catch_warnings` context manager will no longer be active.
+
+.. versionchanged:: 3.14
+
+   Added the :data:`sys.flags.context_aware_warnings` flag and the use of a
+   context variable for :class:`catch_warnings` if the flag is true.  Previous
+   versions of Python acted as if the flag was always set to false.

Doc/using/cmdline.rst

@@ -639,6 +639,23 @@ Miscellaneous options
 
      .. versionadded:: 3.13
 
+   * :samp:`-X thread_inherit_context={0,1}` causes :class:`~threading.Thread`
+     to, by default, use a copy of context of of the caller of
+     ``Thread.start()`` when starting.  Otherwise, threads will start
+     with an empty context.  If unset, the value of this option defaults
+     to ``1`` on free-threaded builds and to ``0`` otherwise.  See also
+     :envvar:`PYTHON_THREAD_INHERIT_CONTEXT`.
+
+     .. versionadded:: 3.14
+
+   * :samp:`-X context_aware_warnings={0,1}` causes the
+     :class:`warnings.catch_warnings` context manager to use a
+     :class:`~contextvars.ContextVar` to store warnings filter state.  If
+     unset, the value of this option defaults to ``1`` on free-threaded builds
+     and to ``0`` otherwise.  See also :envvar:`PYTHON_CONTEXT_AWARE_WARNINGS`.
+
+     .. versionadded:: 3.14
+
    It also allows passing arbitrary values and retrieving them through the
    :data:`sys._xoptions` dictionary.
 
@@ -1241,6 +1258,26 @@ conflict.
 
    .. versionadded:: 3.13
 
+.. envvar:: PYTHON_THREAD_INHERIT_CONTEXT
+
+   If this variable is set to ``1`` then :class:`~threading.Thread` will,
+   by default, use a copy of context of of the caller of ``Thread.start()``
+   when starting.  Otherwise, new threads will start with an empty context.
+   If unset, this variable defaults to ``1`` on free-threaded builds and to
+   ``0`` otherwise.  See also :option:`-X thread_inherit_context<-X>`.
+
+   .. versionadded:: 3.14
+
+.. envvar:: PYTHON_CONTEXT_AWARE_WARNINGS
+
+   If set to ``1`` then the :class:`warnings.catch_warnings` context
+   manager will use a :class:`~contextvars.ContextVar` to store warnings
+   filter state.  If unset, this variable defaults to ``1`` on
+   free-threaded builds and to ``0`` otherwise.  See :option:`-X
+   context_aware_warnings<-X>`.
+
+   .. versionadded:: 3.14
+
 Debug-mode variables
 ~~~~~~~~~~~~~~~~~~~~