Merge branch 'main' into patch-3

Author: Harikrishna-Srinivasan

Doc/data/stable_abi.dat

@@ -249,11 +249,12 @@ The following sections describe how each of these are used.
 prog
 ^^^^
 
-By default, :class:`ArgumentParser` objects use ``sys.argv[0]`` to determine
+By default, :class:`ArgumentParser` objects use the base name
+(see :func:`os.path.basename`) of ``sys.argv[0]`` to determine
 how to display the name of the program in help messages.  This default is almost
-always desirable because it will make the help messages match how the program was
-invoked on the command line.  For example, consider a file named
-``myprogram.py`` with the following code::
+always desirable because it will make the help messages match the name that was
+used to invoke the program on the command line.  For example, consider a file
+named ``myprogram.py`` with the following code::
 
    import argparse
    parser = argparse.ArgumentParser()

Doc/library/argparse.rst

@@ -1458,6 +1458,23 @@ These can be used as types in annotations. They all support subscription using
         >>> X.__metadata__
         ('very', 'important', 'metadata')
 
+   * At runtime, if you want to retrieve the original
+     type wrapped by ``Annotated``, use the :attr:`!__origin__` attribute:
+
+     .. doctest::
+
+        >>> from typing import Annotated, get_origin
+        >>> Password = Annotated[str, "secret"]
+        >>> Password.__origin__
+        <class 'str'>
+
+     Note that using :func:`get_origin` will return ``Annotated`` itself:
+
+     .. doctest::
+
+        >>> get_origin(Password)
+        typing.Annotated
+
    .. seealso::
 
       :pep:`593` - Flexible function and variable annotations
@@ -3298,6 +3315,7 @@ Introspection helpers
       assert get_origin(str) is None
       assert get_origin(Dict[str, int]) is dict
       assert get_origin(Union[int, str]) is Union
+      assert get_origin(Annotated[str, "metadata"]) is Annotated
       P = ParamSpec('P')
       assert get_origin(P.args) is P
       assert get_origin(P.kwargs) is P

Doc/library/typing.rst

@@ -70,6 +70,91 @@ Summary -- Release highlights
 New Features
 ============
 
+.. _whatsnew-314-pep649:
+
+PEP 649: Deferred Evaluation of Annotations
+-------------------------------------------
+
+The :term:`annotations <annotation>` on functions, classes, and modules are no
+longer evaluated eagerly. Instead, annotations are stored in special-purpose
+:term:`annotate functions <annotate function>` and evaluated only when
+necessary. This is specified in :pep:`649` and :pep:`749`.
+
+This change is designed to make annotations in Python more performant and more
+usable in most circumstances. The runtime cost for defining annotations is
+minimized, but it remains possible to introspect annotations at runtime.
+It is usually no longer necessary to enclose annotations in strings if they
+contain forward references.
+
+The new :mod:`annotationlib` module provides tools for inspecting deferred
+annotations. Annotations may be evaluated in the :attr:`~annotationlib.Format.VALUE`
+format (which evaluates annotations to runtime values, similar to the behavior in
+earlier Python versions), the :attr:`~annotationlib.Format.FORWARDREF` format
+(which replaces undefined names with special markers), and the
+:attr:`~annotationlib.Format.SOURCE` format (which returns annotations as strings).
+
+This example shows how these formats behave:
+
+.. doctest::
+
+   >>> from annotationlib import get_annotations, Format
+   >>> def func(arg: Undefined):
+   ...     pass
+   >>> get_annotations(func, format=Format.VALUE)
+   Traceback (most recent call last):
+     ...
+   NameError: name 'Undefined' is not defined
+   >>> get_annotations(func, format=Format.FORWARDREF)
+   {'arg': ForwardRef('Undefined')}
+   >>> get_annotations(func, format=Format.SOURCE)
+   {'arg': 'Undefined'}
+
+Implications for annotated code
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you define annotations in your code (for example, for use with a static type
+checker), then this change probably does not affect you: you can keep
+writing annotations the same way you did with previous versions of Python.
+
+You will likely be able to remove quoted strings in annotations, which are frequently
+used for forward references. Similarly, if you use ``from __future__ import annotations``
+to avoid having to write strings in annotations, you may well be able to
+remove that import. However, if you rely on third-party libraries that read annotations,
+those libraries may need changes to support unquoted annotations before they
+work as expected.
+
+Implications for readers of ``__annotations__``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If your code reads the ``__annotations__`` attribute on objects, you may want
+to make changes in order to support code that relies on deferred evaluation of
+annotations. For example, you may want to use :func:`annotationlib.get_annotations`
+with the :attr:`~annotationlib.Format.FORWARDREF` format, as the :mod:`dataclasses`
+module now does.
+
+Related changes
+^^^^^^^^^^^^^^^
+
+The changes in Python 3.14 are designed to rework how ``__annotations__``
+works at runtime while minimizing breakage to code that contains
+annotations in source code and to code that reads ``__annotations__``. However,
+if you rely on undocumented details of the annotation behavior or on private
+functions in the standard library, there are many ways in which your code may
+not work in Python 3.14. To safeguard your code against future changes,
+use only the documented functionality of the :mod:`annotationlib` module.
+
+``from __future__ import annotations``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In Python 3.7, :pep:`563` introduced the ``from __future__ import annotations``
+directive, which turns all annotations into strings. This directive is now
+considered deprecated and it is expected to be removed in a future version of Python.
+However, this removal will not happen until after Python 3.13, the last version of
+Python without deferred evaluation of annotations, reaches its end of life.
+In Python 3.14, the behavior of code using ``from __future__ import annotations``
+is unchanged.
+
+
 Improved Error Messages
 -----------------------
 
@@ -109,7 +194,9 @@ Other Language Changes
 New Modules
 ===========
 
-* None yet.
+* :mod:`annotationlib`: For introspecting :term:`annotations <annotation>`.
+  See :pep:`749` for more details.
+  (Contributed by Jelle Zijlstra in :gh:`119180`.)
 
 
 Improved Modules
@@ -563,9 +650,10 @@ New Features
 Porting to Python 3.14
 ----------------------
 
-* In the limited C API 3.14 and newer, :c:func:`Py_TYPE` is now implemented as
-  an opaque function call to hide implementation details.
-  (Contributed by Victor Stinner in :gh:`120600`.)
+* In the limited C API 3.14 and newer, :c:func:`Py_TYPE` and
+  :c:func:`Py_REFCNT` are now implemented as an opaque function call to hide
+  implementation details.
+  (Contributed by Victor Stinner in :gh:`120600` and :gh:`124127`.)
 
 
 Deprecated

Doc/whatsnew/3.14.rst

@@ -77,21 +77,29 @@ check by comparing the reference count field to the immortality reference count.
 #endif  // Py_GIL_DISABLED
 
 
-static inline Py_ssize_t Py_REFCNT(PyObject *ob) {
-#if !defined(Py_GIL_DISABLED)
-    return ob->ob_refcnt;
+// Py_REFCNT() implementation for the stable ABI
+PyAPI_FUNC(Py_ssize_t) Py_REFCNT(PyObject *ob);
+
+#if defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030e0000
+    // Stable ABI implements Py_REFCNT() as a function call
+    // on limited C API version 3.14 and newer.
 #else
-    uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
-    if (local == _Py_IMMORTAL_REFCNT_LOCAL) {
-        return _Py_IMMORTAL_REFCNT;
+    static inline Py_ssize_t _Py_REFCNT(PyObject *ob) {
+    #if !defined(Py_GIL_DISABLED)
+        return ob->ob_refcnt;
+    #else
+        uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
+        if (local == _Py_IMMORTAL_REFCNT_LOCAL) {
+            return _Py_IMMORTAL_REFCNT;
+        }
+        Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
+        return _Py_STATIC_CAST(Py_ssize_t, local) +
+               Py_ARITHMETIC_RIGHT_SHIFT(Py_ssize_t, shared, _Py_REF_SHARED_SHIFT);
+    #endif
     }
-    Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
-    return _Py_STATIC_CAST(Py_ssize_t, local) +
-           Py_ARITHMETIC_RIGHT_SHIFT(Py_ssize_t, shared, _Py_REF_SHARED_SHIFT);
-#endif
-}
-#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
-#  define Py_REFCNT(ob) Py_REFCNT(_PyObject_CAST(ob))
+    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
+    #  define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob))
+    #endif
 #endif
 
 

Include/refcount.h

@@ -1532,9 +1532,8 @@ def _get_positional_kwargs(self, dest, **kwargs):
 
         # mark positional arguments as required if at least one is
         # always required
-        if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]:
-            kwargs['required'] = True
-        if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:
+        nargs = kwargs.get('nargs')
+        if nargs not in [OPTIONAL, ZERO_OR_MORE, REMAINDER, SUPPRESS, 0]:
             kwargs['required'] = True
 
         # return the keyword arguments with no option strings
@@ -1949,9 +1948,8 @@ def take_action(action, argument_strings, option_string=None):
             argument_values = self._get_values(action, argument_strings)
 
             # error if this argument is not allowed with other previously
-            # seen arguments, assuming that actions that use the default
-            # value don't really count as "present"
-            if argument_values is not action.default:
+            # seen arguments
+            if action.option_strings or argument_strings:
                 seen_non_default_actions.add(action)
                 for conflict_action in action_conflicts.get(action, []):
                     if conflict_action in seen_non_default_actions:
@@ -2069,11 +2067,15 @@ def consume_positionals(start_index):
             # and add the Positional and its args to the list
             for action, arg_count in zip(positionals, arg_counts):
                 args = arg_strings[start_index: start_index + arg_count]
-                # Strip out the first '--' if it is not in PARSER or REMAINDER arg.
-                if (action.nargs not in [PARSER, REMAINDER]
-                    and arg_strings_pattern.find('-', start_index,
+                # Strip out the first '--' if it is not in REMAINDER arg.
+                if action.nargs == PARSER:
+                    if arg_strings_pattern[start_index] == '-':
+                        assert args[0] == '--'
+                        args.remove('--')
+                elif action.nargs != REMAINDER:
+                    if (arg_strings_pattern.find('-', start_index,
                                                  start_index + arg_count) >= 0):
-                    args.remove('--')
+                        args.remove('--')
                 start_index += arg_count
                 if args and action.deprecated and action.dest not in warned:
                     self._warning(_("argument '%(argument_name)s' is deprecated") %

Lib/argparse.py

@@ -234,9 +234,13 @@ def make_script(script_dir, script_basename, source, omit_suffix=False):
     if not omit_suffix:
         script_filename += os.extsep + 'py'
     script_name = os.path.join(script_dir, script_filename)
-    # The script should be encoded to UTF-8, the default string encoding
-    with open(script_name, 'w', encoding='utf-8') as script_file:
-        script_file.write(source)
+    if isinstance(source, str):
+        # The script should be encoded to UTF-8, the default string encoding
+        with open(script_name, 'w', encoding='utf-8') as script_file:
+            script_file.write(source)
+    else:
+        with open(script_name, 'wb') as script_file:
+            script_file.write(source)
     importlib.invalidate_caches()
     return script_name