Merge branch 'master' into fix-nan-packing/130317

Author: skirpichev

Doc/c-api/exceptions.rst

@@ -921,11 +921,7 @@ because the :ref:`call protocol <call>` takes care of recursion handling.
 
    Marks a point where a recursive C-level call is about to be performed.
 
-   If :c:macro:`!USE_STACKCHECK` is defined, this function checks if the OS
-   stack overflowed using :c:func:`PyOS_CheckStack`.  If this is the case, it
-   sets a :exc:`MemoryError` and returns a nonzero value.
-
-   The function then checks if the recursion limit is reached.  If this is the
+   The function then checks if the stack limit is reached.  If this is the
    case, a :exc:`RecursionError` is set and a nonzero value is returned.
    Otherwise, zero is returned.
 

Doc/glossary.rst

@@ -719,7 +719,7 @@ Glossary
       iterables include all sequence types (such as :class:`list`, :class:`str`,
       and :class:`tuple`) and some non-sequence types like :class:`dict`,
       :term:`file objects <file object>`, and objects of any classes you define
-      with an :meth:`~iterator.__iter__` method or with a
+      with an :meth:`~object.__iter__` method or with a
       :meth:`~object.__getitem__` method
       that implements :term:`sequence` semantics.
 

Doc/library/abc.rst

@@ -141,18 +141,18 @@ a helper class :class:`ABC` to alternatively define ABCs through inheritance:
       MyIterable.register(Foo)
 
    The ABC ``MyIterable`` defines the standard iterable method,
-   :meth:`~iterator.__iter__`, as an abstract method.  The implementation given
+   :meth:`~object.__iter__`, as an abstract method.  The implementation given
    here can still be called from subclasses.  The :meth:`!get_iterator` method
    is also part of the ``MyIterable`` abstract base class, but it does not have
    to be overridden in non-abstract derived classes.
 
    The :meth:`__subclasshook__` class method defined here says that any class
-   that has an :meth:`~iterator.__iter__` method in its
+   that has an :meth:`~object.__iter__` method in its
    :attr:`~object.__dict__` (or in that of one of its base classes, accessed
    via the :attr:`~type.__mro__` list) is considered a ``MyIterable`` too.
 
    Finally, the last line makes ``Foo`` a virtual subclass of ``MyIterable``,
-   even though it does not define an :meth:`~iterator.__iter__` method (it uses
+   even though it does not define an :meth:`~object.__iter__` method (it uses
    the old-style iterable protocol, defined in terms of :meth:`~object.__len__` and
    :meth:`~object.__getitem__`).  Note that this will not make ``get_iterator``
    available as a method of ``Foo``, so it is provided separately.

Doc/library/stdtypes.rst

@@ -949,9 +949,9 @@ Generator Types
 ---------------
 
 Python's :term:`generator`\s provide a convenient way to implement the iterator
-protocol.  If a container object's :meth:`~iterator.__iter__` method is implemented as a
+protocol.  If a container object's :meth:`~object.__iter__` method is implemented as a
 generator, it will automatically return an iterator object (technically, a
-generator object) supplying the :meth:`!__iter__` and :meth:`~generator.__next__`
+generator object) supplying the :meth:`~iterator.__iter__` and :meth:`~generator.__next__`
 methods.
 More information about generators can be found in :ref:`the documentation for
 the yield expression <yieldexpr>`.

Doc/library/string.rst

@@ -319,14 +319,19 @@ non-empty format specification typically modifies the result.
 The general form of a *standard format specifier* is:
 
 .. productionlist:: format-spec
-   format_spec: [[`fill`]`align`][`sign`]["z"]["#"]["0"][`width`][`grouping_option`]["." `precision`][`type`]
+   format_spec: [`options`][`width_and_precision`][`type`]
+   options: [[`fill`]`align`][`sign`]["z"]["#"]["0"]
    fill: <any character>
    align: "<" | ">" | "=" | "^"
    sign: "+" | "-" | " "
+   width_and_precision: [`width_with_grouping`][`precision_with_grouping`]
+   width_with_grouping: [`width`][`grouping_option`]
+   precision_with_grouping: "." [`precision`]`grouping_option`
    width: `~python-grammar:digit`+
    grouping_option: "_" | ","
    precision: `~python-grammar:digit`+
-   type: "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
+   type: "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g"
+       : | "G" | "n" | "o" | "s" | "x" | "X" | "%"
 
 If a valid *align* value is specified, it can be preceded by a *fill*
 character that can be any character and defaults to a space if omitted.
@@ -458,6 +463,13 @@ indicates the maximum field size - in other words, how many characters will be
 used from the field content.  The *precision* is not allowed for integer
 presentation types.
 
+The ``'_'`` or ``','`` option after *precision* means the use of an underscore
+or a comma for a thousands separator of the fractional part for floating-point
+presentation types.
+
+.. versionchanged:: 3.14
+   Support thousands separators for the fractional part.
+
 Finally, the *type* determines how the data should be presented.
 
 The available string presentation types are:
@@ -704,10 +716,18 @@ Replacing ``%x`` and ``%o`` and converting the value to different bases::
    >>> "int: {0:d};  hex: {0:#x};  oct: {0:#o};  bin: {0:#b}".format(42)
    'int: 42;  hex: 0x2a;  oct: 0o52;  bin: 0b101010'
 
-Using the comma as a thousands separator::
+Using the comma or the underscore as a thousands separator::
 
    >>> '{:,}'.format(1234567890)
    '1,234,567,890'
+   >>> '{:_}'.format(1234567890)
+   '1_234_567_890'
+   >>> '{:_}'.format(123456789.123456789)
+   '123_456_789.12345679'
+   >>> '{:._}'.format(123456789.123456789)
+   '123456789.123_456_79'
+   >>> '{:_._}'.format(123456789.123456789)
+   '123_456_789.123_456_79'
 
 Expressing a percentage::
 

Doc/library/uuid.rst

@@ -184,8 +184,6 @@ The :mod:`uuid` module defines the following functions:
       administered MAC addresses, since the former are guaranteed to be
       globally unique, while the latter are not.
 
-.. index:: single: getnode
-
 
 .. function:: uuid1(node=None, clock_seq=None)
 
@@ -194,33 +192,25 @@ The :mod:`uuid` module defines the following functions:
    *clock_seq* is given, it is used as the sequence number; otherwise a random
    14-bit sequence number is chosen.
 
-.. index:: single: uuid1
-
 
 .. function:: uuid3(namespace, name)
 
    Generate a UUID based on the MD5 hash of a namespace identifier (which is a
    UUID) and a name (which is a :class:`bytes` object or a string
    that will be encoded using UTF-8).
 
-.. index:: single: uuid3
-
 
 .. function:: uuid4()
 
    Generate a random UUID.
 
-.. index:: single: uuid4
-
 
 .. function:: uuid5(namespace, name)
 
    Generate a UUID based on the SHA-1 hash of a namespace identifier (which is a
    UUID) and a name (which is a :class:`bytes` object or a string
    that will be encoded using UTF-8).
 
-.. index:: single: uuid5
-
 
 .. function:: uuid8(a=None, b=None, c=None)
 
@@ -235,8 +225,6 @@ The :mod:`uuid` module defines the following functions:
 
    .. versionadded:: 3.14
 
-.. index:: single: uuid8
-
 
 The :mod:`uuid` module defines the following namespace identifiers for use with
 :func:`uuid3` or :func:`uuid5`.

Doc/whatsnew/3.14.rst

@@ -336,6 +336,11 @@ Other language changes
   making it a :term:`generic type`.
   (Contributed by Brian Schubert in :gh:`126012`.)
 
+* Support underscore and comma as thousands separators in the fractional part
+  for floating-point presentation types of the new-style string formatting
+  (with :func:`format` or :ref:`f-strings`).
+  (Contrubuted by Sergey B Kirpichev in :gh:`87790`.)
+
 * ``\B`` in :mod:`regular expression <re>` now matches empty input string.
   Now it is always the opposite of ``\b``.
   (Contributed by Serhiy Storchaka in :gh:`124130`.)

Include/cpython/object.h

@@ -5,6 +5,7 @@
 PyAPI_FUNC(void) _Py_NewReference(PyObject *op);
 PyAPI_FUNC(void) _Py_NewReferenceNoTotal(PyObject *op);
 PyAPI_FUNC(void) _Py_ResurrectReference(PyObject *op);
+PyAPI_FUNC(void) _Py_ForgetReference(PyObject *op);
 
 #ifdef Py_REF_DEBUG
 /* These are useful as debugging aids when chasing down refleaks. */
@@ -487,18 +488,19 @@ PyAPI_FUNC(void) _PyTrash_thread_destroy_chain(PyThreadState *tstate);
  * we have headroom above the trigger limit */
 #define Py_TRASHCAN_HEADROOM 50
 
+/* Helper function for Py_TRASHCAN_BEGIN */
+PyAPI_FUNC(int) _Py_ReachedRecursionLimitWithMargin(PyThreadState *tstate, int margin_count);
+
 #define Py_TRASHCAN_BEGIN(op, dealloc) \
 do { \
     PyThreadState *tstate = PyThreadState_Get(); \
-    if (tstate->c_recursion_remaining <= Py_TRASHCAN_HEADROOM && Py_TYPE(op)->tp_dealloc == (destructor)dealloc) { \
+    if (_Py_ReachedRecursionLimitWithMargin(tstate, 1) && Py_TYPE(op)->tp_dealloc == (destructor)dealloc) { \
         _PyTrash_thread_deposit_object(tstate, (PyObject *)op); \
         break; \
-    } \
-    tstate->c_recursion_remaining--;
+    }
     /* The body of the deallocator is here. */
 #define Py_TRASHCAN_END \
-    tstate->c_recursion_remaining++; \
-    if (tstate->delete_later && tstate->c_recursion_remaining > (Py_TRASHCAN_HEADROOM*2)) { \
+    if (tstate->delete_later && !_Py_ReachedRecursionLimitWithMargin(tstate, 2)) { \
         _PyTrash_thread_destroy_chain(tstate); \
     } \
 } while (0);

Include/cpython/pystate.h

@@ -112,7 +112,7 @@ struct _ts {
     int py_recursion_remaining;
     int py_recursion_limit;
 
-    int c_recursion_remaining;
+    int c_recursion_remaining; /* Retained for backwards compatibility. Do not use */
     int recursion_headroom; /* Allow 50 more calls to handle any errors. */
 
     /* 'tracing' keeps track of the execution depth when tracing/profiling.
@@ -202,36 +202,7 @@ struct _ts {
     PyObject *threading_local_sentinel;
 };
 
-#ifdef Py_DEBUG
-   // A debug build is likely built with low optimization level which implies
-   // higher stack memory usage than a release build: use a lower limit.
-#  define Py_C_RECURSION_LIMIT 500
-#elif defined(__s390x__)
-#  define Py_C_RECURSION_LIMIT 800
-#elif defined(_WIN32) && defined(_M_ARM64)
-#  define Py_C_RECURSION_LIMIT 1000
-#elif defined(_WIN32)
-#  define Py_C_RECURSION_LIMIT 3000
-#elif defined(__ANDROID__)
-   // On an ARM64 emulator, API level 34 was OK with 10000, but API level 21
-   // crashed in test_compiler_recursion_limit.
-#  define Py_C_RECURSION_LIMIT 3000
-#elif defined(_Py_ADDRESS_SANITIZER)
-#  define Py_C_RECURSION_LIMIT 4000
-#elif defined(__sparc__)
-   // test_descr crashed on sparc64 with >7000 but let's keep a margin of error.
-#  define Py_C_RECURSION_LIMIT 4000
-#elif defined(__wasi__)
-   // Based on wasmtime 16.
-#  define Py_C_RECURSION_LIMIT 5000
-#elif defined(__hppa__) || defined(__powerpc64__)
-   // test_descr crashed with >8000 but let's keep a margin of error.
-#  define Py_C_RECURSION_LIMIT 5000
-#else
-   // This value is duplicated in Lib/test/support/__init__.py
-#  define Py_C_RECURSION_LIMIT 10000
-#endif
-
+# define Py_C_RECURSION_LIMIT 5000
 
 /* other API */
 
@@ -246,7 +217,6 @@ _PyThreadState_UncheckedGet(void)
     return PyThreadState_GetUnchecked();
 }
 
-
 // Disable tracing and profiling.
 PyAPI_FUNC(void) PyThreadState_EnterTracing(PyThreadState *tstate);
 

Include/internal/pycore_ceval.h

@@ -193,18 +193,28 @@ extern void _PyEval_DeactivateOpCache(void);
 
 /* --- _Py_EnterRecursiveCall() ----------------------------------------- */
 
-#ifdef USE_STACKCHECK
-/* With USE_STACKCHECK macro defined, trigger stack checks in
-   _Py_CheckRecursiveCall() on every 64th call to _Py_EnterRecursiveCall. */
-static inline int _Py_MakeRecCheck(PyThreadState *tstate)  {
-    return (tstate->c_recursion_remaining-- < 0
-            || (tstate->c_recursion_remaining & 63) == 0);
+#if !_Py__has_builtin(__builtin_frame_address)
+static uintptr_t return_pointer_as_int(char* p) {
+    return (uintptr_t)p;
 }
+#endif
+
+static inline uintptr_t
+_Py_get_machine_stack_pointer(void) {
+#if _Py__has_builtin(__builtin_frame_address)
+    return (uintptr_t)__builtin_frame_address(0);
 #else
-static inline int _Py_MakeRecCheck(PyThreadState *tstate) {
-    return tstate->c_recursion_remaining-- < 0;
-}
+    char here;
+    /* Avoid compiler warning about returning stack address */
+    return return_pointer_as_int(&here);
 #endif
+}
+
+static inline int _Py_MakeRecCheck(PyThreadState *tstate)  {
+    uintptr_t here_addr = _Py_get_machine_stack_pointer();
+    _PyThreadStateImpl *_tstate = (_PyThreadStateImpl *)tstate;
+    return here_addr < _tstate->c_stack_soft_limit;
+}
 
 // Export for '_json' shared extension, used via _Py_EnterRecursiveCall()
 // static inline function.
@@ -220,23 +230,30 @@ static inline int _Py_EnterRecursiveCallTstate(PyThreadState *tstate,
     return (_Py_MakeRecCheck(tstate) && _Py_CheckRecursiveCall(tstate, where));
 }
 
-static inline void _Py_EnterRecursiveCallTstateUnchecked(PyThreadState *tstate)  {
-    assert(tstate->c_recursion_remaining > 0);
-    tstate->c_recursion_remaining--;
-}
-
 static inline int _Py_EnterRecursiveCall(const char *where) {
     PyThreadState *tstate = _PyThreadState_GET();
     return _Py_EnterRecursiveCallTstate(tstate, where);
 }
 
-static inline void _Py_LeaveRecursiveCallTstate(PyThreadState *tstate)  {
-    tstate->c_recursion_remaining++;
+static inline void _Py_LeaveRecursiveCallTstate(PyThreadState *tstate) {
+    (void)tstate;
+}
+
+PyAPI_FUNC(void) _Py_InitializeRecursionLimits(PyThreadState *tstate);
+
+static inline int _Py_ReachedRecursionLimit(PyThreadState *tstate)  {
+    uintptr_t here_addr = _Py_get_machine_stack_pointer();
+    _PyThreadStateImpl *_tstate = (_PyThreadStateImpl *)tstate;
+    if (here_addr > _tstate->c_stack_soft_limit) {
+        return 0;
+    }
+    if (_tstate->c_stack_hard_limit == 0) {
+        _Py_InitializeRecursionLimits(tstate);
+    }
+    return here_addr <= _tstate->c_stack_soft_limit;
 }
 
 static inline void _Py_LeaveRecursiveCall(void)  {
-    PyThreadState *tstate = _PyThreadState_GET();
-    _Py_LeaveRecursiveCallTstate(tstate);
 }
 
 extern struct _PyInterpreterFrame* _PyEval_GetFrame(void);
@@ -327,7 +344,6 @@ void _Py_unset_eval_breaker_bit_all(PyInterpreterState *interp, uintptr_t bit);
 
 PyAPI_FUNC(PyObject *) _PyFloat_FromDouble_ConsumeInputs(_PyStackRef left, _PyStackRef right, double value);
 
-
 #ifdef __cplusplus
 }
 #endif