Merge branch 'main' into multi_inputs

Author: s-ball

Doc/c-api/exceptions.rst

@@ -921,7 +921,11 @@ because the :ref:`call protocol <call>` takes care of recursion handling.
 
    Marks a point where a recursive C-level call is about to be performed.
 
-   The function then checks if the stack limit is reached.  If this is the
+   If :c:macro:`!USE_STACKCHECK` is defined, this function checks if the OS
+   stack overflowed using :c:func:`PyOS_CheckStack`.  If this is the case, it
+   sets a :exc:`MemoryError` and returns a nonzero value.
+
+   The function then checks if the recursion limit is reached.  If this is the
    case, a :exc:`RecursionError` is set and a nonzero value is returned.
    Otherwise, zero is returned.
 

Doc/library/idle.rst

@@ -660,29 +660,61 @@ functions to be used from IDLE's Python shell.
 Command line usage
 ^^^^^^^^^^^^^^^^^^
 
-.. code-block:: none
+.. program:: idle
 
-   idle.py [-c command] [-d] [-e] [-h] [-i] [-r file] [-s] [-t title] [-] [arg] ...
+IDLE can be invoked from the command line with various options. The general syntax is:
 
-   -c command  run command in the shell window
-   -d          enable debugger and open shell window
-   -e          open editor window
-   -h          print help message with legal combinations and exit
-   -i          open shell window
-   -r file     run file in shell window
-   -s          run $IDLESTARTUP or $PYTHONSTARTUP first, in shell window
-   -t title    set title of shell window
-   -           run stdin in shell (- must be last option before args)
+.. code-block:: bash
 
-If there are arguments:
+   python -m idlelib [options] [file ...]
 
-* If ``-``, ``-c``, or ``r`` is used, all arguments are placed in
-  ``sys.argv[1:...]`` and ``sys.argv[0]`` is set to ``''``, ``'-c'``,
-  or ``'-r'``.  No editor window is opened, even if that is the default
-  set in the Options dialog.
+The following options are available:
+
+.. option:: -c <command>
+
+   Run the specified Python command in the shell window.
+   For example, pass ``-c "print('Hello, World!')"``.
+   On Windows, the outer quotes must be double quotes as shown.
+
+.. option:: -d
+
+   Enable the debugger and open the shell window.
+
+.. option:: -e
+
+   Open an editor window.
+
+.. option:: -h
+
+   Print a help message with legal combinations of options and exit.
+
+.. option:: -i
+
+   Open a shell window.
+
+.. option:: -r <file>
+
+   Run the specified file in the shell window.
+
+.. option:: -s
+
+   Run the startup file (as defined by the environment variables :envvar:`IDLESTARTUP` or :envvar:`PYTHONSTARTUP`) before opening the shell window.
+
+.. option:: -t <title>
+
+   Set the title of the shell window.
+
+.. option:: -
+
+   Read and execute standard input in the shell window. This option must be the last one before any arguments.
+
+If arguments are provided:
+
+- If ``-``, ``-c``, or ``-r`` is used, all arguments are placed in ``sys.argv[1:]``,
+  and ``sys.argv[0]`` is set to ``''``, ``'-c'``, or ``'-r'`` respectively.
+  No editor window is opened, even if that is the default set in the *Options* dialog.
+- Otherwise, arguments are treated as files to be opened for editing, and ``sys.argv`` reflects the arguments passed to IDLE itself.
 
-* Otherwise, arguments are files opened for editing and
-  ``sys.argv`` reflects the arguments passed to IDLE itself.
 
 Startup failure
 ^^^^^^^^^^^^^^^

Doc/library/wsgiref.rst

@@ -119,7 +119,8 @@ in type annotations.
    applications to set up dummy environments.  It should NOT be used by actual WSGI
    servers or applications, since the data is fake!
 
-   Example usage::
+   Example usage (see also :func:`~wsgiref.simple_server.demo_app`
+   for another example)::
 
       from wsgiref.util import setup_testing_defaults
       from wsgiref.simple_server import make_server
@@ -312,6 +313,8 @@ request.  (E.g., using the :func:`shift_path_info` function from
    as :mod:`wsgiref.simple_server`) is able to run a simple WSGI application
    correctly.
 
+   The *start_response* callable should follow the :class:`.StartResponse` protocol.
+
 
 .. class:: WSGIServer(server_address, RequestHandlerClass)
 
@@ -679,7 +682,9 @@ input, output, and error streams.
 
       This method can access the current error using ``sys.exception()``,
       and should pass that information to *start_response* when calling it (as
-      described in the "Error Handling" section of :pep:`3333`).
+      described in the "Error Handling" section of :pep:`3333`). In particular,
+      the *start_response* callable should follow the :class:`.StartResponse`
+      protocol.
 
       The default implementation just uses the :attr:`error_status`,
       :attr:`error_headers`, and :attr:`error_body` attributes to generate an output
@@ -781,7 +786,7 @@ in :pep:`3333`.
 .. versionadded:: 3.11
 
 
-.. class:: StartResponse()
+.. class:: StartResponse
 
    A :class:`typing.Protocol` describing :pep:`start_response()
    <3333#the-start-response-callable>`
@@ -816,7 +821,8 @@ in :pep:`3333`.
 Examples
 --------
 
-This is a working "Hello World" WSGI application::
+This is a working "Hello World" WSGI application, where the *start_response*
+callable should follow the :class:`.StartResponse` protocol::
 
    """
    Every WSGI application must have an application object - a callable

Include/cpython/object.h

@@ -487,19 +487,18 @@ PyAPI_FUNC(void) _PyTrash_thread_destroy_chain(PyThreadState *tstate);
  * we have headroom above the trigger limit */
 #define Py_TRASHCAN_HEADROOM 50
 
-/* Helper function for Py_TRASHCAN_BEGIN */
-PyAPI_FUNC(int) _Py_ReachedRecursionLimitWithMargin(PyThreadState *tstate, int margin_count);
-
 #define Py_TRASHCAN_BEGIN(op, dealloc) \
 do { \
     PyThreadState *tstate = PyThreadState_Get(); \
-    if (_Py_ReachedRecursionLimitWithMargin(tstate, 1) && Py_TYPE(op)->tp_dealloc == (destructor)dealloc) { \
+    if (tstate->c_recursion_remaining <= Py_TRASHCAN_HEADROOM && Py_TYPE(op)->tp_dealloc == (destructor)dealloc) { \
         _PyTrash_thread_deposit_object(tstate, (PyObject *)op); \
         break; \
-    }
+    } \
+    tstate->c_recursion_remaining--;
     /* The body of the deallocator is here. */
 #define Py_TRASHCAN_END \
-    if (tstate->delete_later && !_Py_ReachedRecursionLimitWithMargin(tstate, 2)) { \
+    tstate->c_recursion_remaining++; \
+    if (tstate->delete_later && tstate->c_recursion_remaining > (Py_TRASHCAN_HEADROOM*2)) { \
         _PyTrash_thread_destroy_chain(tstate); \
     } \
 } while (0);

Include/cpython/pystate.h

@@ -112,7 +112,7 @@ struct _ts {
     int py_recursion_remaining;
     int py_recursion_limit;
 
-    int c_recursion_remaining; /* Retained for backwards compatibility. Do not use */
+    int c_recursion_remaining;
     int recursion_headroom; /* Allow 50 more calls to handle any errors. */
 
     /* 'tracing' keeps track of the execution depth when tracing/profiling.
@@ -202,7 +202,36 @@ struct _ts {
     PyObject *threading_local_sentinel;
 };
 
-# define Py_C_RECURSION_LIMIT 5000
+#ifdef Py_DEBUG
+   // A debug build is likely built with low optimization level which implies
+   // higher stack memory usage than a release build: use a lower limit.
+#  define Py_C_RECURSION_LIMIT 500
+#elif defined(__s390x__)
+#  define Py_C_RECURSION_LIMIT 800
+#elif defined(_WIN32) && defined(_M_ARM64)
+#  define Py_C_RECURSION_LIMIT 1000
+#elif defined(_WIN32)
+#  define Py_C_RECURSION_LIMIT 3000
+#elif defined(__ANDROID__)
+   // On an ARM64 emulator, API level 34 was OK with 10000, but API level 21
+   // crashed in test_compiler_recursion_limit.
+#  define Py_C_RECURSION_LIMIT 3000
+#elif defined(_Py_ADDRESS_SANITIZER)
+#  define Py_C_RECURSION_LIMIT 4000
+#elif defined(__sparc__)
+   // test_descr crashed on sparc64 with >7000 but let's keep a margin of error.
+#  define Py_C_RECURSION_LIMIT 4000
+#elif defined(__wasi__)
+   // Based on wasmtime 16.
+#  define Py_C_RECURSION_LIMIT 5000
+#elif defined(__hppa__) || defined(__powerpc64__)
+   // test_descr crashed with >8000 but let's keep a margin of error.
+#  define Py_C_RECURSION_LIMIT 5000
+#else
+   // This value is duplicated in Lib/test/support/__init__.py
+#  define Py_C_RECURSION_LIMIT 10000
+#endif
+
 
 /* other API */
 
@@ -217,6 +246,7 @@ _PyThreadState_UncheckedGet(void)
     return PyThreadState_GetUnchecked();
 }
 
+
 // Disable tracing and profiling.
 PyAPI_FUNC(void) PyThreadState_EnterTracing(PyThreadState *tstate);
 

Include/internal/pycore_ceval.h

@@ -193,12 +193,18 @@ extern void _PyEval_DeactivateOpCache(void);
 
 /* --- _Py_EnterRecursiveCall() ----------------------------------------- */
 
+#ifdef USE_STACKCHECK
+/* With USE_STACKCHECK macro defined, trigger stack checks in
+   _Py_CheckRecursiveCall() on every 64th call to _Py_EnterRecursiveCall. */
 static inline int _Py_MakeRecCheck(PyThreadState *tstate)  {
-    char here;
-    uintptr_t here_addr = (uintptr_t)&here;
-    _PyThreadStateImpl *_tstate = (_PyThreadStateImpl *)tstate;
-    return here_addr < _tstate->c_stack_soft_limit;
+    return (tstate->c_recursion_remaining-- < 0
+            || (tstate->c_recursion_remaining & 63) == 0);
 }
+#else
+static inline int _Py_MakeRecCheck(PyThreadState *tstate) {
+    return tstate->c_recursion_remaining-- < 0;
+}
+#endif
 
 // Export for '_json' shared extension, used via _Py_EnterRecursiveCall()
 // static inline function.
@@ -214,31 +220,23 @@ static inline int _Py_EnterRecursiveCallTstate(PyThreadState *tstate,
     return (_Py_MakeRecCheck(tstate) && _Py_CheckRecursiveCall(tstate, where));
 }
 
+static inline void _Py_EnterRecursiveCallTstateUnchecked(PyThreadState *tstate)  {
+    assert(tstate->c_recursion_remaining > 0);
+    tstate->c_recursion_remaining--;
+}
+
 static inline int _Py_EnterRecursiveCall(const char *where) {
     PyThreadState *tstate = _PyThreadState_GET();
     return _Py_EnterRecursiveCallTstate(tstate, where);
 }
 
-static inline void _Py_LeaveRecursiveCallTstate(PyThreadState *tstate) {
-    (void)tstate;
-}
-
-PyAPI_FUNC(void) _Py_InitializeRecursionLimits(PyThreadState *tstate);
-
-static inline int _Py_ReachedRecursionLimit(PyThreadState *tstate)  {
-    char here;
-    uintptr_t here_addr = (uintptr_t)&here;
-    _PyThreadStateImpl *_tstate = (_PyThreadStateImpl *)tstate;
-    if (here_addr > _tstate->c_stack_soft_limit) {
-        return 0;
-    }
-    if (_tstate->c_stack_hard_limit == 0) {
-        _Py_InitializeRecursionLimits(tstate);
-    }
-    return here_addr <= _tstate->c_stack_soft_limit;
+static inline void _Py_LeaveRecursiveCallTstate(PyThreadState *tstate)  {
+    tstate->c_recursion_remaining++;
 }
 
 static inline void _Py_LeaveRecursiveCall(void)  {
+    PyThreadState *tstate = _PyThreadState_GET();
+    _Py_LeaveRecursiveCallTstate(tstate);
 }
 
 extern struct _PyInterpreterFrame* _PyEval_GetFrame(void);
@@ -329,6 +327,7 @@ void _Py_unset_eval_breaker_bit_all(PyInterpreterState *interp, uintptr_t bit);
 
 PyAPI_FUNC(PyObject *) _PyFloat_FromDouble_ConsumeInputs(_PyStackRef left, _PyStackRef right, double value);
 
+
 #ifdef __cplusplus
 }
 #endif

Include/internal/pycore_symtable.h

@@ -82,6 +82,8 @@ struct symtable {
     PyObject *st_private;           /* name of current class or NULL */
     _PyFutureFeatures *st_future;   /* module's future features that affect
                                        the symbol table */
+    int recursion_depth;            /* current recursion depth */
+    int recursion_limit;            /* recursion limit */
 };
 
 typedef struct _symtable_entry {

Include/internal/pycore_tstate.h

@@ -21,11 +21,6 @@ typedef struct _PyThreadStateImpl {
     // semi-public fields are in PyThreadState.
     PyThreadState base;
 
-    // These are addresses, but we need to convert to ints to avoid UB.
-    uintptr_t c_stack_top;
-    uintptr_t c_stack_soft_limit;
-    uintptr_t c_stack_hard_limit;
-
     PyObject *asyncio_running_loop; // Strong reference
     PyObject *asyncio_running_task; // Strong reference
 

Include/pythonrun.h

@@ -21,18 +21,14 @@ PyAPI_FUNC(void) PyErr_DisplayException(PyObject *);
 /* Stuff with no proper home (yet) */
 PyAPI_DATA(int) (*PyOS_InputHook)(void);
 
-/* Stack size, in "pointers". This must be large enough, so
- * no two calls to check recursion depth are more than this far
- * apart. In practice, that means it must be larger than the C
- * stack consumption of PyEval_EvalDefault */
-#if defined(Py_DEBUG) && defined(WIN32)
-#  define PYOS_STACK_MARGIN 3072
-#else
-#  define PYOS_STACK_MARGIN 2048
-#endif
-#define PYOS_STACK_MARGIN_BYTES (PYOS_STACK_MARGIN * sizeof(void *))
-
-#if defined(WIN32)
+/* Stack size, in "pointers" (so we get extra safety margins
+   on 64-bit platforms).  On a 32-bit platform, this translates
+   to an 8k margin. */
+#define PYOS_STACK_MARGIN 2048
+
+#if defined(WIN32) && !defined(MS_WIN64) && !defined(_M_ARM) && defined(_MSC_VER) && _MSC_VER >= 1300
+/* Enable stack checking under Microsoft C */
+// When changing the platforms, ensure PyOS_CheckStack() docs are still correct
 #define USE_STACKCHECK
 #endif