Merge branch 'main' into fix-for-running-tests-issue-131290

Author: MaximGit1

Doc/glossary.rst

@@ -800,6 +800,10 @@ Glossary
       thread removes *key* from *mapping* after the test, but before the lookup.
       This issue can be solved with locks or by using the EAFP approach.
 
+   lexical analyzer
+
+      Formal name for the *tokenizer*; see :term:`token`.
+
    list
       A built-in Python :term:`sequence`.  Despite its name it is more akin
       to an array in other languages than to a linked list since access to
@@ -1291,6 +1295,17 @@ Glossary
       See also :term:`binary file` for a file object able to read and write
       :term:`bytes-like objects <bytes-like object>`.
 
+   token
+
+      A small unit of source code, generated by the
+      :ref:`lexical analyzer <lexical>` (also called the *tokenizer*).
+      Names, numbers, strings, operators,
+      newlines and similar are represented by tokens.
+
+      The :mod:`tokenize` module exposes Python's lexical analyzer.
+      The :mod:`token` module contains information on the various types
+      of tokens.
+
    triple-quoted string
       A string which is bound by three instances of either a quotation mark
       (") or an apostrophe (').  While they don't provide any functionality

Doc/library/dis.rst

@@ -535,7 +535,7 @@ details of bytecode instructions as :class:`Instruction` instances:
       :class:`dis.Positions` object holding the
       start and end locations that are covered by this instruction.
 
-   .. data::cache_info
+   .. data:: cache_info
 
       Information about the cache entries of this instruction, as
       triplets of the form ``(name, size, data)``, where the ``name``

Doc/library/pickletools.rst

@@ -19,7 +19,7 @@ ordinary users of the :mod:`pickle` module probably won't find the
 
 .. _pickletools-cli:
 
-Command line usage
+Command-line usage
 ------------------
 
 .. versionadded:: 3.2
@@ -48,7 +48,7 @@ For example, with a tuple ``(1, 2)`` pickled in file ``x.pickle``:
         9: .    STOP
     highest protocol among opcodes = 2
 
-Command line options
+Command-line options
 ^^^^^^^^^^^^^^^^^^^^
 
 .. program:: pickletools
@@ -72,12 +72,16 @@ Command line options
 
 .. option:: -p, --preamble=<preamble>
 
-   When more than one pickle file are specified, print given preamble
+   When more than one pickle file is specified, print given preamble
    before each disassembly.
 
+.. option:: pickle_file
 
+   A pickle file to read, or ``-`` to indicate reading from standard input.
 
-Programmatic Interface
+
+
+Programmatic interface
 ----------------------
 
 

Doc/reference/lexical_analysis.rst

@@ -8,8 +8,9 @@ Lexical analysis
 .. index:: lexical analysis, parser, token
 
 A Python program is read by a *parser*.  Input to the parser is a stream of
-*tokens*, generated by the *lexical analyzer*.  This chapter describes how the
-lexical analyzer breaks a file into tokens.
+:term:`tokens <token>`, generated by the *lexical analyzer* (also known as
+the *tokenizer*).
+This chapter describes how the lexical analyzer breaks a file into tokens.
 
 Python reads program text as Unicode code points; the encoding of a source file
 can be given by an encoding declaration and defaults to UTF-8, see :pep:`3120`

Doc/tutorial/errors.rst

@@ -24,11 +24,12 @@ complaint you get while you are still learning Python::
    SyntaxError: invalid syntax
 
 The parser repeats the offending line and displays little arrows pointing
-at the token in the line where the error was detected.  The error may be
-caused by the absence of a token *before* the indicated token.  In the
-example, the error is detected at the function :func:`print`, since a colon
-(``':'``) is missing before it.  File name and line number are printed so you
-know where to look in case the input came from a script.
+at the place where the error was detected.  Note that this is not always the
+place that needs to be fixed.  In the example, the error is detected at the
+function :func:`print`, since a colon (``':'``) is missing just before it.
+
+The file name (``<stdin>`` in our example) and line number are printed so you
+know where to look in case the input came from a file.
 
 
 .. _tut-exceptions:

Doc/tutorial/interactive.rst

@@ -37,10 +37,10 @@ Alternatives to the Interactive Interpreter
 
 This facility is an enormous step forward compared to earlier versions of the
 interpreter; however, some wishes are left: It would be nice if the proper
-indentation were suggested on continuation lines (the parser knows if an indent
-token is required next).  The completion mechanism might use the interpreter's
-symbol table.  A command to check (or even suggest) matching parentheses,
-quotes, etc., would also be useful.
+indentation were suggested on continuation lines (the parser knows if an
+:data:`~token.INDENT` token is required next).  The completion mechanism might
+use the interpreter's symbol table.  A command to check (or even suggest)
+matching parentheses, quotes, etc., would also be useful.
 
 One alternative enhanced interactive interpreter that has been around for quite
 some time is IPython_, which features tab completion, object exploration and

Include/cpython/code.h

@@ -8,70 +8,19 @@
 extern "C" {
 #endif
 
-/* Total tool ids available */
-#define  _PY_MONITORING_TOOL_IDS 8
-/* Count of all local monitoring events */
-#define  _PY_MONITORING_LOCAL_EVENTS 11
-/* Count of all "real" monitoring events (not derived from other events) */
-#define _PY_MONITORING_UNGROUPED_EVENTS 16
-/* Count of all  monitoring events */
-#define _PY_MONITORING_EVENTS 19
-
-/* Tables of which tools are active for each monitored event. */
-typedef struct _Py_LocalMonitors {
-    uint8_t tools[_PY_MONITORING_LOCAL_EVENTS];
-} _Py_LocalMonitors;
-
-typedef struct _Py_GlobalMonitors {
-    uint8_t tools[_PY_MONITORING_UNGROUPED_EVENTS];
-} _Py_GlobalMonitors;
-
-
 typedef struct {
     PyObject *_co_code;
     PyObject *_co_varnames;
     PyObject *_co_cellvars;
     PyObject *_co_freevars;
 } _PyCoCached;
 
-/* Ancillary data structure used for instrumentation.
-   Line instrumentation creates this with sufficient
-   space for one entry per code unit. The total size
-   of the data will be `bytes_per_entry * Py_SIZE(code)` */
-typedef struct {
-    uint8_t bytes_per_entry;
-    uint8_t data[1];
-} _PyCoLineInstrumentationData;
-
-
 typedef struct {
     int size;
     int capacity;
     struct _PyExecutorObject *executors[1];
 } _PyExecutorArray;
 
-/* Main data structure used for instrumentation.
- * This is allocated when needed for instrumentation
- */
-typedef struct {
-    /* Monitoring specific to this code object */
-    _Py_LocalMonitors local_monitors;
-    /* Monitoring that is active on this code object */
-    _Py_LocalMonitors active_monitors;
-    /* The tools that are to be notified for events for the matching code unit */
-    uint8_t *tools;
-    /* The version of tools when they instrument the code */
-    uintptr_t tool_versions[_PY_MONITORING_TOOL_IDS];
-    /* Information to support line events */
-    _PyCoLineInstrumentationData *lines;
-    /* The tools that are to be notified for line events for the matching code unit */
-    uint8_t *line_tools;
-    /* Information to support instruction events */
-    /* The underlying instructions, which can themselves be instrumented */
-    uint8_t *per_instruction_opcodes;
-    /* The tools that are to be notified for instruction events for the matching code unit */
-    uint8_t *per_instruction_tools;
-} _PyCoMonitoringData;
 
 #ifdef Py_GIL_DISABLED
 
@@ -151,7 +100,7 @@ typedef struct {
     _PyExecutorArray *co_executors;      /* executors from optimizer */        \
     _PyCoCached *_co_cached;      /* cached co_* attributes */                 \
     uintptr_t _co_instrumentation_version; /* current instrumentation version */ \
-    _PyCoMonitoringData *_co_monitoring; /* Monitoring data */                 \
+    struct _PyCoMonitoringData *_co_monitoring; /* Monitoring data */          \
     Py_ssize_t _co_unique_id;     /* ID used for per-thread refcounting */   \
     int _co_firsttraceable;       /* index of first traceable instruction */   \
     /* Scratch space for extra data relating to the code object.               \

Include/cpython/object.h

@@ -512,7 +512,6 @@ PyAPI_FUNC(int) PyObject_VisitManagedDict(PyObject *obj, visitproc visit, void *
 PyAPI_FUNC(int) _PyObject_SetManagedDict(PyObject *obj, PyObject *new_dict);
 PyAPI_FUNC(void) PyObject_ClearManagedDict(PyObject *obj);
 
-#define TYPE_MAX_WATCHERS 8
 
 typedef int(*PyType_WatchCallback)(PyTypeObject *);
 PyAPI_FUNC(int) PyType_AddWatcher(PyType_WatchCallback callback);

Include/internal/pycore_atexit.h

@@ -12,44 +12,6 @@ extern "C" {
 #endif
 
 
-//###############
-// runtime atexit
-
-typedef void (*atexit_callbackfunc)(void);
-
-struct _atexit_runtime_state {
-    PyMutex mutex;
-#define NEXITFUNCS 32
-    atexit_callbackfunc callbacks[NEXITFUNCS];
-    int ncallbacks;
-};
-
-
-//###################
-// interpreter atexit
-
-typedef void (*atexit_datacallbackfunc)(void *);
-
-typedef struct atexit_callback {
-    atexit_datacallbackfunc func;
-    void *data;
-    struct atexit_callback *next;
-} atexit_callback;
-
-struct atexit_state {
-#ifdef Py_GIL_DISABLED
-    PyMutex ll_callbacks_lock;
-#endif
-    atexit_callback *ll_callbacks;
-
-    // XXX The rest of the state could be moved to the atexit module state
-    // and a low-level callback added for it during module exec.
-    // For the moment we leave it here.
-
-    // List containing tuples with callback information.
-    // e.g. [(func, args, kwargs), ...]
-    PyObject *callbacks;
-};
 
 #ifdef Py_GIL_DISABLED
 #  define _PyAtExit_LockCallbacks(state) PyMutex_Lock(&state->ll_callbacks_lock);

Include/internal/pycore_backoff.h

@@ -10,14 +10,7 @@ extern "C" {
 #endif
 
 #include <assert.h>
-#include <stdbool.h>
-#include <stdint.h>
-
-
-typedef struct {
-    uint16_t value_and_backoff;
-} _Py_BackoffCounter;
-
+#include "pycore_structs.h" // _Py_BackoffCounter
 
 /* 16-bit countdown counters using exponential backoff.