Merge branch 'main' into docs-readline-repl

Author: Yuheng3107

Doc/library/asyncio-eventloop.rst

@@ -611,6 +611,12 @@ Opening network connections
      to bind the socket locally.  The *local_host* and *local_port*
      are looked up using :meth:`getaddrinfo`.
 
+     .. note::
+
+        On Windows, when using the proactor event loop with ``local_addr=None``,
+        an :exc:`OSError` with :attr:`!errno.WSAEINVAL` will be raised
+        when running it.
+
    * *remote_addr*, if given, is a ``(remote_host, remote_port)`` tuple used
      to connect the socket to a remote address.  The *remote_host* and
      *remote_port* are looked up using :meth:`getaddrinfo`.

Doc/library/codecs.rst

@@ -1483,6 +1483,36 @@ to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode`
    Restoration of the aliases for the binary transforms.
 
 
+.. _standalone-codec-functions:
+
+Standalone Codec Functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following functions provide encoding and decoding functionality similar to codecs,
+but are not available as named codecs through :func:`codecs.encode` or :func:`codecs.decode`.
+They are used internally (for example, by :mod:`pickle`) and behave similarly to the
+``string_escape`` codec that was removed in Python 3.
+
+.. function:: codecs.escape_encode(input, errors=None)
+
+   Encode *input* using escape sequences. Similar to how :func:`repr` on bytes
+   produces escaped byte values.
+
+   *input* must be a :class:`bytes` object.
+
+   Returns a tuple ``(output, length)`` where *output* is a :class:`bytes`
+   object and *length* is the number of bytes consumed.
+
+.. function:: codecs.escape_decode(input, errors=None)
+
+   Decode *input* from escape sequences back to the original bytes.
+
+   *input* must be a :term:`bytes-like object`.
+
+   Returns a tuple ``(output, length)`` where *output* is a :class:`bytes`
+   object and *length* is the number of bytes consumed.
+
+
 .. _text-transforms:
 
 Text Transforms

Doc/library/concurrent.futures.rst

@@ -342,6 +342,11 @@ that :class:`ProcessPoolExecutor` will not work in the interactive interpreter.
 Calling :class:`Executor` or :class:`Future` methods from a callable submitted
 to a :class:`ProcessPoolExecutor` will result in deadlock.
 
+Note that the restrictions on functions and arguments needing to picklable as
+per :class:`multiprocessing.Process` apply when using :meth:`~Executor.submit`
+and :meth:`~Executor.map` on a :class:`ProcessPoolExecutor`. A function defined
+in a REPL or a lambda should not be expected to work.
+
 .. class:: ProcessPoolExecutor(max_workers=None, mp_context=None, initializer=None, initargs=(), max_tasks_per_child=None)
 
    An :class:`Executor` subclass that executes calls asynchronously using a pool

Doc/library/multiprocessing.rst

@@ -97,6 +97,10 @@ To show the individual process IDs involved, here is an expanded example::
 For an explanation of why the ``if __name__ == '__main__'`` part is
 necessary, see :ref:`multiprocessing-programming`.
 
+The arguments to :class:`Process` usually need to be unpickleable from within
+the child process. If you tried typing the above example directly into a REPL it
+could lead to an :exc:`AttributeError` in the child process trying to locate the
+*f* function in the ``__main__`` module.
 
 
 .. _multiprocessing-start-methods:
@@ -233,9 +237,12 @@ processes for a different context.  In particular, locks created using
 the *fork* context cannot be passed to processes started using the
 *spawn* or *forkserver* start methods.
 
-A library which wants to use a particular start method should probably
-use :func:`get_context` to avoid interfering with the choice of the
-library user.
+Libraries using :mod:`multiprocessing` or
+:class:`~concurrent.futures.ProcessPoolExecutor` should be designed to allow
+their users to provide their own multiprocessing context.  Using a specific
+context of your own within a library can lead to incompatibilities with the
+rest of the library user's application.  Always document if your library
+requires a specific start method.
 
 .. warning::
 
@@ -538,9 +545,42 @@ The :mod:`multiprocessing` package mostly replicates the API of the
    to pass to *target*.
 
    If a subclass overrides the constructor, it must make sure it invokes the
-   base class constructor (:meth:`Process.__init__`) before doing anything else
+   base class constructor (``super().__init__()``) before doing anything else
    to the process.
 
+   .. note::
+
+      In general, all arguments to :class:`Process` must be picklable.  This is
+      frequently observed when trying to create a :class:`Process` or use a
+      :class:`concurrent.futures.ProcessPoolExecutor` from a REPL with a
+      locally defined *target* function.
+
+      Passing a callable object defined in the current REPL session causes the
+      child process to die via an uncaught :exc:`AttributeError` exception when
+      starting as *target* must have been defined within an importable module
+      in order to be loaded during unpickling.
+
+      Example of this uncatchable error from the child::
+
+         >>> import multiprocessing as mp
+         >>> def knigit():
+         ...     print("Ni!")
+         ...
+         >>> process = mp.Process(target=knigit)
+         >>> process.start()
+         >>> Traceback (most recent call last):
+           File ".../multiprocessing/spawn.py", line ..., in spawn_main
+           File ".../multiprocessing/spawn.py", line ..., in _main
+         AttributeError: module '__main__' has no attribute 'knigit'
+         >>> process
+         <SpawnProcess name='SpawnProcess-1' pid=379473 parent=378707 stopped exitcode=1>
+
+      See :ref:`multiprocessing-programming-spawn`.  While this restriction is
+      not true if using the ``"fork"`` start method, as of Python ``3.14`` that
+      is no longer the default on any platform.  See
+      :ref:`multiprocessing-start-methods`.
+      See also :gh:`132898`.
+
    .. versionchanged:: 3.3
       Added the *daemon* parameter.
 
@@ -3070,10 +3110,10 @@ start method.
 
 More picklability
 
-    Ensure that all arguments to :meth:`Process.__init__` are picklable.
-    Also, if you subclass :class:`~multiprocessing.Process` then make sure that
-    instances will be picklable when the :meth:`Process.start
-    <multiprocessing.Process.start>` method is called.
+    Ensure that all arguments to :class:`~multiprocessing.Process` are
+    picklable.  Also, if you subclass ``Process.__init__``, you must make sure
+    that instances will be picklable when the
+    :meth:`Process.start <multiprocessing.Process.start>` method is called.
 
 Global variables
 

Makefile.pre.in

@@ -227,6 +227,7 @@ ENSUREPIP=      @ENSUREPIP@
 # Internal static libraries
 LIBMPDEC_A= Modules/_decimal/libmpdec/libmpdec.a
 LIBEXPAT_A= Modules/expat/libexpat.a
+LIBHASHLIB_INTERNAL_A=Modules/_hashlib/libhashlib.a
 
 # HACL* build configuration
 LIBHACL_CFLAGS=@LIBHACL_CFLAGS@
@@ -761,6 +762,17 @@ LIBHACL_HMAC_HEADERS= \
 		$(LIBHACL_BLAKE2_HEADERS) \
 		$(LIBHACL_HEADERS)
 
+##########################################################################
+# Internal library for cryptographic primitives
+
+LIBHASHLIB_INTERNAL_OBJS= \
+		Modules/_hashlib/hashlib_buffer.o
+
+LIBHASHLIB_INTERNAL_HEADERS= \
+		Modules/_hashlib/hashlib_buffer.h \
+		Modules/_hashlib/hashlib_fetch.h \
+		Modules/_hashlib/hashlib_mutex.h
+
 #########################################################################
 # Rules
 
@@ -1511,6 +1523,17 @@ $(LIBEXPAT_A): $(LIBEXPAT_OBJS)
 	-rm -f $@
 	$(AR) $(ARFLAGS) $@ $(LIBEXPAT_OBJS)
 
+##########################################################################
+# '_hashlib', '_hmac' and HACL*-based modules helpers
+LIBHASHLIB_INTERNAL_CFLAGS=@LIBHASHLIB_INTERNAL_CFLAGS@ $(PY_STDMODULE_CFLAGS) $(CCSHARED)
+
+Modules/_hashlib/hashlib_buffer.o: Modules/_hashlib/hashlib_buffer.c $(LIBHASHLIB_INTERNAL_HEADERS) $(PYTHON_HEADERS)
+	$(CC) -I$(srcdir)/Modules/_hashlib -c $(LIBHASHLIB_INTERNAL_CFLAGS) -o $@ $(srcdir)/Modules/_hashlib/hashlib_buffer.c
+
+$(LIBHASHLIB_INTERNAL_A): $(LIBHASHLIB_INTERNAL_OBJS)
+	-rm -f $@
+	$(AR) $(ARFLAGS) $@ $(LIBHASHLIB_INTERNAL_OBJS)
+
 ##########################################################################
 # HACL* library build
 #
@@ -3353,21 +3376,21 @@ MODULE__CTYPES_TEST_DEPS=$(srcdir)/Modules/_ctypes/_ctypes_test_generated.c.h
 MODULE__CTYPES_MALLOC_CLOSURE=@MODULE__CTYPES_MALLOC_CLOSURE@
 MODULE__DECIMAL_DEPS=$(srcdir)/Modules/_decimal/docstrings.h @LIBMPDEC_INTERNAL@
 MODULE__ELEMENTTREE_DEPS=$(srcdir)/Modules/pyexpat.c @LIBEXPAT_INTERNAL@
-MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h
+MODULE__HASHLIB_DEPS=@LIBHASHLIB_INTERNAL@
 MODULE__IO_DEPS=$(srcdir)/Modules/_io/_iomodule.h
 
 # HACL*-based cryptographic primitives
-MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_MD5_HEADERS) $(LIBHACL_MD5_LIB_@LIBHACL_LDEPS_LIBTYPE@)
+MODULE__MD5_DEPS=$(MODULE__HASHLIB_DEPS) $(LIBHACL_MD5_HEADERS) $(LIBHACL_MD5_LIB_@LIBHACL_LDEPS_LIBTYPE@)
 MODULE__MD5_LDEPS=$(LIBHACL_MD5_LIB_@LIBHACL_LDEPS_LIBTYPE@)
-MODULE__SHA1_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_SHA1_HEADERS) $(LIBHACL_SHA1_LIB_@LIBHACL_LDEPS_LIBTYPE@)
+MODULE__SHA1_DEPS=$(MODULE__HASHLIB_DEPS) $(LIBHACL_SHA1_HEADERS) $(LIBHACL_SHA1_LIB_@LIBHACL_LDEPS_LIBTYPE@)
 MODULE__SHA1_LDEPS=$(LIBHACL_SHA1_LIB_@LIBHACL_LDEPS_LIBTYPE@)
-MODULE__SHA2_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_SHA2_HEADERS) $(LIBHACL_SHA2_LIB_@LIBHACL_LDEPS_LIBTYPE@)
+MODULE__SHA2_DEPS=$(MODULE__HASHLIB_DEPS) $(LIBHACL_SHA2_HEADERS) $(LIBHACL_SHA2_LIB_@LIBHACL_LDEPS_LIBTYPE@)
 MODULE__SHA2_LDEPS=$(LIBHACL_SHA2_LIB_@LIBHACL_LDEPS_LIBTYPE@)
-MODULE__SHA3_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_SHA3_HEADERS) $(LIBHACL_SHA3_LIB_@LIBHACL_LDEPS_LIBTYPE@)
+MODULE__SHA3_DEPS=$(MODULE__HASHLIB_DEPS) $(LIBHACL_SHA3_HEADERS) $(LIBHACL_SHA3_LIB_@LIBHACL_LDEPS_LIBTYPE@)
 MODULE__SHA3_LDEPS=$(LIBHACL_SHA3_LIB_@LIBHACL_LDEPS_LIBTYPE@)
-MODULE__BLAKE2_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_BLAKE2_HEADERS) $(LIBHACL_BLAKE2_LIB_@LIBHACL_LDEPS_LIBTYPE@)
+MODULE__BLAKE2_DEPS=$(MODULE__HASHLIB_DEPS) $(LIBHACL_BLAKE2_HEADERS) $(LIBHACL_BLAKE2_LIB_@LIBHACL_LDEPS_LIBTYPE@)
 MODULE__BLAKE2_LDEPS=$(LIBHACL_BLAKE2_LIB_@LIBHACL_LDEPS_LIBTYPE@)
-MODULE__HMAC_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_HMAC_HEADERS) $(LIBHACL_HMAC_LIB_@LIBHACL_LDEPS_LIBTYPE@)
+MODULE__HMAC_DEPS=$(MODULE__HASHLIB_DEPS) $(LIBHACL_HMAC_HEADERS) $(LIBHACL_HMAC_LIB_@LIBHACL_LDEPS_LIBTYPE@)
 MODULE__HMAC_LDEPS=$(LIBHACL_HMAC_LIB_@LIBHACL_LDEPS_LIBTYPE@)
 
 MODULE__SOCKET_DEPS=$(srcdir)/Modules/socketmodule.h $(srcdir)/Modules/addrinfo.h $(srcdir)/Modules/getaddrinfo.c $(srcdir)/Modules/getnameinfo.c

Misc/NEWS.d/next/Build/2025-07-22-14-47-45.gh-issue-131876.oaYEEP.rst

@@ -0,0 +1,2 @@
+Remove :file:`!Modules/hashlib.h` and move its content into dedicated files
+now located in ``Modules/_hashlib``. Patch by Bénédikt Tran.

Modules/_hashlib/hashlib_buffer.c

@@ -0,0 +1,65 @@
+#include "hashlib_buffer.h"
+
+int
+_Py_hashlib_data_argument(PyObject **res, PyObject *data, PyObject *string)
+{
+    if (data != NULL && string == NULL) {
+        // called as H(data) or H(data=...)
+        *res = data;
+        return 1;
+    }
+    else if (data == NULL && string != NULL) {
+        // called as H(string=...)
+        if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                         "the 'string' keyword parameter is deprecated since "
+                         "Python 3.15 and slated for removal in Python 3.19; "
+                         "use the 'data' keyword parameter or pass the data "
+                         "to hash as a positional argument instead", 1) < 0)
+        {
+            *res = NULL;
+            return -1;
+        }
+        *res = string;
+        return 1;
+    }
+    else if (data == NULL && string == NULL) {
+        // fast path when no data is given
+        assert(!PyErr_Occurred());
+        *res = NULL;
+        return 0;
+    }
+    else {
+        // called as H(data=..., string)
+        *res = NULL;
+        PyErr_SetString(PyExc_TypeError,
+                        "'data' and 'string' are mutually exclusive "
+                        "and support for 'string' keyword parameter "
+                        "is slated for removal in a future version.");
+        return -1;
+    }
+}
+
+int
+_Py_hashlib_get_buffer_view(PyObject *obj, Py_buffer *view)
+{
+    if (PyUnicode_Check(obj)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "Strings must be encoded before hashing");
+        return -1;
+    }
+    if (!PyObject_CheckBuffer(obj)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "object supporting the buffer API required");
+        return -1;
+    }
+    if (PyObject_GetBuffer(obj, view, PyBUF_SIMPLE) == -1) {
+        return -1;
+    }
+    if (view->ndim > 1) {
+        PyErr_SetString(PyExc_BufferError,
+                        "Buffer must be single dimension");
+        PyBuffer_Release(view);
+        return -1;
+    }
+    return 0;
+}

Modules/_hashlib/hashlib_buffer.h

@@ -0,0 +1,48 @@
+#ifndef _HASHLIB_HASHLIB_BUFFER_H
+#define _HASHLIB_HASHLIB_BUFFER_H
+
+#include "Python.h"
+
+/*
+ * Allow to use the 'data' or 'string' keyword in hashlib.new()
+ * and other hash functions named constructors.
+ *
+ * - If 'data' and 'string' are both non-NULL, set an exception and return -1.
+ * - If 'data' and 'string' are both NULL, set '*res' to NULL and return 0.
+ * - Otherwise, set '*res' to 'data' or 'string' and return 1. A deprecation
+ *   warning is set when 'string' is specified.
+ *
+ * The symbol is exported for '_hashlib' and HACL*-based extension modules.
+ */
+PyAPI_FUNC(int)
+_Py_hashlib_data_argument(PyObject **res, PyObject *data, PyObject *string);
+
+/*
+ * Obtain a buffer view from a buffer-like object 'obj'.
+ *
+ * On success, store the result in 'view' and return 0.
+ * On error, set an exception and return -1.
+ *
+ * The symbol is exported for '_hashlib' and HACL*-based extension modules.
+ */
+PyAPI_FUNC(int)
+_Py_hashlib_get_buffer_view(PyObject *obj, Py_buffer *view);
+
+/*
+ * Call _Py_hashlib_get_buffer_view() and check if it succeeded.
+ *
+ * On error, set an exception and execute the ERRACTION statements.
+ */
+#define GET_BUFFER_VIEW_OR_ERROR(OBJ, VIEW, ERRACTION)      \
+    do {                                                    \
+        if (_Py_hashlib_get_buffer_view(OBJ, VIEW) < 0) {   \
+            assert(PyErr_Occurred());                       \
+            ERRACTION;                                      \
+        }                                                   \
+    } while (0)
+
+/* Specialization of GET_BUFFER_VIEW_OR_ERROR() returning NULL on error. */
+#define GET_BUFFER_VIEW_OR_ERROUT(OBJ, VIEW)                \
+    GET_BUFFER_VIEW_OR_ERROR(OBJ, VIEW, return NULL)
+
+#endif // !_HASHLIB_HASHLIB_BUFFER_H

Modules/_hashlib/hashlib_fetch.h

@@ -0,0 +1,19 @@
+/*
+ * Utilities used when fetching a message digest from a digest-like identifier.
+ */
+
+#ifndef _HASHLIB_HASHLIB_FETCH_H
+#define _HASHLIB_HASHLIB_FETCH_H
+
+#include "Python.h"
+
+/*
+ * Internal error messages used for reporting an unsupported hash algorithm.
+ * The algorithm can be given by its name, a callable or a PEP-247 module.
+ * The same message is raised by Lib/hashlib.py::__get_builtin_constructor()
+ * and _hmacmodule.c::find_hash_info().
+ */
+#define _Py_HASHLIB_UNSUPPORTED_ALGORITHM       "unsupported hash algorithm %S"
+#define _Py_HASHLIB_UNSUPPORTED_STR_ALGORITHM   "unsupported hash algorithm %s"
+
+#endif // !_HASHLIB_HASHLIB_FETCH_H