Merge branch 'main' into gh-129605

Author: srinivasreddy

Doc/c-api/init.rst

@@ -622,7 +622,8 @@ Process-wide parameters
       It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
 
    .. deprecated-removed:: 3.13 3.15
-      Get :data:`sys.executable` instead.
+      Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
+      (:data:`sys.executable`) instead.
 
 
 .. c:function:: wchar_t* Py_GetPrefix()
@@ -644,8 +645,10 @@ Process-wide parameters
       It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
 
    .. deprecated-removed:: 3.13 3.15
-      Get :data:`sys.base_prefix` instead, or :data:`sys.prefix` if
-      :ref:`virtual environments <venv-def>` need to be handled.
+      Use :c:func:`PyConfig_Get("base_prefix") <PyConfig_Get>`
+      (:data:`sys.base_prefix`) instead. Use :c:func:`PyConfig_Get("prefix")
+      <PyConfig_Get>` (:data:`sys.prefix`) if :ref:`virtual environments
+      <venv-def>` need to be handled.
 
 
 .. c:function:: wchar_t* Py_GetExecPrefix()
@@ -690,9 +693,11 @@ Process-wide parameters
       It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
 
    .. deprecated-removed:: 3.13 3.15
-      Get :data:`sys.base_exec_prefix` instead, or :data:`sys.exec_prefix` if
-      :ref:`virtual environments <venv-def>` need to be handled.
-
+      Use :c:func:`PyConfig_Get("base_exec_prefix") <PyConfig_Get>`
+      (:data:`sys.base_exec_prefix`) instead. Use
+      :c:func:`PyConfig_Get("exec_prefix") <PyConfig_Get>`
+      (:data:`sys.exec_prefix`) if :ref:`virtual environments <venv-def>` need
+      to be handled.
 
 .. c:function:: wchar_t* Py_GetProgramFullPath()
 
@@ -712,7 +717,8 @@ Process-wide parameters
       It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
 
    .. deprecated-removed:: 3.13 3.15
-      Get :data:`sys.executable` instead.
+      Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
+      (:data:`sys.executable`) instead.
 
 
 .. c:function:: wchar_t* Py_GetPath()
@@ -740,8 +746,8 @@ Process-wide parameters
       It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
 
    .. deprecated-removed:: 3.13 3.15
-      Get :data:`sys.path` instead.
-
+      Use :c:func:`PyConfig_Get("module_search_paths") <PyConfig_Get>`
+      (:data:`sys.path`) instead.
 
 .. c:function:: const char* Py_GetVersion()
 
@@ -926,8 +932,8 @@ Process-wide parameters
       It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
 
    .. deprecated-removed:: 3.13 3.15
-      Get :c:member:`PyConfig.home` or :envvar:`PYTHONHOME` environment
-      variable instead.
+      Use :c:func:`PyConfig_Get("home") <PyConfig_Get>` or the
+      :envvar:`PYTHONHOME` environment variable instead.
 
 
 .. _threads:

Doc/c-api/module.rst

@@ -523,9 +523,6 @@ state:
 
    On success, return ``0``. On error, raise an exception and return ``-1``.
 
-   Return ``-1`` if *value* is ``NULL``. It must be called with an exception
-   raised in this case.
-
    Example usage::
 
        static int
@@ -540,6 +537,10 @@ state:
            return res;
         }
 
+   To be convenient, the function accepts ``NULL`` *value* with an exception
+   set. In this case, return ``-1`` and just leave the raised exception
+   unchanged.
+
    The example can also be written without checking explicitly if *obj* is
    ``NULL``::
 

Doc/deprecations/c-api-pending-removal-in-3.15.rst

@@ -10,25 +10,35 @@ Pending removal in Python 3.15
   :c:func:`PyWeakref_GetRef` on Python 3.12 and older.
 * :c:type:`Py_UNICODE` type and the :c:macro:`!Py_UNICODE_WIDE` macro:
   Use :c:type:`wchar_t` instead.
-* Python initialization functions:
+* Python initialization functions, deprecated in Python 3.13:
 
-  * :c:func:`PySys_ResetWarnOptions`:
-    Clear :data:`sys.warnoptions` and :data:`!warnings.filters` instead.
-  * :c:func:`Py_GetExecPrefix`:
-    Get :data:`sys.base_exec_prefix` and :data:`sys.exec_prefix` instead.
   * :c:func:`Py_GetPath`:
-    Get :data:`sys.path` instead.
+    Use :c:func:`PyConfig_Get("module_search_paths") <PyConfig_Get>`
+    (:data:`sys.path`) instead.
   * :c:func:`Py_GetPrefix`:
-    Get :data:`sys.base_prefix` and :data:`sys.prefix` instead.
+    Use :c:func:`PyConfig_Get("base_prefix") <PyConfig_Get>`
+    (:data:`sys.base_prefix`) instead. Use :c:func:`PyConfig_Get("prefix")
+    <PyConfig_Get>` (:data:`sys.prefix`) if :ref:`virtual environments
+    <venv-def>` need to be handled.
+  * :c:func:`Py_GetExecPrefix`:
+    Use :c:func:`PyConfig_Get("base_exec_prefix") <PyConfig_Get>`
+    (:data:`sys.base_exec_prefix`) instead. Use
+    :c:func:`PyConfig_Get("exec_prefix") <PyConfig_Get>`
+    (:data:`sys.exec_prefix`) if :ref:`virtual environments <venv-def>` need to
+    be handled.
   * :c:func:`Py_GetProgramFullPath`:
-    Get :data:`sys.executable` instead.
+    Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
+    (:data:`sys.executable`) instead.
   * :c:func:`Py_GetProgramName`:
-    Get :data:`sys.executable` instead.
+    Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
+    (:data:`sys.executable`) instead.
   * :c:func:`Py_GetPythonHome`:
-    Get :c:func:`PyConfig_Get("home") <PyConfig_Get>`
-    or the :envvar:`PYTHONHOME` environment variable instead.
+    Use :c:func:`PyConfig_Get("home") <PyConfig_Get>` or the
+    :envvar:`PYTHONHOME` environment variable instead.
 
-  See also the :c:func:`PyConfig_Get` function.
+  The `pythoncapi-compat project
+  <https://github.com/python/pythoncapi-compat/>`__ can be used to get
+  :c:func:`PyConfig_Get` on Python 3.13 and older.
 
 * Functions to configure Python's initialization, deprecated in Python 3.11:
 
@@ -40,6 +50,8 @@ Pending removal in Python 3.15
     Set :c:member:`PyConfig.program_name` instead.
   * :c:func:`!Py_SetPythonHome()`:
     Set :c:member:`PyConfig.home` instead.
+  * :c:func:`PySys_ResetWarnOptions`:
+    Clear :data:`sys.warnoptions` and :data:`!warnings.filters` instead.
 
   The :c:func:`Py_InitializeFromConfig` API should be used with
   :c:type:`PyConfig` instead.

Doc/library/exceptions.rst

@@ -562,9 +562,13 @@ The following exceptions are the exceptions that are usually raised.
 
    Raised when the interpreter finds an internal error, but the situation does not
    look so serious to cause it to abandon all hope. The associated value is a
-   string indicating what went wrong (in low-level terms).
+   string indicating what went wrong (in low-level terms). In :term:`CPython`,
+   this could be raised by incorrectly using Python's C API, such as returning
+   a ``NULL`` value without an exception set.
 
-   You should report this to the author or maintainer of your Python interpreter.
+   If you're confident that this exception wasn't your fault, or the fault of
+   a package you're using, you should report this to the author or maintainer
+   of your Python interpreter.
    Be sure to report the version of the Python interpreter (``sys.version``; it is
    also printed at the start of an interactive Python session), the exact error
    message (the exception's associated value) and if possible the source of the

Doc/library/shutil.rst

@@ -512,7 +512,9 @@ the use of userspace buffers in Python as in "``outfd.write(infd.read())``".
 
 On macOS `fcopyfile`_ is used to copy the file content (not metadata).
 
-On Linux and Solaris :func:`os.sendfile` is used.
+On Linux :func:`os.copy_file_range` or :func:`os.sendfile` is used.
+
+On Solaris :func:`os.sendfile` is used.
 
 On Windows :func:`shutil.copyfile` uses a bigger default buffer size (1 MiB
 instead of 64 KiB) and a :func:`memoryview`-based variant of
@@ -527,6 +529,10 @@ file then shutil will silently fallback on using less efficient
 .. versionchanged:: 3.14
     Solaris now uses :func:`os.sendfile`.
 
+.. versionchanged:: next
+   Copy-on-write or server-side copy may be used internally via
+   :func:`os.copy_file_range` on supported Linux filesystems.
+
 .. _shutil-copytree-example:
 
 copytree example

Include/internal/pycore_opcode_metadata.h

@@ -49,6 +49,7 @@
 # https://bugs.python.org/issue43743#msg393429
 _USE_CP_SENDFILE = (hasattr(os, "sendfile")
                     and sys.platform.startswith(("linux", "android", "sunos")))
+_USE_CP_COPY_FILE_RANGE = hasattr(os, "copy_file_range")
 _HAS_FCOPYFILE = posix and hasattr(posix, "_fcopyfile")  # macOS
 
 # CMD defaults in Windows 10
@@ -107,6 +108,66 @@ def _fastcopy_fcopyfile(fsrc, fdst, flags):
         else:
             raise err from None
 
+def _determine_linux_fastcopy_blocksize(infd):
+    """Determine blocksize for fastcopying on Linux.
+
+    Hopefully the whole file will be copied in a single call.
+    The copying itself should be performed in a loop 'till EOF is
+    reached (0 return) so a blocksize smaller or bigger than the actual
+    file size should not make any difference, also in case the file
+    content changes while being copied.
+    """
+    try:
+        blocksize = max(os.fstat(infd).st_size, 2 ** 23)  # min 8 MiB
+    except OSError:
+        blocksize = 2 ** 27  # 128 MiB
+    # On 32-bit architectures truncate to 1 GiB to avoid OverflowError,
+    # see gh-82500.
+    if sys.maxsize < 2 ** 32:
+        blocksize = min(blocksize, 2 ** 30)
+    return blocksize
+
+def _fastcopy_copy_file_range(fsrc, fdst):
+    """Copy data from one regular mmap-like fd to another by using
+    a high-performance copy_file_range(2) syscall that gives filesystems
+    an opportunity to implement the use of reflinks or server-side copy.
+
+    This should work on Linux >= 4.5 only.
+    """
+    try:
+        infd = fsrc.fileno()
+        outfd = fdst.fileno()
+    except Exception as err:
+        raise _GiveupOnFastCopy(err)  # not a regular file
+
+    blocksize = _determine_linux_fastcopy_blocksize(infd)
+    offset = 0
+    while True:
+        try:
+            n_copied = os.copy_file_range(infd, outfd, blocksize, offset_dst=offset)
+        except OSError as err:
+            # ...in oder to have a more informative exception.
+            err.filename = fsrc.name
+            err.filename2 = fdst.name
+
+            if err.errno == errno.ENOSPC:  # filesystem is full
+                raise err from None
+
+            # Give up on first call and if no data was copied.
+            if offset == 0 and os.lseek(outfd, 0, os.SEEK_CUR) == 0:
+                raise _GiveupOnFastCopy(err)
+
+            raise err
+        else:
+            if n_copied == 0:
+                # If no bytes have been copied yet, copy_file_range
+                # might silently fail.
+                # https://lore.kernel.org/linux-fsdevel/[email protected]/T/#m05753578c7f7882f6e9ffe01f981bc223edef2b0
+                if offset == 0:
+                    raise _GiveupOnFastCopy()
+                break
+            offset += n_copied
+
 def _fastcopy_sendfile(fsrc, fdst):
     """Copy data from one regular mmap-like fd to another by using
     high-performance sendfile(2) syscall.
@@ -128,20 +189,7 @@ def _fastcopy_sendfile(fsrc, fdst):
     except Exception as err:
         raise _GiveupOnFastCopy(err)  # not a regular file
 
-    # Hopefully the whole file will be copied in a single call.
-    # sendfile() is called in a loop 'till EOF is reached (0 return)
-    # so a bufsize smaller or bigger than the actual file size
-    # should not make any difference, also in case the file content
-    # changes while being copied.
-    try:
-        blocksize = max(os.fstat(infd).st_size, 2 ** 23)  # min 8MiB
-    except OSError:
-        blocksize = 2 ** 27  # 128MiB
-    # On 32-bit architectures truncate to 1GiB to avoid OverflowError,
-    # see bpo-38319.
-    if sys.maxsize < 2 ** 32:
-        blocksize = min(blocksize, 2 ** 30)
-
+    blocksize = _determine_linux_fastcopy_blocksize(infd)
     offset = 0
     while True:
         try:
@@ -266,12 +314,20 @@ def copyfile(src, dst, *, follow_symlinks=True):
                         except _GiveupOnFastCopy:
                             pass
                     # Linux / Android / Solaris
-                    elif _USE_CP_SENDFILE:
-                        try:
-                            _fastcopy_sendfile(fsrc, fdst)
-                            return dst
-                        except _GiveupOnFastCopy:
-                            pass
+                    elif _USE_CP_SENDFILE or _USE_CP_COPY_FILE_RANGE:
+                        # reflink may be implicit in copy_file_range.
+                        if _USE_CP_COPY_FILE_RANGE:
+                            try:
+                                _fastcopy_copy_file_range(fsrc, fdst)
+                                return dst
+                            except _GiveupOnFastCopy:
+                                pass
+                        if _USE_CP_SENDFILE:
+                            try:
+                                _fastcopy_sendfile(fsrc, fdst)
+                                return dst
+                            except _GiveupOnFastCopy:
+                                pass
                     # Windows, see:
                     # https://github.com/python/cpython/pull/7160#discussion_r195405230
                     elif _WINDOWS and file_size > 0: