Skip to content

Fix portability of sendfile(2) support detection for Lustre filesystems #139416

@bnikolic

Description

@bnikolic

Bug report

Bug description:

On Lustre on Linux (specifically the AWS implementation) there is a rare and difficult to reproduce failure of shutil.copyfile in its sendfile(2) fast path because the syscall returns ENODATA. The situation is recoverable by falling back to the standard read/write file copy (patch forthcoming).

Reproducing the bug requires the filesystem to be in a specific state which I have not been able to reproduce at will but we've seen it on occasions by multiple users. Additionally similar bug has been seen on rust fs::copy which similarly does not trap the ENODATA error.

Following is sufficient to reproduce bug for the specific (rare) file that fails:

import shutil
shutil.copyfile('src', 'dst')

error:

sent = os.sendfile(outfd, infd, offset, blocksize)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 61] No data available: '/shared/fsx1/filename' 

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13bugs and security fixes3.14bugs and security fixes3.15new features, bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions