diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 0333fe9f9967f8..ecff10274d5115 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3224,6 +3224,9 @@ features: .. versionchanged:: 3.12 ``st_birthtime`` is now available on Windows. + .. versionchanged:: next + ``st_birthtime`` is now present on Linux. The value will be ``0.0`` + on kernel versions < 4.11 or if not supported by the filesystem. .. attribute:: st_birthtime_ns @@ -3232,6 +3235,9 @@ features: :exc:`AttributeError`. .. versionadded:: 3.12 + .. versionchanged:: next + ``st_birthtime_ns`` is now present on Linux. The value will be ``0`` + on kernel versions < 4.11 or if not supported by the filesystem. .. note:: @@ -3364,6 +3370,140 @@ features: Added the :attr:`st_birthtime` member on Windows. + .. versionchanged:: next + Added the :attr:`st_birthtime` and :attr:`st_birthtime_ns` members on + Linux. + + +.. function:: statx(path, mask, *, dir_fd=None, follow_symlinks=True, sync=None) + + Get the status of a file or file descriptor by performing a :c:func:`!statx` + system call on the given path. *path* may be specified as either a string or + bytes -- directly or indirectly through the :class:`PathLike` interface -- + or as an open file descriptor. *mask* is a combination of the module-level + :const:`STATX_* ` constants specifying the information to + retrieve. Returns a :class:`statx_result` object whose + :attr:`~os.statx_result.stx_mask` attribute specifies the information + actually retrieved (which may differ from *mask*). + + The optional parameter *sync* controls the freshness of the returned + information. ``sync=True`` requests that the kernel return up-to-date + information, even when doing so is expensive (for example, requiring a + round trip to the server for a file on a network filesystem). + ``sync=False`` requests that the kernel return cached information if + available. ``sync=None`` expresses no preference, in which case the kernel + will return information as fresh as :func:`~os.stat` does. + + This function supports :ref:`specifying a file descriptor `, + :ref:`paths relative to directory descriptors `, and + :ref:`not following symlinks `. + + .. seealso:: The :manpage:`statx(2)` man page. + + .. availability:: Linux >= 4.11 with glibc >= 2.28. + + .. versionadded:: next + + +.. class:: statx_result + + Object whose attributes correspond roughly to the members of the + :c:struct:`!statx` structure. It is used for the result of :func:`os.statx`. + :class:`!statx_result` has all of the attributes of :class:`stat_result` + available on Linux, but is not a subclass of :class:`stat_result` nor a + tuple. :class:`!statx_result` has the following additional attributes: + + .. attribute:: stx_mask + + Bitmask of :const:`STATX_* ` constants specifying the + information retrieved, which may differ from what was requested depending + on the filesystem, filesystem type, and kernel version. All attributes + of this class are accessible regardless of the value of + :attr:`!stx_mask`, and they may have useful fictitious values. For + example, for a file on a network filesystem, :const:`STATX_UID` and + :const:`STATX_GID` may be unset because file ownership on the server is + based on an external user database, but :attr:`!st_uid` and + :attr:`!st_gid` may contain the IDs of the local user who controls the + mount. + + .. attribute:: stx_attributes_mask + + Bitmask of :const:`!STATX_ATTR_* ` constants + specifying the attributes bits supported for this file. + + .. attribute:: stx_attributes + + Bitmask of :const:`!STATX_ATTR_* ` constants + specifying the attributes of this file. + + .. attribute:: stx_mnt_id + + Mount ID. + + .. attribute:: stx_dio_mem_align + + Direct I/O memory buffer alignment requirement. + + .. attribute:: stx_dio_offset_align + + Direct I/O file offset alignment requirement. + + .. attribute:: stx_subvol + + Subvolume ID. + + .. attribute:: stx_atomic_write_unit_min + + Minimum size for direct I/O with torn-write protection. + + .. attribute:: stx_atomic_write_unit_max + + Maximum size for direct I/O with torn-write protection. + + .. attribute:: stx_atomic_write_segments_max + + Maximum iovecs for direct I/O with torn-write protection. + + .. attribute:: stx_dio_read_offset_align + + Direct I/O file offset alignment requirement for reads. + + .. attribute:: stx_atomic_write_unit_max_opt + + Maximum optimized size for direct I/O with torn-write protection. + + .. seealso:: The :manpage:`statx(2)` man page. + + .. availability:: Linux >= 4.11 with glibc >= 2.28. + + .. versionadded:: next + +.. data:: STATX_TYPE + STATX_MODE + STATX_NLINK + STATX_UID + STATX_GID + STATX_ATIME + STATX_MTIME + STATX_CTIME + STATX_INO + STATX_SIZE + STATX_BLOCKS + STATX_BASIC_STATS + STATX_BTIME + STATX_MNT_ID + STATX_DIOALIGN + STATX_MNT_ID_UNIQUE + STATX_SUBVOL + STATX_WRITE_ATOMIC + STATX_DIO_READ_ALIGN + + Bitflags for use as the *mask* parameter to :func:`os.statx`. + + .. availability:: Linux >= 4.11 with glibc >= 2.28. + + .. versionadded:: next + .. function:: statvfs(path) diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 8434b2e8c75cf4..1cbec3ab847c5f 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -493,3 +493,22 @@ constants, but are not an exhaustive list. IO_REPARSE_TAG_APPEXECLINK .. versionadded:: 3.8 + +On Linux, the following file attribute constants are available for use when +testing bits in the :attr:`~os.statx_result.stx_attributes` and +:attr:`~os.statx_result.stx_attributes_mask` members returned by +:func:`os.statx`. See the :manpage:`statx(2)` man page for more detail on the +meaning of these constants. + +.. data:: STATX_ATTR_COMPRESSED + STATX_ATTR_IMMUTABLE + STATX_ATTR_APPEND + STATX_ATTR_NODUMP + STATX_ATTR_ENCRYPTED + STATX_ATTR_AUTOMOUNT + STATX_ATTR_MOUNT_ROOT + STATX_ATTR_VERITY + STATX_ATTR_DAX + STATX_ATTR_WRITE_ATOMIC + + .. versionadded:: next diff --git a/Include/internal/pycore_global_objects_fini_generated.h b/Include/internal/pycore_global_objects_fini_generated.h index 63888eab7b4481..147a4b80c33b34 100644 --- a/Include/internal/pycore_global_objects_fini_generated.h +++ b/Include/internal/pycore_global_objects_fini_generated.h @@ -1096,6 +1096,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) { _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(loop)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(manual_reset)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(mapping)); + _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(mask)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(match)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(max_length)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(maxdigits)); @@ -1287,6 +1288,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) { _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(sub_key)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(subcalls)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(symmetric_difference_update)); + _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(sync)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(tabsize)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(tag)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(target)); diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index b863a7c970e3d4..41c821fc18da39 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -587,6 +587,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(loop) STRUCT_FOR_ID(manual_reset) STRUCT_FOR_ID(mapping) + STRUCT_FOR_ID(mask) STRUCT_FOR_ID(match) STRUCT_FOR_ID(max_length) STRUCT_FOR_ID(maxdigits) @@ -778,6 +779,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(sub_key) STRUCT_FOR_ID(subcalls) STRUCT_FOR_ID(symmetric_difference_update) + STRUCT_FOR_ID(sync) STRUCT_FOR_ID(tabsize) STRUCT_FOR_ID(tag) STRUCT_FOR_ID(target) diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h index 3ce7200ffeb6a4..7a9c8aaff3ceb6 100644 --- a/Include/internal/pycore_runtime_init_generated.h +++ b/Include/internal/pycore_runtime_init_generated.h @@ -1094,6 +1094,7 @@ extern "C" { INIT_ID(loop), \ INIT_ID(manual_reset), \ INIT_ID(mapping), \ + INIT_ID(mask), \ INIT_ID(match), \ INIT_ID(max_length), \ INIT_ID(maxdigits), \ @@ -1285,6 +1286,7 @@ extern "C" { INIT_ID(sub_key), \ INIT_ID(subcalls), \ INIT_ID(symmetric_difference_update), \ + INIT_ID(sync), \ INIT_ID(tabsize), \ INIT_ID(tag), \ INIT_ID(target), \ diff --git a/Include/internal/pycore_unicodeobject_generated.h b/Include/internal/pycore_unicodeobject_generated.h index e76e603230a5db..8e664cf9a998ea 100644 --- a/Include/internal/pycore_unicodeobject_generated.h +++ b/Include/internal/pycore_unicodeobject_generated.h @@ -2136,6 +2136,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) { _PyUnicode_InternStatic(interp, &string); assert(_PyUnicode_CheckConsistency(string, 1)); assert(PyUnicode_GET_LENGTH(string) != 1); + string = &_Py_ID(mask); + _PyUnicode_InternStatic(interp, &string); + assert(_PyUnicode_CheckConsistency(string, 1)); + assert(PyUnicode_GET_LENGTH(string) != 1); string = &_Py_ID(match); _PyUnicode_InternStatic(interp, &string); assert(_PyUnicode_CheckConsistency(string, 1)); @@ -2900,6 +2904,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) { _PyUnicode_InternStatic(interp, &string); assert(_PyUnicode_CheckConsistency(string, 1)); assert(PyUnicode_GET_LENGTH(string) != 1); + string = &_Py_ID(sync); + _PyUnicode_InternStatic(interp, &string); + assert(_PyUnicode_CheckConsistency(string, 1)); + assert(PyUnicode_GET_LENGTH(string) != 1); string = &_Py_ID(tabsize); _PyUnicode_InternStatic(interp, &string); assert(_PyUnicode_CheckConsistency(string, 1)); diff --git a/Lib/os.py b/Lib/os.py index 710d6f8cfcdf74..09c9e47f419771 100644 --- a/Lib/os.py +++ b/Lib/os.py @@ -131,6 +131,8 @@ def _add(str, fn): _add("HAVE_UNLINKAT", "unlink") _add("HAVE_UNLINKAT", "rmdir") _add("HAVE_UTIMENSAT", "utime") + if _exists("statx"): + _set.add(statx) supports_dir_fd = _set _set = set() @@ -152,6 +154,8 @@ def _add(str, fn): _add("HAVE_FPATHCONF", "pathconf") if _exists("statvfs") and _exists("fstatvfs"): # mac os x10.3 _add("HAVE_FSTATVFS", "statvfs") + if _exists("statx"): + _set.add(statx) supports_fd = _set _set = set() @@ -190,6 +194,8 @@ def _add(str, fn): _add("HAVE_FSTATAT", "stat") _add("HAVE_UTIMENSAT", "utime") _add("MS_WINDOWS", "stat") + if _exists("statx"): + _set.add(statx) supports_follow_symlinks = _set del _set diff --git a/Lib/stat.py b/Lib/stat.py index 1b4ed1ebc940ef..ab1b25b9d6351c 100644 --- a/Lib/stat.py +++ b/Lib/stat.py @@ -200,6 +200,21 @@ def filemode(mode): FILE_ATTRIBUTE_VIRTUAL = 65536 +# Linux STATX_ATTR constants for interpreting os.statx()'s +# "stx_attributes" and "stx_attributes_mask" members + +STATX_ATTR_COMPRESSED = 0x00000004 +STATX_ATTR_IMMUTABLE = 0x00000010 +STATX_ATTR_APPEND = 0x00000020 +STATX_ATTR_NODUMP = 0x00000040 +STATX_ATTR_ENCRYPTED = 0x00000800 +STATX_ATTR_AUTOMOUNT = 0x00001000 +STATX_ATTR_MOUNT_ROOT = 0x00002000 +STATX_ATTR_VERITY = 0x00100000 +STATX_ATTR_DAX = 0x00200000 +STATX_ATTR_WRITE_ATOMIC = 0x00400000 + + # If available, use C implementation try: from _stat import * diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index cd15aa10f16de8..c85af87d25e2cd 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -640,6 +640,14 @@ def setUp(self): self.addCleanup(os_helper.unlink, self.fname) create_file(self.fname, b"ABC") + def check_timestamp_agreement(self, result, names): + # Make sure that the st_?time and st_?time_ns fields roughly agree + # (they should always agree up to around tens-of-microseconds) + for name in names: + floaty = int(getattr(result, name) * 100000) + nanosecondy = getattr(result, name + "_ns") // 10000 + self.assertAlmostEqual(floaty, nanosecondy, delta=2, msg=name) + def check_stat_attributes(self, fname): result = os.stat(fname) @@ -660,21 +668,15 @@ def trunc(x): return x result[getattr(stat, name)]) self.assertIn(attr, members) - # Make sure that the st_?time and st_?time_ns fields roughly agree - # (they should always agree up to around tens-of-microseconds) - for name in 'st_atime st_mtime st_ctime'.split(): - floaty = int(getattr(result, name) * 100000) - nanosecondy = getattr(result, name + "_ns") // 10000 - self.assertAlmostEqual(floaty, nanosecondy, delta=2) - - # Ensure both birthtime and birthtime_ns roughly agree, if present + time_attributes = ['st_atime', 'st_mtime', 'st_ctime'] try: - floaty = int(result.st_birthtime * 100000) - nanosecondy = result.st_birthtime_ns // 10000 + result.st_birthtime + result.st_birthtime_ns except AttributeError: pass else: - self.assertAlmostEqual(floaty, nanosecondy, delta=2) + time_attributes.append('st_birthtime') + self.check_timestamp_agreement(result, time_attributes) try: result[200] @@ -735,6 +737,89 @@ def test_stat_result_pickle(self): unpickled = pickle.loads(p) self.assertEqual(result, unpickled) + def check_statx_attributes(self, fname): + maximal_mask = 0 + for name in dir(os): + if name.startswith('STATX_'): + maximal_mask |= getattr(os, name) + result = os.statx(self.fname, maximal_mask) + + time_attributes = ('st_atime', 'st_mtime', 'st_ctime', 'st_birthtime') + self.check_timestamp_agreement(result, time_attributes) + + # Check that valid attributes match os.stat. + requirements = ( + ('st_mode', os.STATX_TYPE | os.STATX_MODE), + ('st_nlink', os.STATX_NLINK), + ('st_uid', os.STATX_UID), + ('st_gid', os.STATX_GID), + ('st_atime', os.STATX_ATIME), + ('st_atime_ns', os.STATX_ATIME), + ('st_mtime', os.STATX_MTIME), + ('st_mtime_ns', os.STATX_MTIME), + ('st_ctime', os.STATX_CTIME), + ('st_ctime_ns', os.STATX_CTIME), + ('st_ino', os.STATX_INO), + ('st_size', os.STATX_SIZE), + ('st_blocks', os.STATX_BLOCKS), + ('st_birthtime', os.STATX_BTIME), + ('st_birthtime_ns', os.STATX_BTIME), + # unconditionally valid members + ('st_blksize', 0), + ('st_dev', 0), + ('st_rdev', 0), + ) + basic_result = os.stat(self.fname) + for name, bits in requirements: + if result.stx_mask & bits == bits: + x = getattr(result, name) + b = getattr(basic_result, name) + if isinstance(x, float): + self.assertAlmostEqual(x, b, msg=name) + else: + self.assertEqual(x, b, msg=name) + + # Access all the attributes multiple times to test cache refcounting. + members = [name for name in dir(result) + if name.startswith('st_') or name.startswith('stx_')] + for _ in range(10): + for name in members: + getattr(result, name) + + for name in members: + try: + setattr(result, name, 1) + self.fail("No exception raised") + except AttributeError: + pass + + self.assertEqual(result.stx_attributes & result.stx_attributes_mask, + result.stx_attributes) + + @unittest.skipUnless(hasattr(os, 'statx'), 'test needs os.statx()') + def test_statx_attributes(self): + self.check_statx_attributes(self.fname) + + @unittest.skipUnless(hasattr(os, 'statx'), 'test needs os.statx()') + def test_statx_attributes_bytes(self): + try: + fname = self.fname.encode(sys.getfilesystemencoding()) + except UnicodeEncodeError: + self.skipTest("cannot encode %a for the filesystem" % self.fname) + self.check_statx_attributes(fname) + + @unittest.skipUnless(hasattr(os, 'statx'), 'test needs os.statx()') + def test_statx_attributes_pathlike(self): + self.check_statx_attributes(FakePath(self.fname)) + + @unittest.skipUnless(hasattr(os, 'statx'), 'test needs os.statx()') + def test_statx_sync(self): + # Test sync= kwarg parsing. (We can't predict if or how the result + # will change.) + for sync in (False, True): + with self.subTest(sync=sync): + os.statx(self.fname, os.STATX_BASIC_STATS, sync=sync) + @unittest.skipUnless(hasattr(os, 'statvfs'), 'test needs os.statvfs()') def test_statvfs_attributes(self): result = os.statvfs(self.fname) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index 2af11888b17c1d..fc23a24466b791 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1611,33 +1611,41 @@ def test_chown_dir_fd(self): with self.prepare_file() as (dir_fd, name, fullname): posix.chown(name, os.getuid(), os.getgid(), dir_fd=dir_fd) - @unittest.skipUnless(os.stat in os.supports_dir_fd, "test needs dir_fd support in os.stat()") - def test_stat_dir_fd(self): + def check_statlike_dir_fd(self, func): with self.prepare() as (dir_fd, name, fullname): with open(fullname, 'w') as outfile: outfile.write("testline\n") self.addCleanup(posix.unlink, fullname) - s1 = posix.stat(fullname) - s2 = posix.stat(name, dir_fd=dir_fd) - self.assertEqual(s1, s2) - s2 = posix.stat(fullname, dir_fd=None) - self.assertEqual(s1, s2) + s1 = func(fullname) + s2 = func(name, dir_fd=dir_fd) + self.assertEqual((s1.st_dev, s1.st_ino), (s2.st_dev, s2.st_ino)) + s2 = func(fullname, dir_fd=None) + self.assertEqual((s1.st_dev, s1.st_ino), (s2.st_dev, s2.st_ino)) self.assertRaisesRegex(TypeError, 'should be integer or None, not', - posix.stat, name, dir_fd=posix.getcwd()) + func, name, dir_fd=posix.getcwd()) self.assertRaisesRegex(TypeError, 'should be integer or None, not', - posix.stat, name, dir_fd=float(dir_fd)) + func, name, dir_fd=float(dir_fd)) self.assertRaises(OverflowError, - posix.stat, name, dir_fd=10**20) + func, name, dir_fd=10**20) for fd in False, True: with self.assertWarnsRegex(RuntimeWarning, 'bool is used as a file descriptor') as cm: with self.assertRaises(OSError): - posix.stat('nonexisting', dir_fd=fd) + func('nonexisting', dir_fd=fd) self.assertEqual(cm.filename, __file__) + @unittest.skipUnless(os.stat in os.supports_dir_fd, "test needs dir_fd support in os.stat()") + def test_stat_dir_fd(self): + self.check_statlike_dir_fd(posix.stat) + + @unittest.skipUnless(hasattr(posix, 'statx'), "test needs os.statx()") + def test_statx_dir_fd(self): + func = lambda path, **kwargs: posix.statx(path, os.STATX_INO, **kwargs) + self.check_statlike_dir_fd(func) + @unittest.skipUnless(os.utime in os.supports_dir_fd, "test needs dir_fd support in os.utime()") def test_utime_dir_fd(self): with self.prepare_file() as (dir_fd, name, fullname): diff --git a/Misc/ACKS b/Misc/ACKS index 37b7988606fa99..a7ecc8ccebb92f 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -210,6 +210,7 @@ Médéric Boquien Matias Bordese Jonas Borgström Jurjen Bos +Jeffrey Bosboom Peter Bosch Dan Boswell Eric Bouck diff --git a/Misc/NEWS.d/next/Library/2025-07-07-22-19-48.gh-issue-83714.F_vbIm.rst b/Misc/NEWS.d/next/Library/2025-07-07-22-19-48.gh-issue-83714.F_vbIm.rst new file mode 100644 index 00000000000000..ffbf5232e0d2b0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-07-07-22-19-48.gh-issue-83714.F_vbIm.rst @@ -0,0 +1,3 @@ +Provide :func:`os.statx` and populate :attr:`~os.stat_result.st_birthtime` +in the return value of :func:`os.stat` on Linux kernel 4.11 and later using +the statx system call. Contributed by Jeffrey Bosboom and Erin of Yukis. diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 45e7c0d6451c15..79b2fed0621821 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -186,6 +186,141 @@ os_lstat(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kw return return_value; } +#if defined(HAVE_STATX) + +PyDoc_STRVAR(os_statx__doc__, +"statx($module, /, path, mask, *, dir_fd=None, follow_symlinks=True,\n" +" sync=None)\n" +"--\n" +"\n" +"Perform a statx system call on the given path.\n" +"\n" +" path\n" +" Path to be examined; can be string, bytes, a path-like object or\n" +" open-file-descriptor int.\n" +" mask\n" +" A bitmask of STATX_* constants defining the requested information.\n" +" dir_fd\n" +" If not None, it should be a file descriptor open to a directory,\n" +" and path should be a relative string; path will then be relative to\n" +" that directory.\n" +" follow_symlinks\n" +" If False, and the last element of the path is a symbolic link,\n" +" statx will examine the symbolic link itself instead of the file\n" +" the link points to.\n" +" sync\n" +" If True, statx will return up-to-date values, even if doing so is\n" +" expensive. If False, statx will return cached values if possible.\n" +" If None, statx lets the operating system decide.\n" +"\n" +"It\'s an error to use dir_fd or follow_symlinks when specifying path as\n" +" an open file descriptor."); + +#define OS_STATX_METHODDEF \ + {"statx", _PyCFunction_CAST(os_statx), METH_FASTCALL|METH_KEYWORDS, os_statx__doc__}, + +static PyObject * +os_statx_impl(PyObject *module, path_t *path, unsigned int mask, int dir_fd, + int follow_symlinks, int sync); + +static PyObject * +os_statx(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) + + #define NUM_KEYWORDS 5 + static struct { + PyGC_Head _this_is_not_used; + PyObject_VAR_HEAD + Py_hash_t ob_hash; + PyObject *ob_item[NUM_KEYWORDS]; + } _kwtuple = { + .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS) + .ob_hash = -1, + .ob_item = { &_Py_ID(path), &_Py_ID(mask), &_Py_ID(dir_fd), &_Py_ID(follow_symlinks), &_Py_ID(sync), }, + }; + #undef NUM_KEYWORDS + #define KWTUPLE (&_kwtuple.ob_base.ob_base) + + #else // !Py_BUILD_CORE + # define KWTUPLE NULL + #endif // !Py_BUILD_CORE + + static const char * const _keywords[] = {"path", "mask", "dir_fd", "follow_symlinks", "sync", NULL}; + static _PyArg_Parser _parser = { + .keywords = _keywords, + .fname = "statx", + .kwtuple = KWTUPLE, + }; + #undef KWTUPLE + PyObject *argsbuf[5]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 2; + path_t path = PATH_T_INITIALIZE_P("statx", "path", 0, 0, 0, 1); + unsigned int mask; + int dir_fd = DEFAULT_DIR_FD; + int follow_symlinks = 1; + int sync = -1; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, + /*minpos*/ 2, /*maxpos*/ 2, /*minkw*/ 0, /*varpos*/ 0, argsbuf); + if (!args) { + goto exit; + } + if (!path_converter(args[0], &path)) { + goto exit; + } + { + Py_ssize_t _bytes = PyLong_AsNativeBytes(args[1], &mask, sizeof(unsigned int), + Py_ASNATIVEBYTES_NATIVE_ENDIAN | + Py_ASNATIVEBYTES_ALLOW_INDEX | + Py_ASNATIVEBYTES_UNSIGNED_BUFFER); + if (_bytes < 0) { + goto exit; + } + if ((size_t)_bytes > sizeof(unsigned int)) { + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "integer value out of range", 1) < 0) + { + goto exit; + } + } + } + if (!noptargs) { + goto skip_optional_kwonly; + } + if (args[2]) { + if (!dir_fd_converter(args[2], &dir_fd)) { + goto exit; + } + if (!--noptargs) { + goto skip_optional_kwonly; + } + } + if (args[3]) { + follow_symlinks = PyObject_IsTrue(args[3]); + if (follow_symlinks < 0) { + goto exit; + } + if (!--noptargs) { + goto skip_optional_kwonly; + } + } + if (!optional_bool_converter(args[4], &sync)) { + goto exit; + } +skip_optional_kwonly: + return_value = os_statx_impl(module, &path, mask, dir_fd, follow_symlinks, sync); + +exit: + /* Cleanup for path */ + path_cleanup(&path); + + return return_value; +} + +#endif /* defined(HAVE_STATX) */ + PyDoc_STRVAR(os_access__doc__, "access($module, /, path, mode, *, dir_fd=None, effective_ids=False,\n" " follow_symlinks=True)\n" @@ -12769,6 +12904,10 @@ os__emscripten_log(PyObject *module, PyObject *const *args, Py_ssize_t nargs, Py #endif /* defined(__EMSCRIPTEN__) */ +#ifndef OS_STATX_METHODDEF + #define OS_STATX_METHODDEF +#endif /* !defined(OS_STATX_METHODDEF) */ + #ifndef OS_TTYNAME_METHODDEF #define OS_TTYNAME_METHODDEF #endif /* !defined(OS_TTYNAME_METHODDEF) */ @@ -13444,4 +13583,4 @@ os__emscripten_log(PyObject *module, PyObject *const *args, Py_ssize_t nargs, Py #ifndef OS__EMSCRIPTEN_LOG_METHODDEF #define OS__EMSCRIPTEN_LOG_METHODDEF #endif /* !defined(OS__EMSCRIPTEN_LOG_METHODDEF) */ -/*[clinic end generated code: output=92662828d49f5d88 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=43915d63088debf2 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 50d0ff1dc2127c..3fb78345d7e26b 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -40,6 +40,7 @@ // --- System includes ------------------------------------------------------ +#include // offsetof() #include // ctermid() #include // system() @@ -408,6 +409,33 @@ extern char *ctermid_r(char *); # define STRUCT_STAT struct stat #endif +#ifdef HAVE_STATX +# pragma weak statx +/* provide constants introduced later than statx itself */ +# ifndef STATX_MNT_ID +# define STATX_MNT_ID 0x00001000U +# endif +# ifndef STATX_DIOALIGN +# define STATX_DIOALIGN 0x00002000U +# endif +# ifndef STATX_MNT_ID_UNIQUE +# define STATX_MNT_ID_UNIQUE 0x00004000U +# endif +# ifndef STATX_SUBVOL +# define STATX_SUBVOL 0x00008000U +# endif +# ifndef STATX_WRITE_ATOMIC +# define STATX_WRITE_ATOMIC 0x00010000U +# endif +# ifndef STATX_DIO_READ_ALIGN +# define STATX_DIO_READ_ALIGN 0x00020000U +# endif +# define _Py_STATX_KNOWN (STATX_BASIC_STATS | STATX_BTIME | STATX_MNT_ID | \ + STATX_DIOALIGN | STATX_MNT_ID_UNIQUE | \ + STATX_SUBVOL | STATX_WRITE_ATOMIC | \ + STATX_DIO_READ_ALIGN) +#endif /* HAVE_STATX */ + #if !defined(EX_OK) && defined(EXIT_SUCCESS) # define EX_OK EXIT_SUCCESS @@ -1159,6 +1187,9 @@ typedef struct { #endif newfunc statresult_new_orig; PyObject *StatResultType; +#ifdef HAVE_STATX + PyObject *StatxResultType; +#endif PyObject *StatVFSResultType; PyObject *TerminalSizeType; PyObject *TimesResultType; @@ -2353,10 +2384,10 @@ static PyStructSequence_Field stat_result_fields[] = { #ifdef HAVE_STRUCT_STAT_ST_GEN {"st_gen", "generation number"}, #endif -#if defined(HAVE_STRUCT_STAT_ST_BIRTHTIME) || defined(MS_WINDOWS) +#if defined(HAVE_STRUCT_STAT_ST_BIRTHTIME) || defined(HAVE_STATX) || defined(MS_WINDOWS) {"st_birthtime", "time of creation"}, #endif -#ifdef MS_WINDOWS +#if defined(HAVE_STATX) || defined(MS_WINDOWS) {"st_birthtime_ns", "time of creation in nanoseconds"}, #endif #ifdef HAVE_STRUCT_STAT_ST_FILE_ATTRIBUTES @@ -2401,13 +2432,13 @@ static PyStructSequence_Field stat_result_fields[] = { #define ST_GEN_IDX ST_FLAGS_IDX #endif -#if defined(HAVE_STRUCT_STAT_ST_BIRTHTIME) || defined(MS_WINDOWS) +#if defined(HAVE_STRUCT_STAT_ST_BIRTHTIME) || defined(HAVE_STATX) || defined(MS_WINDOWS) #define ST_BIRTHTIME_IDX (ST_GEN_IDX+1) #else #define ST_BIRTHTIME_IDX ST_GEN_IDX #endif -#ifdef MS_WINDOWS +#if defined(HAVE_STATX) || defined(MS_WINDOWS) #define ST_BIRTHTIME_NS_IDX (ST_BIRTHTIME_IDX+1) #else #define ST_BIRTHTIME_NS_IDX ST_BIRTHTIME_IDX @@ -2539,6 +2570,9 @@ _posix_clear(PyObject *module) Py_CLEAR(state->SchedParamType); #endif Py_CLEAR(state->StatResultType); +#ifdef HAVE_STATX + Py_CLEAR(state->StatxResultType); +#endif Py_CLEAR(state->StatVFSResultType); Py_CLEAR(state->TerminalSizeType); Py_CLEAR(state->TimesResultType); @@ -2564,6 +2598,9 @@ _posix_traverse(PyObject *module, visitproc visit, void *arg) Py_VISIT(state->SchedParamType); #endif Py_VISIT(state->StatResultType); +#ifdef HAVE_STATX + Py_VISIT(state->StatxResultType); +#endif Py_VISIT(state->StatVFSResultType); Py_VISIT(state->TerminalSizeType); Py_VISIT(state->TimesResultType); @@ -2584,12 +2621,45 @@ _posix_free(void *module) _posix_clear((PyObject *)module); } +#define SEC_TO_NS (1000000000LL) +static PyObject * +nanosecond_timestamp(_posixstate *state, time_t sec, unsigned long nsec) { + /* 1677-09-21 00:12:44 to 2262-04-11 23:47:15 UTC inclusive */ + if ((LLONG_MIN/SEC_TO_NS) <= sec && sec <= (LLONG_MAX/SEC_TO_NS - 1)) { + return PyLong_FromLongLong(sec * SEC_TO_NS + nsec); + } + else { + PyObject *s_in_ns = NULL; + PyObject *s = _PyLong_FromTime_t(sec); + PyObject *ns_fractional = PyLong_FromUnsignedLong(nsec); + if (s == NULL || ns_fractional == NULL) { + goto exit; + } + + s_in_ns = PyNumber_Multiply(s, state->billion); + if (s_in_ns == NULL) { + goto exit; + } + + PyObject *ns_total = PyNumber_Add(s_in_ns, ns_fractional); + if (ns_total == NULL) { + goto exit; + } + return ns_total; + + exit: + Py_XDECREF(s); + Py_XDECREF(ns_fractional); + Py_XDECREF(s_in_ns); + return NULL; + } +} + static int fill_time(_posixstate *state, PyObject *v, int s_index, int f_index, int ns_index, time_t sec, unsigned long nsec) { assert(!PyErr_Occurred()); -#define SEC_TO_NS (1000000000LL) assert(nsec < SEC_TO_NS); if (s_index >= 0) { @@ -2608,50 +2678,18 @@ fill_time(_posixstate *state, PyObject *v, int s_index, int f_index, PyStructSequence_SET_ITEM(v, f_index, float_s); } - int res = -1; if (ns_index >= 0) { - /* 1677-09-21 00:12:44 to 2262-04-11 23:47:15 UTC inclusive */ - if ((LLONG_MIN/SEC_TO_NS) <= sec && sec <= (LLONG_MAX/SEC_TO_NS - 1)) { - PyObject *ns_total = PyLong_FromLongLong(sec * SEC_TO_NS + nsec); - if (ns_total == NULL) { - return -1; - } - PyStructSequence_SET_ITEM(v, ns_index, ns_total); - assert(!PyErr_Occurred()); - res = 0; - } - else { - PyObject *s_in_ns = NULL; - PyObject *ns_total = NULL; - PyObject *s = _PyLong_FromTime_t(sec); - PyObject *ns_fractional = PyLong_FromUnsignedLong(nsec); - if (s == NULL || ns_fractional == NULL) { - goto exit; - } - - s_in_ns = PyNumber_Multiply(s, state->billion); - if (s_in_ns == NULL) { - goto exit; - } - - ns_total = PyNumber_Add(s_in_ns, ns_fractional); - if (ns_total == NULL) { - goto exit; - } - PyStructSequence_SET_ITEM(v, ns_index, ns_total); - assert(!PyErr_Occurred()); - res = 0; - - exit: - Py_XDECREF(s); - Py_XDECREF(ns_fractional); - Py_XDECREF(s_in_ns); + PyObject *ns_total = nanosecond_timestamp(state, sec, nsec); + if (ns_total == NULL) { + return -1; } + PyStructSequence_SET_ITEM(v, ns_index, ns_total); } - return res; - #undef SEC_TO_NS + assert(!PyErr_Occurred()); + return 0; } +#undef SEC_TO_NS #ifdef MS_WINDOWS static PyObject* @@ -2775,6 +2813,14 @@ _pystat_fromstructstat(PyObject *module, STRUCT_STAT *st) #endif SET_ITEM(ST_BIRTHTIME_IDX, PyFloat_FromDouble(bsec + bnsec * 1e-9)); } +#elif defined(HAVE_STATX) + /* We were built with statx support, so stat_result.st_birthtime[_ns] + exists, but we fell back to stat because statx isn't available at + runtime. structseq members are _Py_T_OBJECT (for which NULL means None, + not AttributeError), but user programs assume st_birthtime, when + present, is not None. */ + SET_ITEM(ST_BIRTHTIME_IDX, PyFloat_FromDouble(0.0)); + SET_ITEM(ST_BIRTHTIME_NS_IDX, _PyLong_GetZero()); #elif defined(MS_WINDOWS) if (fill_time(state, v, -1, ST_BIRTHTIME_IDX, ST_BIRTHTIME_NS_IDX, st->st_birthtime, st->st_birthtime_nsec) < 0) { @@ -2801,12 +2847,78 @@ _pystat_fromstructstat(PyObject *module, STRUCT_STAT *st) error: Py_DECREF(v); return NULL; +} -#undef SET_ITEM +#ifdef HAVE_STATX +static PyObject* +_pystat_fromstructstatx(PyObject *module, struct statx *st) +{ + assert(!PyErr_Occurred()); + + _posixstate *state = get_posix_state(module); + PyObject *StatResultType = state->StatResultType; + PyObject *v = PyStructSequence_New((PyTypeObject *)StatResultType); + if (v == NULL) { + return NULL; + } + + /* Per the comment in /usr/include/linux/stat.h, even if a bit is cleared + in stx_mask, the corresponding field is "set to an appropriate + fabricated value" or cleared, never left uninitialized. */ + SET_ITEM(0, PyLong_FromLong((long)st->stx_mode)); + static_assert(sizeof(unsigned long long) >= sizeof(st->stx_ino), + "stat.st_ino is larger than unsigned long long"); + SET_ITEM(1, PyLong_FromUnsignedLongLong(st->stx_ino)); + + dev_t dev = makedev(st->stx_dev_major, st->stx_dev_minor); + SET_ITEM(2, _PyLong_FromDev(dev)); + + SET_ITEM(3, PyLong_FromLong((long)st->stx_nlink)); + SET_ITEM(4, _PyLong_FromUid(st->stx_uid)); + SET_ITEM(5, _PyLong_FromGid(st->stx_gid)); + static_assert(sizeof(long long) >= sizeof(st->stx_size), + "stat.st_size is larger than long long"); + SET_ITEM(6, PyLong_FromLongLong(st->stx_size)); + + if (fill_time(state, v, 7, 10, 13, st->stx_atime.tv_sec, + st->stx_atime.tv_nsec) < 0) { + goto error; + } + if (fill_time(state, v, 8, 11, 14, st->stx_mtime.tv_sec, + st->stx_mtime.tv_nsec) < 0) { + goto error; + } + if (fill_time(state, v, 9, 12, 15, st->stx_ctime.tv_sec, + st->stx_ctime.tv_nsec) < 0) { + goto error; + } + if (fill_time(state, v, -1, ST_BIRTHTIME_IDX, ST_BIRTHTIME_NS_IDX, + st->stx_btime.tv_sec, st->stx_btime.tv_nsec) < 0) { + goto error; + } + + SET_ITEM(ST_BLKSIZE_IDX, PyLong_FromLong((long)st->stx_blksize)); + SET_ITEM(ST_BLOCKS_IDX, PyLong_FromLong((long)st->stx_blocks)); + + dev_t rdev = makedev(st->stx_rdev_major, st->stx_rdev_minor); + SET_ITEM(ST_RDEV_IDX, _PyLong_FromDev(rdev)); + + assert(!PyErr_Occurred()); + return v; + +error: + Py_DECREF(v); + return NULL; } +#endif /* HAVE_STATX */ +#undef SET_ITEM /* POSIX methods */ +#ifdef HAVE_STATX +/* set to 1 in posixmodule_exec after a successful test call */ +static int statx_works = 0; +#endif static PyObject * posix_do_stat(PyObject *module, const char *function_name, path_t *path, @@ -2829,6 +2941,32 @@ posix_do_stat(PyObject *module, const char *function_name, path_t *path, fd_and_follow_symlinks_invalid("stat", path->fd, follow_symlinks)) return NULL; +#ifdef HAVE_STATX + struct statx stx = {}; + if (statx_works != 0) { + unsigned int mask = STATX_BASIC_STATS | STATX_BTIME; + int flags = AT_NO_AUTOMOUNT; + flags |= follow_symlinks ? 0 : AT_SYMLINK_NOFOLLOW; + + Py_BEGIN_ALLOW_THREADS + if (path->fd != -1) { + flags |= AT_EMPTY_PATH; + result = statx(path->fd, "", flags, mask, &stx); + } + else { + result = statx(dir_fd, path->narrow, flags, mask, &stx); + } + Py_END_ALLOW_THREADS + + if (result == -1) { + return path_error(path); + } + else { + return _pystat_fromstructstatx(module, &stx); + } + } +#endif /* HAVE_STATX */ + Py_BEGIN_ALLOW_THREADS if (path->fd != -1) result = FSTAT(path->fd, &st); @@ -3277,6 +3415,322 @@ os_lstat_impl(PyObject *module, path_t *path, int dir_fd) } +#ifdef HAVE_STATX +#define STATX_RESULT_CACHE_SLOTS 17 +typedef struct { + PyObject_HEAD + struct statx stx; +#if STATX_RESULT_CACHE_SLOTS > 0 + PyObject *cache[STATX_RESULT_CACHE_SLOTS]; +#endif +} statx_result; + +static PyObject * +_PyFloat_FromStatxTimestamp(struct statx_timestamp ts) { + return PyFloat_FromDouble((double)ts.tv_sec + 1e-9 * ts.tv_nsec); +} + +/* The reserved space in struct statx was originally defined as arrays of u64. + Accessing u32 members (or other future non-u64 members) directly by offset + would be a strict aliasing violation, so use memcpy. */ +#define DECLARE_UNCACHED_GET(name, type, func) \ + static PyObject * \ + statx_result_get_##name(PyObject *op, void *context) { \ + statx_result *self = (statx_result *) op; \ + uint16_t offset = (uintptr_t)context; \ + type val; \ + memcpy(&val, (void *)self + offset, sizeof(val)); \ + return func(val); \ + } + +DECLARE_UNCACHED_GET(uid, uint32_t, _PyLong_FromUid) +DECLARE_UNCACHED_GET(gid, uint32_t, _PyLong_FromGid) +DECLARE_UNCACHED_GET(sec, struct statx_timestamp, _PyFloat_FromStatxTimestamp) +/* Don't use these uncached getters directly -- define members instead. */ +DECLARE_UNCACHED_GET(u16, uint16_t, PyLong_FromUInt32) +DECLARE_UNCACHED_GET(u32, uint32_t, PyLong_FromUInt32) +DECLARE_UNCACHED_GET(u64, uint64_t, PyLong_FromUInt64) +#undef DECLARE_UNCACHED_GET + +static PyObject * +statx_result_get_nsec(PyObject *op, void *context) { + statx_result *self = (statx_result *) op; + uint16_t offset = (uintptr_t)context; + struct statx_timestamp val; + memcpy(&val, (void *)self + offset, sizeof(val)); + _posixstate *state = PyType_GetModuleState(Py_TYPE(op)); + assert(state != NULL); + return nanosecond_timestamp(state, val.tv_sec, val.tv_nsec); +} + +static PyObject * +statx_result_get_dev(PyObject *op, void *context) { + statx_result *self = (statx_result *) op; + uint16_t offset = (uintptr_t)context; + uint32_t major, minor; + memcpy(&major, (void *)self + offset, sizeof(major)); + memcpy(&minor, (void *)self + offset + sizeof(major), sizeof(minor)); + return _PyLong_FromDev(makedev(major, minor)); +} + +#define DECLARE_CACHED_GET(func) \ + static PyObject * \ + func##_cached(PyObject *op, void *context) { \ + statx_result *self = (statx_result *) op; \ + uint16_t slot = (uintptr_t)context >> 16; \ + PyObject *cached = FT_ATOMIC_LOAD_PTR(self->cache[slot]); \ + if (cached != NULL) { \ + return Py_NewRef(cached); \ + } \ + Py_BEGIN_CRITICAL_SECTION(self); \ + cached = self->cache[slot]; \ + if (cached == NULL) { \ + cached = func(op, context); \ + if (cached != NULL) { \ + FT_ATOMIC_STORE_PTR(self->cache[slot], cached); \ + } \ + } \ + Py_END_CRITICAL_SECTION(); \ + return Py_XNewRef(cached); \ + } + +DECLARE_CACHED_GET(statx_result_get_uid) +DECLARE_CACHED_GET(statx_result_get_gid) +DECLARE_CACHED_GET(statx_result_get_sec) +DECLARE_CACHED_GET(statx_result_get_u16) +DECLARE_CACHED_GET(statx_result_get_u32) +DECLARE_CACHED_GET(statx_result_get_u64) +DECLARE_CACHED_GET(statx_result_get_nsec) +DECLARE_CACHED_GET(statx_result_get_dev) +#undef DECLARE_CACHED_GET + +/* The low 16 bits of the context pointer are the offset from the start of + statx_result to the struct statx member; the next 16 bits are the cache + index or -1. The high 32 bits (if present) are unused. */ +#define OFFSET_CACHE_CONTEXT(offset, index) \ + (void *)((offsetof(statx_result, stx) + offset) | ((size_t)(index) << 16)) +#define OFFSET_CONTEXT(offset) OFFSET_CACHE_CONTEXT(offset, (uint16_t)-1) +#define MEMBER_CACHE_CONTEXT(name, index) \ + OFFSET_CACHE_CONTEXT(offsetof(struct statx, stx_##name), index) +#define MEMBER_CONTEXT(name) MEMBER_CACHE_CONTEXT(name, (uint16_t)-1) + +#define G(attr, type, doc, context) \ + {attr, statx_result_get_##type, NULL, PyDoc_STR(doc), context} +#define GMC(attr, type, member, index, doc) \ + G(#attr, type##_cached, doc, MEMBER_CACHE_CONTEXT(member, index)) +#define GM(attr, type, member, doc) \ + G(#attr, type, doc, MEMBER_CONTEXT(member)) +#define GOC(attr, type, offset, index, doc) \ + G(#attr, type##_cached, doc, OFFSET_CACHE_CONTEXT(offset, index)) +#define GO(attr, type, offset, doc) \ + G(#attr, type, doc, OFFSET_CONTEXT(offset)) + +static PyGetSetDef statx_result_getset[] = { + GMC(stx_mask, u32, mask, 0, "member validity mask"), + GMC(stx_attributes, u64, attributes, 1, "Linux inode attribute bits"), + GMC(st_uid, uid, uid, 2, "user ID of owner"), + GMC(st_gid, gid, gid, 3, "group ID of owner"), + GMC(st_mode, u16, mode, 4, "protection bits"), + GMC(st_ino, u64, ino, 5, "inode"), + GMC(st_size, u64, size, 6, "total size, in bytes"), + GMC(stx_attributes_mask, u64, attributes_mask, 7, + "Linux inode attribute bits supported for this file"), + GMC(st_atime, sec, atime, 8, "time of last access"), + GMC(st_atime_ns, nsec, atime, 9, "time of last access in nanoseconds"), + GMC(st_birthtime, sec, btime, 10, "time of creation"), + GMC(st_birthtime_ns, nsec, btime, 11, "time of creation in nanoseconds"), + GMC(st_ctime, sec, ctime, 12, "time of last change"), + GMC(st_ctime_ns, nsec, ctime, 13, "time of last change in nanoseconds"), + GMC(st_mtime, sec, mtime, 14, "time of last modification"), + GMC(st_mtime_ns, nsec, mtime, 15, + "time of last modification in nanoseconds"), + GM(st_rdev, dev, rdev_major, "device type (if inode device)"), + GMC(st_dev, dev, dev_major, 16, "device"), + {NULL}, +}; + +#undef GO +#undef GOC +#undef GM +#undef GMC +#undef G +#undef MEMBER_CONTEXT +#undef MEMBER_CACHE_CONTEXT +#undef OFFSET_CONTEXT +#undef OFFSET_CACHE_CONTEXT + +#define MO(attr, type, offset, doc) \ + {#attr, type, offsetof(statx_result, stx) + offset, Py_READONLY, PyDoc_STR(doc)} +#define MM(attr, type, member, doc) \ + MO(attr, type, offsetof(struct statx, stx_##member), doc) + +static PyMemberDef statx_result_members[] = { + MM(st_blksize, Py_T_UINT, blksize, "blocksize for filesystem I/O"), + MM(st_nlink, Py_T_UINT, nlink, "number of hard links"), + MM(st_blocks, Py_T_ULONGLONG, blocks, "number of blocks allocated"), + MO(stx_mnt_id, Py_T_ULONGLONG, 144, "mount ID"), + MO(stx_dio_mem_align, Py_T_UINT, 152, + "direct I/O memory buffer alignment"), + MO(stx_dio_offset_align, Py_T_UINT, 156, + "direct I/O file offset alignment"), + MO(stx_subvol, Py_T_ULONGLONG, 160, "subvolume ID"), + MO(stx_atomic_write_unit_min, Py_T_UINT, 168, + "minimum size for direct I/O with torn-write protection"), + MO(stx_atomic_write_unit_max, Py_T_UINT, 172, + "maximum size for direct I/O with torn-write protection"), + MO(stx_atomic_write_segments_max, Py_T_UINT, 176, + "maximum iovecs for direct I/O with torn-write protection"), + MO(stx_dio_read_offset_align, Py_T_UINT, 180, + "direct I/O file offset alignment for reads"), + MO(stx_atomic_write_unit_max_opt, Py_T_UINT, 184, + "maximum optimized size for direct I/O with torn-write protection"), + {NULL}, +}; + +#undef MM +#undef MO + +static int +statx_result_traverse(PyObject *self, visitproc visit, void *arg) { + Py_VISIT(Py_TYPE(self)); + /* self->cache only points to longs and floats */ + return 0; +} + +static void +statx_result_dealloc(PyObject *op) { + statx_result *self = (statx_result *) op; + PyTypeObject *tp = Py_TYPE(self); + PyObject_GC_UnTrack(self); +#if STATX_RESULT_CACHE_SLOTS > 0 + for (int i = 0; i < STATX_RESULT_CACHE_SLOTS; ++i) { + Py_CLEAR(self->cache[i]); + } +#endif + tp->tp_free(self); + Py_DECREF(tp); +} + +static PyType_Slot statx_result_slots[] = { + {Py_tp_traverse, statx_result_traverse}, + {Py_tp_dealloc, statx_result_dealloc}, + {Py_tp_getset, statx_result_getset}, + {Py_tp_members, statx_result_members}, + {0, NULL}, +}; + +static PyType_Spec statx_result_spec = { + .name = "statx_result", + .basicsize = sizeof(statx_result), + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HEAPTYPE | Py_TPFLAGS_HAVE_GC | + Py_TPFLAGS_IMMUTABLETYPE | Py_TPFLAGS_DISALLOW_INSTANTIATION, + .slots = statx_result_slots, +}; + +static int +optional_bool_converter(PyObject *arg, void *addr) { + int value; + if (arg == Py_None) { + value = -1; + } + else { + value = Py_IsTrue(arg); + if (value < 0) { + return 0; + } + } + *((int *)addr) = value; + return 1; +} + +/*[python input] +class optional_bool_converter(CConverter): + type = 'int' + converter = 'optional_bool_converter' +[python start generated code]*/ +/*[python end generated code: output=da39a3ee5e6b4b0d input=47de85b300eeb19e]*/ + +/*[clinic input] + +os.statx + + path : path_t(allow_fd=True) + Path to be examined; can be string, bytes, a path-like object or + open-file-descriptor int. + + mask: unsigned_int(bitwise=True) + A bitmask of STATX_* constants defining the requested information. + + * + + dir_fd : dir_fd = None + If not None, it should be a file descriptor open to a directory, + and path should be a relative string; path will then be relative to + that directory. + + follow_symlinks: bool = True + If False, and the last element of the path is a symbolic link, + statx will examine the symbolic link itself instead of the file + the link points to. + + sync: optional_bool(c_default='-1') = None + If True, statx will return up-to-date values, even if doing so is + expensive. If False, statx will return cached values if possible. + If None, statx lets the operating system decide. + +Perform a statx system call on the given path. + +It's an error to use dir_fd or follow_symlinks when specifying path as + an open file descriptor. + +[clinic start generated code]*/ + +static PyObject * +os_statx_impl(PyObject *module, path_t *path, unsigned int mask, int dir_fd, + int follow_symlinks, int sync) +/*[clinic end generated code: output=fe385235585f3d07 input=148c4fce440ca53a]*/ +{ + if (path_and_dir_fd_invalid("statx", path, dir_fd) || + dir_fd_and_fd_invalid("statx", dir_fd, path->fd) || + fd_and_follow_symlinks_invalid("statx", path->fd, follow_symlinks)) + return NULL; + + int result; + /* Future bits may refer to members beyond the current size of struct + statx, so we need to mask them off to prevent memory corruption. */ + mask &= _Py_STATX_KNOWN; + int flags = AT_NO_AUTOMOUNT | (follow_symlinks ? 0 : AT_SYMLINK_NOFOLLOW); + if (sync != -1) { + flags |= sync ? AT_STATX_FORCE_SYNC : AT_STATX_DONT_SYNC; + } + + _posixstate *state = get_posix_state(module); + PyTypeObject *tp = (PyTypeObject *)state->StatxResultType; + statx_result *v = (statx_result *)tp->tp_alloc(tp, 0); + if (v == NULL) { + return NULL; + } + + Py_BEGIN_ALLOW_THREADS + if (path->fd != -1) { + result = statx(path->fd, "", flags | AT_EMPTY_PATH, mask, &v->stx); + } + else { + result = statx(dir_fd, path->narrow, flags, mask, &v->stx); + } + Py_END_ALLOW_THREADS + + if (result != 0) { + Py_DECREF(v); + return path_error(path); + } + + assert(!PyErr_Occurred()); + return (PyObject *)v; +} +#endif + + /*[clinic input] os.access -> bool @@ -17040,6 +17494,7 @@ os__emscripten_log_impl(PyObject *module, const char *arg) static PyMethodDef posix_methods[] = { OS_STAT_METHODDEF + OS_STATX_METHODDEF OS_ACCESS_METHODDEF OS_TTYNAME_METHODDEF OS_CHDIR_METHODDEF @@ -17878,6 +18333,30 @@ all_ins(PyObject *m) #endif #endif /* HAVE_EVENTFD && EFD_CLOEXEC */ +#ifdef HAVE_STATX + if (PyModule_AddIntMacro(m, STATX_TYPE)) return -1; + if (PyModule_AddIntMacro(m, STATX_MODE)) return -1; + if (PyModule_AddIntMacro(m, STATX_NLINK)) return -1; + if (PyModule_AddIntMacro(m, STATX_UID)) return -1; + if (PyModule_AddIntMacro(m, STATX_GID)) return -1; + if (PyModule_AddIntMacro(m, STATX_ATIME)) return -1; + if (PyModule_AddIntMacro(m, STATX_MTIME)) return -1; + if (PyModule_AddIntMacro(m, STATX_CTIME)) return -1; + if (PyModule_AddIntMacro(m, STATX_INO)) return -1; + if (PyModule_AddIntMacro(m, STATX_SIZE)) return -1; + if (PyModule_AddIntMacro(m, STATX_BLOCKS)) return -1; + if (PyModule_AddIntMacro(m, STATX_BASIC_STATS)) return -1; + if (PyModule_AddIntMacro(m, STATX_BTIME)) return -1; + if (PyModule_AddIntMacro(m, STATX_MNT_ID)) return -1; + if (PyModule_AddIntMacro(m, STATX_DIOALIGN)) return -1; + if (PyModule_AddIntMacro(m, STATX_MNT_ID_UNIQUE)) return -1; + if (PyModule_AddIntMacro(m, STATX_SUBVOL)) return -1; + if (PyModule_AddIntMacro(m, STATX_WRITE_ATOMIC)) return -1; + if (PyModule_AddIntMacro(m, STATX_DIO_READ_ALIGN)) return -1; + /* STATX_ALL intentionally omitted because it is deprecated */ + /* STATX_ATTR_* constants are in the stat module */ +#endif /* HAVE_STATX */ + #if defined(__APPLE__) if (PyModule_AddIntConstant(m, "_COPYFILE_DATA", COPYFILE_DATA)) return -1; if (PyModule_AddIntConstant(m, "_COPYFILE_STAT", COPYFILE_STAT)) return -1; @@ -18149,6 +18628,53 @@ posixmodule_exec(PyObject *m) } #endif +#ifdef HAVE_STATX +#ifndef NDEBUG + /* Verify the cache slot assignment. */ + size_t used_slots = 0; + for (int i = 0; statx_result_getset[i].name != NULL; ++i) { + uint16_t slot = (uintptr_t)statx_result_getset[i].closure >> 16; + if (slot != (uint16_t)-1) { + assert(slot < STATX_RESULT_CACHE_SLOTS); + assert((used_slots & (1 << slot)) == 0); + used_slots |= 1 << slot; + } + } + assert(used_slots == (1 << STATX_RESULT_CACHE_SLOTS) - 1); +#endif + + /* statx is available unless: + - the weakly-linked statx wrapper function is not available (old libc) + - the wrapper function fails with ENOSYS (libc built without fallback + running on an old kernel) + - the wrapper function fails with EINVAL on sync flags (glibc's + emulation of statx via stat fails in this way) + In the last case, it is safe to use statx to implement os.stat, but + because glibc unconditionally makes the statx syscall before falling + back to fstatat, doing so is measurably slower. */ + struct statx stx; + if (statx == NULL + || (statx(-1, "/", AT_STATX_DONT_SYNC, 0, &stx) == -1 + && (errno == EINVAL || errno == ENOSYS))) { + PyObject* dct = PyModule_GetDict(m); + if (dct == NULL) { + return -1; + } + if (PyDict_PopString(dct, "statx", NULL) < 0) { + return -1; + } + } + else { + statx_works = 1; + + statx_result_spec.name = "os.statx_result"; + state->StatxResultType = PyType_FromModuleAndSpec(m, &statx_result_spec, NULL); + if (PyModule_AddObjectRef(m, "statx_result", state->StatxResultType) < 0) { + return -1; + } + } +#endif + /* Initialize environ dictionary */ if (PyModule_Add(m, "environ", convertenviron()) != 0) { return -1; diff --git a/Tools/c-analyzer/cpython/ignored.tsv b/Tools/c-analyzer/cpython/ignored.tsv index 4fdb7b3cd1abf2..f335a42c0323b0 100644 --- a/Tools/c-analyzer/cpython/ignored.tsv +++ b/Tools/c-analyzer/cpython/ignored.tsv @@ -21,6 +21,7 @@ Python/fileutils.c set_inheritable ioctl_works - # (set lazily, *after* first init) # XXX Is this thread-safe? Modules/posixmodule.c os_dup2_impl dup3_works - +Modules/posixmodule.c posix_do_stat statx_works - ## guards around resource init Python/thread_pthread.h PyThread__init_thread lib_initialized - diff --git a/configure b/configure index 7dd333690a3a2c..03cdd5a37679a7 100755 --- a/configure +++ b/configure @@ -20175,6 +20175,12 @@ if test "x$ac_cv_func_splice" = xyes then : printf "%s\n" "#define HAVE_SPLICE 1" >>confdefs.h +fi +ac_fn_c_check_func "$LINENO" "statx" "ac_cv_func_statx" +if test "x$ac_cv_func_statx" = xyes +then : + printf "%s\n" "#define HAVE_STATX 1" >>confdefs.h + fi ac_fn_c_check_func "$LINENO" "strftime" "ac_cv_func_strftime" if test "x$ac_cv_func_strftime" = xyes diff --git a/configure.ac b/configure.ac index 8fc8f1eced20cf..a5d2de5e5da895 100644 --- a/configure.ac +++ b/configure.ac @@ -5252,7 +5252,7 @@ AC_CHECK_FUNCS([ \ setitimer setlocale setpgid setpgrp setpriority setregid setresgid \ setresuid setreuid setsid setuid setvbuf shutdown sigaction sigaltstack \ sigfillset siginterrupt sigpending sigrelse sigtimedwait sigwait \ - sigwaitinfo snprintf splice strftime strlcpy strsignal symlinkat sync \ + sigwaitinfo snprintf splice statx strftime strlcpy strsignal symlinkat sync \ sysconf tcgetpgrp tcsetpgrp tempnam timegm times tmpfile \ tmpnam tmpnam_r truncate ttyname_r umask uname unlinkat unlockpt utimensat utimes vfork \ wait wait3 wait4 waitid waitpid wcscoll wcsftime wcsxfrm wmemcmp writev \ diff --git a/pyconfig.h.in b/pyconfig.h.in index 0d6ad4465c0e93..3c5cb53ea3ea7d 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1279,6 +1279,9 @@ /* Define to 1 if you have the 'statvfs' function. */ #undef HAVE_STATVFS +/* Define to 1 if you have the 'statx' function. */ +#undef HAVE_STATX + /* Define if you have struct stat.st_mtim.tv_nsec */ #undef HAVE_STAT_TV_NSEC