Skip to content

Conversation

jbosboom
Copy link
Contributor

@jbosboom jbosboom commented Jul 6, 2025

I've posted some questions in gh-83714. For now, this PR should be considered earnest effort (in the sense of earnest money). It will change based on the answers to the questions in the issue. It needs tests for the new features and the fallback that are not obvious to implement. I need guidance as to when it should strictly conform to PEP 7 and when it should match the surrounding code style (especially with regards to brace placement). But it should demonstrate I'm willing and able.

The libc statx wrapper is required at build time, but weakly linked. There is no configure test; we just look for STATX_BASIC_STATS being defined by sys/stat.h. posix_do_stat will fall back to calling stat if statx is not available at runtime due to an old libc or an old kernel. The fallback variable statx_works is based on dup3_works, for the thread-safety of which see @ericsnowcurrently's comment. The values are (to me) nonobvious: -1 means statx has never failed, 0 means its first failure was ENOSYS and 1 means its first failure was something other than ENOSYS.

This PR is not directly based on gh-19125, but was greatly informed by it, and I'd be happy to give Co-authored-by and/or blurb credit to @ntninja.


📚 Documentation preview 📚: https://cpython-previews--136334.org.readthedocs.build/

The libc statx wrapper is required at build time, but weakly linked, and
posix_do_stat will fall back to calling stat if statx is not available
at runtime due to an old libc or an old kernel.
@python-cla-bot
Copy link

python-cla-bot bot commented Jul 6, 2025

All commit authors signed the Contributor License Agreement.

CLA signed

@bedevere-app
Copy link

bedevere-app bot commented Jul 6, 2025

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@StanFromIreland
Copy link
Member

Please update the title to use the gh issue number.

@jbosboom jbosboom changed the title bpo-39533: Use statx on Linux 4.11 and later in os.stat gh-83714: Use statx on Linux 4.11 and later in os.stat Jul 6, 2025
@bedevere-app
Copy link

bedevere-app bot commented Jul 8, 2025

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Copy link
Contributor

@cmaloney cmaloney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some docs suggestions / CPython sphinx has special tags for a couple of the things; haven't looked at main implementation a lot yet

@gpshead
Copy link
Member

gpshead commented Sep 7, 2025

!buildbot .Android.

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gpshead for commit b0e8276 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136334%2Fmerge

The command will test the builders whose names match following regular expression: .*Android.*

The builders matched are:

  • aarch64 Android PR
  • AMD64 Android PR

@mhsmith
Copy link
Member

mhsmith commented Sep 7, 2025

FYI, we recently added Android to the GitHub Actions CI, so there's no longer much need to invoke the buildbots manually. The exception is if a PR has something which might be specific to Android aarch64, because GitHub Actions can't run the tests on that platform yet, although it will still perform a complete build.

@jbosboom
Copy link
Contributor Author

jbosboom commented Sep 9, 2025

Looking at the failures, the Android buildbots didn't run the most up-to-date code from this PR. Looking at the "Build Properties" tab, the "revision" is "b0e827681f45588ce67cd0924b3c320bb64351ff" (the latest), but "got_revision" is "a387390693c4f4ff793717a91a5137b7f608f9e4", a merge commit attributed to me but that I don't remember making of d53650a (the commit just before I added the configure check) with main.

@mhsmith
Copy link
Member

mhsmith commented Sep 9, 2025

Both the buildbots and GitHub Actions don't actually test the head of the PR branch, but a speculative commit showing what you would get if you merged the PR into main. Because of a merge conflict, it's not possible to merge the PR into main at the moment. It looks like GitHub Actions deals with this by not running any tests at all, while the buildbots test the last commit before the conflict.

@vstinner
Copy link
Member

vstinner commented Sep 9, 2025

I dislike this approach: modifying Python os.stat() to use statx. I would prefer to expose C statx() as Python os.statx() instead.

… raw kwarg

We no longer return None for invalid attributes for the benefit of
drop-in replacement of os.stat when not all basic stats are required or
sync=False is acceptable.  Such code was already accepting faked values
from os.stat, so can also accept faked values from os.statx.  Remove the
raw kwarg as it would have no effect.
These members were removed from os.stat_result in a previous commit.
@jbosboom
Copy link
Contributor Author

I've marked this ready for review. @vstinner I'd also appreciate a review from you; GitHub isn't showing me the interface to formally request one.

This PR has changed quite a bit from the first commit, and as it did so, I wrote quite a few words in the issue #83714 that may be informative (probably best to read from the bottom up).

Copy link
Contributor

@cmaloney cmaloney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the pieces I have reasonable knowledge of; don't have enough experience with macros and defining new structs to go in more detail through those pieces. For first contributions to CPython really impressive in getting things to the CPython style (docs, C code style, etc)

High level:

  1. This PR is large for CPython; Smaller chunks will make it easier to review and land; My thought would be adding one part adding os.statx; second updating os.stat to use statx when available
  2. The os.stat change will need (on newer + older kernels). Just adding statx doesn't need as much performance validation
    1. pyperf microbenchmark that shows stat didn't get slower
    2. Validate that Python startup time does not change significantly (I believe github.com/python/pyperformance has two benchmarks around that)
  3. statx will probably make sense to add to pathlib.Path for easy access like stat is (but that should probably be another PR with the same issue)

With the thoroughness you're doing definitely feels like will get to the confidence to change os.stat to point to os.statx. Again; really impressive for a first CPython contributions.


.. function:: statx(path, mask, *, dir_fd=None, follow_symlinks=True, sync=None)

Get the status of a file or file descriptor by performing a :c:func:`statx`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be able to make the missing c func / struct pieces stop erroring by adding to:

nitpick_ignore = [
(in general try with new things to not add new warnings; are people working on resolving existing sphinx warnings)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Get the status of a file or file descriptor by performing a :c:func:`statx`
Get the status of a file or file descriptor by performing a :c:func:`!statx`

This will also work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running make html and make check as directed by the devguide, I don't see any warnings.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting all the warnings out of the docs build is difficult (there are many warnings in non-changed code which don't matter for PRs; but new ones do); the CI does check for new warnings and attaches them to PRs. I can see some here on the Github UI:

image

actually retrieved (which may differ from *mask*).

The optional parameter *sync* controls the freshness of the returned
information. ``sync=True`` requests that the kernel return up-to-date
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should document the value of None which is the default; may make sense to just make it always True/False for now

self.assertAlmostEqual(floaty, nanosecondy, delta=2)

# Ensure both birthtime and birthtime_ns roughly agree, if present
time_attributes = 'st_atime st_mtime st_ctime'.split()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: why not just make a three element list here? (I see the old code did this, but seems reasonable to just update)

def check_statx_attributes(self, fname):
maximal_mask = 0
for name in dir(os):
if name.startswith('STATX_'):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels like there should be a better way to group / gather these (maybe have them all live as constants in a Python class that acts as a namespace?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar constants for other os functions are also dumped in the module namespace. That doesn't mean we can't do something different, but we'd be setting somewhat of a precedent.

(It would be nice to have an IntFlag enum class that can report which members are valid given a bitmask, but adding an import dependency to os doesn't seem like a good idea.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly strongly against "find everything that starts with STATX_ in os" as that will just get slower as os gets larger.

Can be just a manually maintained list for now; would be nice to do something more namespace like (but also don't want to add a new CPython standard here).

for sync in (False, True):
with self.subTest(sync=sync):
os.statx(self.fname, os.STATX_BASIC_STATS, sync=sync)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will need more test cases for different ways of constructing/using statx (path_fd, dir_fd, PathLike)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the dir_fd tests go in here in test_os.py or in TestPosixDirFd in test_posix.py?

Should all the statx tests be in test_posix.py? The comment at the top of test_os.py says it's for "a few functions which have been determined to be more portable than they had been thought to be", and statx is not.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in test_os works; don't need to test that "do the dir_fd helpers use handle every case right", want to just make sure "if someone deleted the dir_fd handling code; a test would fail"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_posix seems to be around https://docs.python.org/3/library/posix.html; I don't have experience / knowledge around that so not sure

int flags = AT_NO_AUTOMOUNT;
flags |= follow_symlinks ? 0 : AT_SYMLINK_NOFOLLOW;

Py_BEGIN_ALLOW_THREADS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I don't see a reason not to start allowing threads a bit earlier; aren't touching the interpreter in the lines just before this

typedef struct {
PyObject_HEAD
struct statx stx;
#if STATX_RESULT_CACHE_SLOTS > 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this always > 0? (the constant 17)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's defined this way to allow reviewers and future maintainers to more easily experiment with not having a cache (because member arrays can't be empty in C). On reflection, it doesn't help much, because the no-cache configuration would change most of the getset defs to members for faster access, and after that, commenting out the member is trivial.

int result;
/* Future bits may refer to members beyond the current size of struct
statx, so we need to mask them off to prevent memory corruption. */
mask &= _Py_STATX_KNOWN;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this error if the mask was changed by this code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, maybe. We'd only adjust the mask for Python code that specified the mask numerically instead of using the os.STATX_* constants. That's probably only code that wants to use members Python 3.16 knows about but Python 3.15 doesn't. os.statx_result doesn't provide Python with an interface to access members the interpreter doesn't know about, so while code could use the numeric value of STATX_FOO, accessing stx_foo would raise AttributeError. And that's how current code copes with os.stat_result's members varying across versions. But I'm open to arguments that failing fast is better and version-crossing code should instead change their mask based on the constants available.

#endif /* HAVE_EVENTFD && EFD_CLOEXEC */

#ifdef HAVE_STATX
if (PyModule_AddIntMacro(m, STATX_TYPE)) return -1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Any reason not to alphabetize this list?

I like having either a comment why not alphabetical or to alphabetize for easier later code reading "is this member in the list" when trying to add more

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order matches the definitions in /usr/include/linux/stat.h. When new bits are defined, they're added at the end (there are no gaps in the definitions). Alphabetizing would make things harder, not easier.

@jbosboom
Copy link
Contributor Author

With the thoroughness you're doing definitely feels like will get to the confidence to change os.stat to point to os.statx.

I don't recommend this. os.statx_result is only mostly compatible with os.stat_result. In particular, os.statx_result is not a tuple, so it can't be indexed, iterated, unpacked, hashed, or pickled. os.stat_result being a tuple and having tuple-like behaviors is (AFAICT) a historical accident, but it isn't deprecated.

@cmaloney
Copy link
Contributor

👍 / agree with you, meant to say os.stat use the system call statx under the hood (not that the two functions would become the same).

glibc doesn't remember that statx previously failed with ENOSYS, so on
old kernels, glibc's emulation of statx via stat has the overhead of a
failed syscall.  Only call the statx wrapper under the same conditions
that we make os.statx available.

statx_works was always a global C variable, but now its scope is the
file instead of just posix_do_stat.  Whether statx works is a
process-global property, so using a global C variable is appropriate.
Previously access to statx_works was unsynchronized, following the
pattern of dup3_works.  Now we rely on posixmodule_exec happening-before
any thread can call os.stat.
Also two minor fixes to statx tests in test_os.py.
@jbosboom
Copy link
Contributor Author

Some benchmark numbers comparing os.stat(".") main at cf9ef73 vs this PR at c84f783 merged with main. Configured with --enable-optimizations --with-lto, running on a Ryzen 5950x, turbo boost disabled on highest-CPPC-perf core 0/16, Arch Linux kernel 6.16.7-arch1-1, glibc 2.42+r17+gd7274d718e6f-1, btrfs filesystem. Histograms in Gist.

main 1.93 us +- 0.01 us
PR nobtime 1.94 us +- 0.01 us
PR 1.97 us +- 0.01 us

The 'PR nobtime' row replaces the last fill_time call in _pystat_fromstructstatx with setting ST_BIRTHTIME_IDX and ST_BIRTHTIME_NS_IDX to None. So while os.stat is slower with this PR, most of the slowdown is due to creating an extra Python float and int object to fill the btime members in the os.stat_result.


I don't have a VM running a pre-statx kernel handy, but I did find the enosys utility from util-linux, which allows running under a seccomp filter that fails a syscall with ENOSYS. That will take the same code path as running on a kernel without statx, but doesn't necessarily have the same performance characteristics. It found a performance problem anyway: glibc's statx wrapper function doesn't remember if statx failed with ENOSYS, so it makes the statx system call every time before falling back to fstatat. That's why this PR has changed to set statx_works based on the test call in posixmodule_exec.

The following numbers are from under enosys after the change. Now that we don't rely on glibc falling back, os.stat doesn't try statx at all, so for our purposes this might be close to running on an old kernel with a new libc. (Also, based on this Zig issue there are at least some people running on a new kernel with a statx-blocking seccomp filter, so this case is relevant to them.)

mean median
main 2.01 us +- 0.04 us 2.01 us +- 0.01 us
PR 2.03 us +- 0.02 us 2.03 us +- 0.01 us

We're maybe slightly slower, and this time not due to an extra float and int, but if you look at the histograms in the gist, there are more outliers here. It's hard to conclude much.


pyperformance startup benchmark, main vs this PR:

### python_startup ###
Mean +- std dev: 12.7 ms +- 0.1 ms -> 12.8 ms +- 0.2 ms: 1.00x slower
Not significant

### python_startup_no_site ###
Mean +- std dev: 8.48 ms +- 0.09 ms -> 8.54 ms +- 0.07 ms: 1.01x slower
Not significant

main vs PR nobtime:

### python_startup ###
Mean +- std dev: 12.7 ms +- 0.1 ms -> 12.7 ms +- 0.3 ms: 1.00x faster
Not significant

### python_startup_no_site ###
Mean +- std dev: 8.48 ms +- 0.09 ms -> 8.45 ms +- 0.21 ms: 1.00x faster
Not significant

PR nobtime vs PR:

### python_startup ###
Mean +- std dev: 12.7 ms +- 0.3 ms -> 12.8 ms +- 0.2 ms: 1.01x slower
Not significant

### python_startup_no_site ###
Mean +- std dev: 8.45 ms +- 0.21 ms -> 8.54 ms +- 0.07 ms: 1.01x slower
Not significant

This PR doesn't change any of the code in fileutils.c to use statx, so if we're slower, it's because of Python os.stat calls. I haven't actually profiled to count how many there are per startup.

main vs this PR running under enosys (I'm not sure if enosys is doing anything here):

### python_startup ###
Mean +- std dev: 12.7 ms +- 0.2 ms -> 12.8 ms +- 0.2 ms: 1.00x slower
Not significant

### python_startup_no_site ###
Mean +- std dev: 8.51 ms +- 0.06 ms -> 8.55 ms +- 0.04 ms: 1.00x slower
Not significant

@vstinner
Copy link
Member

As I wrote previously, I would prefer to leave os.stat() unchanged: keep it as a thin wrapper to the C stat() function. I dislike the idea of calling statx() which depends on the kernel and glibc version and might behave differently that stat(). I don't buy the argument of providing st_birthtime by default in os.stat(), most users don't need it (it doesn't exist currently).

I support the idea of adding a new os.statx() function which gives control on which attributes are retrieved. I suppose that the result of this function can be a types.SimpleNamespace object (see _PyNamespace_New() in C).

I suggest creating a new PR which only adds a new os.statx() function, I will review and support it :-)

@jbosboom
Copy link
Contributor Author

As I wrote previously, I would prefer to leave os.stat() unchanged: keep it as a thin wrapper to the C stat() function. I dislike the idea of calling statx() which depends on the kernel and glibc version and might behave differently that stat().

Hmm. In 2024 a user asked for an update on the issue and you replied that ntninja's PR needed to be updated. That PR changed os.stat to call statx unconditionally, with no fallback for older kernels. What changed your mind?

I don't buy the argument of providing st_birthtime by default in os.stat(), most users don't need it (it doesn't exist currently).

That's a non sequitur. st_birthtime doesn't currently exist on Linux because Linux didn't provide a syscall to get btime until statx, and then when users asked for st_birthtime -- that's what the linked issue asked for -- no one finished the work to hook up statx in Python. It wasn't considered and declined because "most users don't need it".

The reason I am pushing so hard for st_birthtime from os.stat is because it is also available on Windows. If we make it available on Linux, that code starts working on Linux automatically, and new Linux code will also work on Windows. If getting btime requires a Linux-specific function, code targeting one OS will have to be changed to also support the other.

I support the idea of adding a new os.statx() function which gives control on which attributes are retrieved. I suppose that the result of this function can be a types.SimpleNamespace object (see _PyNamespace_New() in C).

And this PR does in fact add a os.statx function, though it returns a custom object type using getset and member descriptors. I would be (pleasantly) surprised if SimpleNamespace has competitive performance -- grepping for _PyNamespace_New, none of the uses are on fast paths -- but sure, I'll test it.

@gpshead
Copy link
Member

gpshead commented Sep 18, 2025

Victor and I synced on the above at our ongoing core team sprint yesterday, I'm in agreement: Lets start with only os.statx first, as that is most in keeping with the philosophy of the os module as most often being "just" a simple wrapper (at least for POSIX APIs) around the underlying single libc interfaced system call.

While that could be done in this PR you may find it cleaner for review purposes to make a new PR focused on just that.

this PR does in fact add a os.statx function, though it returns a custom object type using getset and member descriptors. I would be (pleasantly) surprised if SimpleNamespace has competitive performance -- grepping for _PyNamespace_New, none of the uses are on fast paths -- but sure, I'll test it.

I really appreciate that you're focusing on the performance of this. I'm interested to see what you find!


Long term, we envision there becoming a higher level API than the os module as a path stat interface. It could use os.stat, os.statx, and other APIs under the hood. But os.stat itself should not become such an API.

I realize this is proposing that we ultimately create a new higher level API for stat. But things like .st_birthtime are not currently widely used (even if due to our own irony of simply being unavailabile from Python) so I don't worry about the nearer term need for platform conditional code in order to get something of that form today. Those are already necessary to handle code to handle Python versions that lack the feature on a given platform.

We should track a new higher level API with its own issue. That may wind up becoming a PEP-style discussion as it'd be a new high level API - to be determined.

@vstinner
Copy link
Member

Long term, we envision there becoming a higher level API than the os module as a path stat interface. It could use os.stat, os.statx, and other APIs under the hood. But os.stat itself should not become such an API.

pathlib can be such higher level API.

@jbosboom
Copy link
Contributor Author

@gpshead @vstinner While I agree pathlib should provide a nicer interface for stat, you are more optimistic than I am as to how soon that nicer interface will arrive. I guess as core team members you have more control over when it happens. :)

I created a new PR #139178.

@jbosboom jbosboom closed this Sep 20, 2025
@cmaloney
Copy link
Contributor

Pathlib updates for stat attributes are currently being discussed -- https://discuss.python.org/t/expose-file-size-mode-etc-from-pathlib-path-info/103828 questions/criticism/ideas welcome :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants