Performance regression of pathlib.Path hashing

[csernazs]

Bug report

Bug description:

Converting a list of pathlib.Path objects to a set() takes significantly more time in python 3.12 and beyond compared to 3.11.

The below code is a minimal reproduction:

from pathlib import Path
import time
import sys

print(sys.version)

# make 6000 paths as a string
paths_str = ["/tmp/foo/bar/{i}" for i in range(1000, 7000)]

t = time.perf_counter()
# convert it to a list of Path objects
paths_path = [Path(path) for path in paths_str]

elapsed = time.perf_counter() - t
print(f"List construct: {int(elapsed * 1000)} msec")

# convert list to set
t = time.perf_counter()
set(paths_path)
elapsed = time.perf_counter() - t
print(f"Set conversion: {int(elapsed * 1000)} msec")


# extra step: convert list to set (demonstrate caching makes it faster)
t = time.perf_counter()
set(paths_path)
elapsed = time.perf_counter() - t
print(f"Set conversion (cached): {int(elapsed * 1000)} msec")

It shows with 3.12:

3.12.11 (main, Jun  3 2025, 15:41:47) [GCC 14.3.0]
List construct: 5 msec
Set conversion: 28 msec
Set conversion (cached): 1 msec

With 3.14rc1 (similar):

3.14.0rc1 (main, Jul 22 2025, 16:42:44) [GCC 14.3.0]
List construct: 4 msec
Set conversion: 26 msec
Set conversion (cached): 1 msec

But with python 3.11 (faster):

3.11.13 (main, Jun  3 2025, 18:38:25) [GCC 14.3.0]
List construct: 12 msec
Set conversion: 8 msec
Set conversion (cached): 1 msec

I also found that not only hash() got worse but str() as well - if you replace the set() call with a [str(p) for p in paths_path].

I found a possible candidate for this issue, but that's more just a theory, I could not test it:
a68e585

Could you please check what causes this?
I think newer python versions should not make the code slower, pathlib.Path is a great object (much superior than str) and now using lots of paths with a set or dict (which are also powerful data structures) kills the performance.

If you need any help, let me know.

Thank you,
Zsolt

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-138407: avoid useless join operation when calculate the hash vaule for pathlib.Path #138645

[csernazs]

I have ran git bisect on cpython between 3.11 and 3.12 and it showed this commit:

11c3020

With manual testing I can also confirm that with the above commit it is slow, with the parent (2a72125) it is fast.

Zsolt

[StanFromIreland]

cc @barneygale

[LightningFarter]

I ran your code, fixed the formatted string, and scaled up the number of files. I reproduced your results, with an even larger difference.

with current 3.15

3.15.0a0 (heads/main:fc0305a2d8, Sep  5 2025, 11:28:18) [GCC 14.3.0]
List construct: 26 msec
Set conversion: 136 msec
Set conversion (cached): 3 msec

with 3.11

3.11.13 (main, Jul 11 2025, 22:43:55) [Clang 20.1.4 ]
List construct: 92 msec
Set conversion: 39 msec
Set conversion (cached): 3 msec

Next, I modified the test slightly: first I timed str() to see how long string materialization took, then I timed hash() while relying on the cached self._str to measure hashing alone. This showed the issue is in string conversion, not hashing.

with 3.15

3.15.0a0 (heads/main:fc0305a2d8, Sep  5 2025, 11:28:18) [GCC 14.3.0]
List construct: 24 msec
toStr: 93 msec
Hashing: 54 msec
Set conversion (cached): 3 msec

with 3.11

3.11.13 (main, Jul 11 2025, 22:43:55) [Clang 20.1.4 ]
List construct: 90 msec
toStr: 32 msec
Hashing: 43 msec
Set conversion (cached): 3 msec

I then examined _format_parsed_parts and compared the 3.15 and 3.11 sources; there were no differences, so the problem likely lies in string joining. I ran a focused join benchmark, which showed a significant performance gap between 3.15 and 3.11.

with 3.15

3.15.0a0 (heads/main:fc0305a2d8, Sep  5 2025, 11:28:18) [GCC 14.3.0]
55.367 ms

with 3.11

3.11.13 (main, Jul 11 2025, 22:43:55) [Clang 20.1.4 ]
14.975 ms

Conclusion: The main regression appears to be in str.join() (and, by extension, string construction), not in hashing. The toStr step is substantially slower in 3.15, and a direct join microbenchmark confirms the slowdown.

---

The code I used to test str() and hashing performance:

from pathlib import Path
import time
import sys

print(sys.version)

paths_str = [f"/tmp/foo/bar/{i}" for i in range(10000, 70000)]

t = time.perf_counter()
paths_path = [Path(path) for path in paths_str]

elapsed = time.perf_counter() - t
print(f"List construct: {int(elapsed * 1000)} msec")

t = time.perf_counter()
h = ''
for i in paths_path:
    h = str(i)
elapsed = time.perf_counter() - t
print(f"toStr: {int(elapsed * 1000)} msec")

t = time.perf_counter()
h = 0
for i in paths_path:
    h ^= hash(i)
elapsed = time.perf_counter() - t
print(f"Hashing: {int(elapsed * 1000)} msec")

t = time.perf_counter()
set(paths_path)
elapsed = time.perf_counter() - t
print(f"Set conversion (cached): {int(elapsed * 1000)} msec")

The code I used to test string-join performance:

import time, sys
print(sys.version)
def bench():
    R = 1
    parts = ["x"*32]*256
    sep = "/"
    loops = 15000
    best = 1e9
    for _ in range(R):
        t0 = time.perf_counter()
        acc = 0
        for _ in range(loops):
            s = sep.join(parts)
            acc += len(s)
        dt = time.perf_counter()-t0
        best = min(best, dt)
    print(f'{best*1000:.3f} ms')
bench()

[eendebakpt]

I cannot reproduce the performance regression with the script for join on my system (Linux).

@LightningFarter I do notice in your results comparing 3.15 with 3.11 that you use a different compiler (gcc for 3.15, clang for 3.11). Can you try with the same compiler (and same optimization settings)?

[LightningFarter]

@eendebakpt I am previously testing on nixos laptop, I re-run the test on my ubuntu using my desktop the same issue still persists.

I noticed that I have missed to mention the flag to build python 3.15. I was using ./configure --enable-optimizations --with-lto CFLAGS="-O3 -g" to build python 3.15 to disable the checkings and debuging to optimize the performance and make comparison condition similar.

using ./configure --with-pydebug will create significant performance bias as shown below

3.15.0a0 (heads/main:fc0305a2d8, Sep  7 2025, 03:41:25) [GCC 13.3.0]
List construct: 146 msec
toStr: 673 msec
Hashing: 412 msec
Set conversion (cached): 19 msec

using ./configure --enable-optimizations --with-lto CFLAGS="-O3 -g" reproduces the performance regression with running time similar to the result above

3.15.0a0 (heads/main:fc0305a2d8, Sep  7 2025, 03:27:52) [GCC 13.3.0]
List construct: 27 msec
toStr: 99 msec
Hashing: 59 msec
Set conversion (cached): 4 msec

while testing on python 3.11 with the same gcc is shown as below

3.11.13 | packaged by conda-forge | (main, Jun  4 2025, 14:48:23) [GCC 13.3.0]
List construct: 100 msec
toStr: 42 msec
Hashing: 52 msec
Set conversion (cached): 4 msec

python 3.12 and 3.13 can also reproduce the performance regression

3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
List construct: 38 msec
toStr: 124 msec
Hashing: 55 msec
Set conversion (cached): 4 msec

3.13.5 | packaged by conda-forge | (main, Jun 16 2025, 08:27:50) [GCC 13.3.0]
List construct: 37 msec
toStr: 115 msec
Hashing: 57 msec
Set conversion (cached): 4 msec

[picnixz]

We had some performance drops in str.rstrip as well #131947 in 3.13 and later but I still don't why there was this performance drop. Maybe something else changed, maybe it's something at the interpreter level that made all string operations slower or more generally unicode but I don't know. Could it be possible to bisect the bad commit when performances dropped for strings?

[Zheaoli]

I'm working on a standard pyperf benchmark and trying to bisect the bad commit

[csernazs]

I also could not reproduce the str.join performance difference between python versions (3.11, 3.12 and the head of main branch showing roughly the same numbers). I'm also using nixos and what I initially overlooked is the difference between the nixos provided python and the one I built naively for testing (which shows significant difference).

For the pathlib performance issue, I already found the commit which caused this issue: 11c3020.

Zsolt

[picnixz]

I'm also using nixos and what I initially overlooked is the difference between the nixos provided python

That's what I suspected because one is provided by conda-forge and the other is built from source.

For the pathlib performance issue, I already found the commit which caused this issue

I wouldn't call it an issue as this commit is a necessary change for a real bug. I think we need to live with this. On my machine, I actually improvements but I think this is within the realm of noise (my system is not entirely idle):

$ uv run --python 3.11 -m pyperf timeit -s 'from pathlib import Path; p = Path("/tmp/foo/bar/12345")' 'hash(p); del p._hash' --compare-to=python3.14 --python-names 3.14:3.11
Mean +- std dev: [3.14] 305 ns +- 9 ns -> [3.11] 339 ns +- 7 ns: 1.11x slower

In order to bypass the caching, I need to delete the cached hash everytime I call hash(p). I think it's not something that is a real bug. Can you (1) compile from source Python 3.11 and Python 3.14 with same optimization flags (but don't include debugging symbols with -g), that is only do --enable-optimizations --with-lto=yes and run the following:

$ python3.11 -m pyperf timeit -s 'from pathlib import Path; p = Path("/tmp/foo/bar/12345")' 'hash(p); del p._hash' --compare-to=python3.14 --python-names 3.14:3.11

You can download pyperf and put in another location and inject the dependency into Python 3.11 and Python 3.14 by specifying PYTHONPATH=/path/to/pyperf python3.11 -m pyperf [...]

[csernazs]

Let me rephase my wording: it is not an issue, this is a performance regression which makes code run significantly slower when it deals with 6000+ paths. I don't think that commit is buggy or errorneous, I just identified with git bisect that that commit introduced the slowness.

I think pathlib.Path is a pretty good object which makes python code more readable and maintainable but comparing its performance especially with str is significant. Overhead can be justified of course (between str and Path) but this got worse between two python releases.

In your last pyperf code:

del p._hash is not sufficient as Path object has other cached attributres:

_str_normcase: https://github.com/python/cpython/blob/main/Lib/pathlib/__init__.py#L200

_str: https://github.com/python/cpython/blob/main/Lib/pathlib/__init__.py#L253

This is what I also reported that str() got also slower.

Zsolt