gh-118761: Improve import time of `mimetypes` #126979

hugovk · 2024-11-18T20:03:53Z

Makes import time 11 to 16 times as fast. Measured with a PGO and LTO non-debug build on macOS.

The slowest import in mimetypes is urllib.parse (taking 5,174 of mimetypes 5,448 μs = 95%).
After deferring urllib.parse, import time is 480 μs. That's 11.35 times as fast.
We could stop here. The other imports are easy enough to defer as well and have some benefit when running with -S to not import site on initialisation: 7,540 μs -> 469 μs = 16.08 times as fast.

`-X importtime`

`python.exe`

total import time: 0.010s -> 0.006s -> 0.005s

main	defer `urllib.parse`	defer rest

`mimetypes` import time: 0.005s -> 0.000s -> 0.000s

main	defer `urllib.parse`	defer rest

`python.exe -S`

total import time: 0.013s -> 0.009s -> 0.006s

main	defer `urllib.parse`	defer rest

`mimetypes` import time: 0.013s -> 0.004s -> 0.001s

main	defer `urllib.parse`	defer rest

hyperfine

`python.exe`: 24.2 ms -> 11.2 ms

`main`

❯ hyperfine --warmup 8 "./python.exe -c 'import mimetypes'"
Benchmark 1: ./python.exe -c 'import mimetypes'
  Time (mean ± σ):      24.2 ms ±  11.3 ms    [User: 15.2 ms, System: 5.7 ms]
  Range (min … max):    16.2 ms …  59.6 ms    133 runs

PR

❯ hyperfine --warmup 8 "./python.exe -c 'import mimetypes'"
Benchmark 1: ./python.exe -c 'import mimetypes'
  Time (mean ± σ):      11.2 ms ±   0.7 ms    [User: 8.1 ms, System: 2.6 ms]
  Range (min … max):    10.3 ms …  14.2 ms    170 runs

`python.exe -S`: 15.1 ms -> 8.4 ms

`main`

❯ hyperfine --warmup 8 "./python.exe -S -c 'import mimetypes'"
Benchmark 1: ./python.exe -S -c 'import mimetypes'
  Time (mean ± σ):      15.1 ms ±   0.7 ms    [User: 11.4 ms, System: 3.1 ms]
  Range (min … max):    14.3 ms …  19.2 ms    133 runs

PR

❯ hyperfine --warmup 8 "./python.exe -S -c 'import mimetypes'"
Benchmark 1: ./python.exe -S -c 'import mimetypes'
  Time (mean ± σ):       8.4 ms ±   0.4 ms    [User: 5.7 ms, System: 2.2 ms]
  Range (min … max):     7.9 ms …  12.3 ms    216 runs

Issue: Improve import time of various stdlib modules #118761

asvetlov

Import time improvement looks great!
My question is: how do these import mod lines in a function body affect the function execution time itself?
IIRC import statement accuires a lock, isn't it?

danielhollas

Nice! 🎉

danielhollas · 2024-11-19T02:00:27Z

Lib/mimetypes.py

 read_mime_types(file) -- parse one file, return a dictionary or None
 """

-import os


I wonder if lazy import of os module is worth it here, it is needed three times and the improvement is small and only in a special case.

Python always imports os and sys at startup. Moving import sys makes sense, it's only used by _main(). But I'm not sure about moving os.

In the OP, @hugovk provides timings for when the python is run without the site.py module (python -S), in which case os and sys are not loaded I believe?

Yes, it makes a difference with python -S, compare the last three images above.

But I'm fine reverting any of these, because I expect with site.py is by far the most usual thing.

hugovk · 2024-11-21T13:20:08Z

Import time improvement looks great! My question is: how do these import mod lines in a function body affect the function execution time itself? IIRC import statement accuires a lock, isn't it?

They do affect execution time but in a much smaller way. Let's time these scripts that call a function a million times:

# 1.py
import urllib.parse


def thing():
    urllib.parse.urlparse("https://example.com")


for _ in range(1_000_000):
    thing()

# 2.py
def thing():
    import urllib.parse

    urllib.parse.urlparse("https://example.com")


for _ in range(1_000_000):
    thing()

Importing outside the loop is 1.06 times faster, but this is with a million loops:

❯ hyperfine --warmup 1 "python3.14 1.py" "python3.14 2.py"
Benchmark 1: python3.14 1.py
  Time (mean ± σ):      1.403 s ±  0.014 s    [User: 1.380 s, System: 0.019 s]
  Range (min … max):    1.392 s …  1.441 s    10 runs

Benchmark 2: python3.14 2.py
  Time (mean ± σ):      1.486 s ±  0.006 s    [User: 1.464 s, System: 0.020 s]
  Range (min … max):    1.477 s …  1.498 s    10 runs

Summary
  python3.14 1.py ran
    1.06 ± 0.01 times faster than python3.14 2.py

Looping 1,000 times shows no difference:

❯ hyperfine --warmup 3 "python3.14 1.py" "python3.14 2.py"
Benchmark 1: python3.14 1.py
  Time (mean ± σ):      17.5 ms ±   1.5 ms    [User: 14.1 ms, System: 2.8 ms]
  Range (min … max):    16.7 ms …  34.1 ms    152 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: python3.14 2.py
  Time (mean ± σ):      17.5 ms ±   1.0 ms    [User: 14.1 ms, System: 2.7 ms]
  Range (min … max):    16.9 ms …  28.2 ms    150 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  python3.14 2.py ran
    1.00 ± 0.10 times faster than python3.14 1.py

asvetlov · 2024-11-21T13:55:52Z

6% slower in the worst case and no difference in a more realistic scenario sounds good to me.
Thanks!

vstinner

LGTM.

hugovk · 2024-11-21T14:55:24Z

Thanks for the reviews!

hugovk added 4 commits November 18, 2024 21:23

Improve import time of mimetypes: defer urllib.parse

4069061

Update link redirect

12fc4fe

Improve import time of mimetypes: defer os, sys, posixpath

e6e4752

Add blurb

0f2ccf0

hugovk requested a review from a team as a code owner November 18, 2024 20:03

bedevere-app bot added the awaiting core review label Nov 18, 2024

hugovk changed the title ~~Improve import time of mimetypes~~ gh-118761: Improve import time of mimetypes Nov 18, 2024

bedevere-app bot mentioned this pull request Nov 18, 2024

Improve import time of various stdlib modules #118761

Closed

asvetlov reviewed Nov 18, 2024

View reviewed changes

danielhollas reviewed Nov 19, 2024

View reviewed changes

asvetlov approved these changes Nov 21, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Nov 21, 2024

vstinner approved these changes Nov 21, 2024

View reviewed changes

hugovk merged commit dc7a2b6 into python:main Nov 21, 2024
42 checks passed

bedevere-app bot removed the awaiting merge label Nov 21, 2024

hugovk deleted the mimetypes-defer-imports branch November 21, 2024 14:55

ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025

pythongh-118761: Improve import time of mimetypes (python#126979)

883457a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-118761: Improve import time of `mimetypes` #126979

gh-118761: Improve import time of `mimetypes` #126979

Uh oh!

hugovk commented Nov 18, 2024 •

edited by bedevere-app bot

Loading

Uh oh!

asvetlov left a comment

Uh oh!

danielhollas left a comment

Uh oh!

danielhollas Nov 19, 2024

Uh oh!

vstinner Nov 19, 2024

Uh oh!

danielhollas Nov 19, 2024

Uh oh!

hugovk Nov 21, 2024

Uh oh!

hugovk commented Nov 21, 2024

Uh oh!

asvetlov commented Nov 21, 2024

Uh oh!

vstinner left a comment

Uh oh!

hugovk commented Nov 21, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

gh-118761: Improve import time of mimetypes #126979

gh-118761: Improve import time of mimetypes #126979

Uh oh!

Conversation

hugovk commented Nov 18, 2024 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

-X importtime

python.exe

total import time: 0.010s -> 0.006s -> 0.005s

mimetypes import time: 0.005s -> 0.000s -> 0.000s

python.exe -S

total import time: 0.013s -> 0.009s -> 0.006s

mimetypes import time: 0.013s -> 0.004s -> 0.001s

hyperfine

python.exe: 24.2 ms -> 11.2 ms

main

PR

python.exe -S: 15.1 ms -> 8.4 ms

main

PR

Uh oh!

asvetlov left a comment

Choose a reason for hiding this comment

Uh oh!

danielhollas left a comment

Choose a reason for hiding this comment

Uh oh!

danielhollas Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

vstinner Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

danielhollas Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

hugovk Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

hugovk commented Nov 21, 2024

Uh oh!

asvetlov commented Nov 21, 2024

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

hugovk commented Nov 21, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-118761: Improve import time of `mimetypes` #126979

gh-118761: Improve import time of `mimetypes` #126979

hugovk commented Nov 18, 2024 •

edited by bedevere-app bot

Loading

`-X importtime`

`python.exe`

`mimetypes` import time: 0.005s -> 0.000s -> 0.000s

`python.exe -S`

`mimetypes` import time: 0.013s -> 0.004s -> 0.001s

`python.exe`: 24.2 ms -> 11.2 ms

`main`

`python.exe -S`: 15.1 ms -> 8.4 ms

`main`