Skip to content

Add parallel compile from pybind11 #8990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 15, 2025
Merged

Add parallel compile from pybind11 #8990

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2059e06
Add parallel compile from pybind11
wiredfool May 30, 2025
b931402
add pybind11 elsewhere so mypy can find it
wiredfool May 31, 2025
2316c93
Removed default argument
radarhere Jun 19, 2025
a549c55
Merge branch 'main' into pybind11
radarhere Jun 25, 2025
ecd264f
Use "parallel" config setting and 4 as defaults
radarhere Jun 21, 2025
23ed906
Removed default limit of 4
radarhere Jun 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .ci/requirements-mypy.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ipython
numpy
packaging
pyarrow-stubs
pybind11
pytest
sphinx
types-atheris
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[build-system]
build-backend = "backend"
requires = [
"pybind11",
"setuptools>=77",
]
backend-path = [
Expand Down
3 changes: 3 additions & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@
from collections.abc import Iterator
from typing import Any

from pybind11.setup_helpers import ParallelCompile
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext

ParallelCompile("MAX_CONCURRENCY").install()
Copy link
Member

@radarhere radarhere Jun 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ParallelCompile("MAX_CONCURRENCY").install()
ParallelCompile("MAX_CONCURRENCY", default=4).install()

Actually, from our documentation, this should be 4.

https://pillow.readthedocs.io/en/stable/installation/building-from-source.html#build-options

Config setting: -C parallel=n. Can also be given with environment variable: MAX_CONCURRENCY=n. Pillow can use multiprocessing to build the extension. Setting -C parallel=n sets the number of CPUs to use to n, or can disable parallel building by using a setting of 1. By default, it uses 4 CPUs, or if 4 are not available, as many as are present.

Copy link
Member

@radarhere radarhere Jun 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've created wiredfool#12 to use -C parallel=n as well.

Copy link
Member

@hugovk hugovk Jun 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From pybind11 docs:

You can also pass default=N to set the default number of threads (0 will take the number of threads available) and max=N, the maximum number of threads; if you have a large extension you may want set this to a memory dependent number.

If not set, pybind11 uses 0.

Pillow's limit of 4 was added in 2014 (#1043) but machines have more CPUs today.

Last I looked, the lowest amount on a macOS on the market was 8, and Linux laptops/desktops usually have at least around this many. Servers and shared systems may be lower.

Shall we leave the default to 0 (and update the docs)?

Copy link
Member

@radarhere radarhere Jun 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've pushed a commit.



def get_version() -> str:
version_file = "src/PIL/_version.py"
Expand Down
Loading