[FR] Abaility to specifiy what files to create a precompiled header and consume it inside of a C extension.

[AraHaan]

What's the problem this feature will solve?

Since all 3 of the major compilers (MSVC), GCC, and clang all support creating and using Precompiled Headers which can help speed up building C extension modules, I would love it if the setuptools C extension builder class had a way for it to pass the proper command line options to create and use the precompiled headers for us without manually having to build the specific C source files manually in order to use the Precompiled Header feature for a build performance improvement for large C extensions.

Describe the solution you'd like

Expose a field that represents a file that will output a precompiled header for which this precompiled header would be consumed while building the C source files of a project (say for example pch.c or pch.h).

Alternative Solutions

Manually doing what setuptools does in finding the compiler to use and selecting the proper command line options to use (more painful than just using setuptools and exposing way to use precompiled headers).

Additional context

It's needed to improve compile performance in large multiple C file C extensions which all use a common header as the compiler would process the exact same header for the exact number of C source files. This results in a build performance penalty that could be avoided with allowing to specify in setuptools that a precompiled header should be made for that header the first time a C source file is compiled resulting in a faster compile on the other C source files.

https://clang.llvm.org/docs/UsersManual.html#precompiled-headers

in MSVC there is a different option for this as well.

https://learn.microsoft.com/en-us/cpp/build/reference/yc-create-precompiled-header-file?view=msvc-170
https://learn.microsoft.com/en-us/cpp/build/reference/yu-use-precompiled-header-file?view=msvc-170

As such, it would be nice if setuptools could expose this.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[eli-schwartz]

It's needed to improve compile performance in large multiple C file C extensions which all use a common header as the compiler would process the exact same header for the exact number of C source files. This results in a build performance penalty that could be avoided with allowing to specify in setuptools that a precompiled header should be made for that header the first time a C source file is compiled resulting in a faster compile on the other C source files.

In general, you'll probably find a different build system is going to yield greater speedups. setuptools isn't really able to optimize by building multiple sources in parallel (well, it has some parallel capability but only across extensions, not across a multi-file extension).

Distributing across cores generally tends to beat out preprocessing headers, and truly-accurate incremental rebuild detection for headers makes it far less likely to resort to cleaning and rebuilding from scratch. Build systems such as https://mesonbuild.com/ (exposed via https://mesonbuild.com/meson-python/) can do that for you, and additionally support PCH files and Unity builds.

[AraHaan]

While that may be true, I need it to be supported in setuptools because a project already uses it even for some test c extensions and they have asked me here to implement a test for PCH. Having PCH inside of setuptools just like the other build-systems would benefit as well.

ZeroIntensity/pyawaitable#46

And while I would tend to agree that building on multiple cores might benefit better than a PCH, I did build a C project before that had tens of thousands of C source files (if not more) and still took over an hour to compile without PCH even with multiple cores being used. Switching on PCH for that project did cut build time down exponentially on top of building with multiple cores.

[ZeroIntensity]

I agree with @eli-schwartz. IMO, pyawaitable (my library in question here) is too small to really benefit from PCH, at least for now. That, and we're planning to move away from setuptools anyway. If we really wanted to, I suspect we could implement this relatively-easy without upstream help (probably via setuptools._distutils.ccompiler).

Meson is promising, but I'm planning on using scikit-build-core because it's got some nice integration with Hatch. Seeing that you're a maintainer of meson, is meson-python planning on adding anything similar? If so, I'll happily switch!

[eli-schwartz]

@ZeroIntensity by integration with Hatch do you mean Hatch the frontend (an environment manager and workflow tool) or Hatchling the pep 517 backend?

I think both scikit-build-core and meson-python should work fine with Hatch today -- Hatch didn't always support building a project with any pep 517 build backend but it now does. No changes needed to be made to build backends for Hatch to add support, I believe.

Since meson-python is already a full build backend it's not immediately clear to me why you'd use hatchling and meson-python together. Does hatchling e.g. postprocess wheels the way auditwheel repair does?

[ZeroIntensity]

Hatchling, the backend.

Hatch didn't always support building a project with any pep 517 build backend but it now does.

Ah, that's good. It probably didn't support it last time I tried it. I'll play around with it then, thanks!