Make python's pc files findable by pkgconf

[virtuald]

Currently they aren't findable.

On Linux, the pc files are often installed in the system pkgconfig directory, which is undesirable to add to the PKG_CONFIG_PATH. We could create a temporary directory and add a symlink to the files (since pkgconf/pkgconf#363), but that could have a performance impact -- but maybe it doesn't matter? Ideally, there would be a way to tell pkgconf to only find specific .pc files.

On macOS and Windows, as far as I can see the .pc files are typically isolated to that specific python interpreter (and not installed in a global shared location) so just adding the directory to PKG_CONFIG_PATH would be sufficient.

Question open at pkgconf/pkgconf#391

[virtuald]

pkgconf master has support for PKG_CONFIG_PRELOADED_FILES which should enable this. Should be able to get to this soon.

[virtuald]

Womp, discovered another requirement needed here.

If I look at my mac's python, it has a python-3.13-embed.pc and python-3.13.pc. However, my original goal was to allow something like pybind11 add 'python3' to depend on the python interpreter so that the user didn't have to do anything extra to make that work.

On my Fedora machine, I can see that the distro adds a python3.pc symlink for the preferred python -- but it obviously might not be the interpreter we're currently using/building for.

Will have to think on this. Adding another environment variable (or modifying the existing one they added for us) to make a virtual symlink probably isn't a general enough capability to justify adding it.

[virtuald]

One idea I started on: replacing 'python{major}' or 'python{major}-embed' as found in the pkgconf command line with 'python-{major}.{minor}'. At some point I realized that this would work to directly find a package, but would fail whenever pkgconf would try to resolve a dependency.

[rgommers]

Interesting idea in principle, but the question is "which interpreter" I'd think. Were you thinking about the interpreter in the active environment, or in another one? For the latter, there is https://peps.python.org/pep-0739/#c-api-pkgconfig-path

[virtuald]

This package is only able to locate .pc files that were installed for the current python environment, so I'm not sure any other interpreter really would make sense, until we have a way to resolve entry points in other environments (so... #74).

[virtuald]

So one terrible solution for this is to have a pkgconf-python-pc wheel which is python version specific and distributes an entrypoint and a python3/python3-embed .pc file that points at the interpreter-specific version, and then have pkgconf depend on that and use PKG_CONFIG_PRELOADED_FILES to include the actual interpreter .pc files.

[virtuald]

Expanding on the terrible solution, we can actually make it slightly less terrible by using pkgconf variable support -- this allows us to just ship two .pc files with the pkgconf wheel which can point at the actual python .pc files.

baz.pc (this represents the library that a user is trying to link to that requires python):

Name: baz
Description: baz lib links to foo3
Version: 1.0
Requires: foo3

bar.pc (this represents the .pc shipped with pkgconf wheel)

Name: bar
Description: foo virtual lib
Version: 1.0
Provides: foo3 = 3
Requires: ${foo}

foo-3.13.pc (this represents the python .pc file)

Name: foo
Description: foo lib
Version: 1.0
Requires: ${foo}
Libs: -Lfoo -lfoo

If you run PKG_CONFIG_PATH=$(pwd) pkgconf --libs baz this doesn't do anything special, but this does:

$ PKG_CONFIG_PATH=$(pwd) pkgconf --define-variable=foo=foo-3.13 --libs baz
-Lfoo -lfoo

As you can see, pkgconf will return the linking flags for foo-3.13 as part of its resolving process. You can even set the version of the 'Provides' by using a variable there too.

Name: bar
Description: foo virtual lib
Version: 1.0
Provides: ${provided-foo}
Requires: ${foo}

$ PKG_CONFIG_PATH=$(pwd) pkgconf --define-variable=foo=foo-3.13 '--define-variable=provided-foo=foo3 = 3.13' --libs baz
-Lfoo -lfoo

It's awful, but I think it's the best path forward for this.

[rgommers]

Let me ask a question here: is this exercise really worthwhile? There are a couple of possible things that this idea could achieve, but I'm not sure if any of them are worth it:

1. Allow detection of Python for native compiles, without having to install pkg-config at the system level.
   • Already covered just fine by python-config though.
2. Allow detection of Python for cross compiles, without having to install pkg-config at the system level.
   • Can help, but is pretty niche and people who cross compile can deal with installing pkg-config themselves just fine.
3. Allow .pc files of Python packages to depend on Python
   • Assumes that a package like pybind11 would want to change to using this, since they currently inject Python flags. Unclear they'd want to change though, nor if what they're currently doing is good practice.
   • I think what numpy does is better: only provide its own compile flags. Yes it relies on Python, but it's always for building extension modules and the needed flags for that must anyway be added by the build system.

However, my original goal was to allow something like pybind11 add 'python3' to depend on the python interpreter so that the user didn't have to do anything extra to make that work.

With what build systems does the user actually have to do something in practice?

[virtuald]

With what build systems does the user actually have to do something in practice?

The place where I ran into this limitation is in my autogen for semiwrap which parses C++ headers -- it needs to run a C++ preprocessor to do the parsing, which fails if there are missing headers (like python.h). My autogen script initially was just using pkgconf to resolve all the include directories / etc, but I had to add a special case to also include the python include directory. Not the end of the world, but seemed like an unnecessary special case that would be easy to solve (ha).

---

In my mind, the primary usefulness of this package isn't just that it contains a pkg-config implementation, but that it is able to find .pc files published by installed wheels. As such, it seems reasonable that things contained in a wheel might link to python (or at least include the header in one of their header files).

1. Sure, python-config exists, but if pkgconf can find the same information then that's one less special case that tools/build systems have to account for
2. Same
3. a. since they currently inject Python flags... It depends on how you find pybind11 and set the flags up. They have a script that does this, but you don't have to use it -- and my preference would be to solely use pkgconf-pypi to find it and use it (and it works too now that pybind11 publishes a .pc file except for the python special case)
   b. If NumPy published a .pc file I would expect it to also have Requires: python in it also if its headers aren't usable without the python headers?

One place where this could be a problem is for PyPy? If pybind11.pc depends on python3, I haven't looked but my guess is that PyPy is going to publish a different .pc file? Though maybe since we could determine it dynamically here it could just point at that instead.

... probably both PyPy and Python should change their .pc files to have a Provides: python3 = X line in them, then my nasty solution wouldn't be necessary. However, I think Provides: is pkgconf-specific?

[rgommers]

Re pybind11, that's inconsistent behavior between pybind11-config and pybind11.pc indeed. The latter does not have Requires: python though: https://github.com/pybind/pybind11/blob/master/tools/pybind11.pc.in. Which I think is correct.

If NumPy published a .pc file I would expect it to also have Requires: python in it also if its headers aren't usable without the python headers?

No, that's at best going to lead to duplicate flags and at worst to finding and linking two incompatible Python targets. This is not a good idea.

but if pkgconf can find the same information then that's one less special case that tools/build systems have to account for

That ship has definitely sailed. Building extension modules requires a lot of special cases. Doing what you propose is not going to change that, and no build tool will be able to rely on this proposed change to drop another special case.

In my mind, the primary usefulness of this package isn't just that it contains a pkg-config implementation, but that it is able to find .pc files published by installed wheels.

Yes, agreed, that is one of its primary purposes.

As such, it seems reasonable that things contained in a wheel might link to python (or at least include the header in one of their header files).

They may, but as explained above there is no use case here for regular building of extension modules. I can imagine there's another use case though. For semiwrap, from reading from its description about what it does, I'm unclear on whether this is such a case - I would not think so, since you're generating meson.build files and Meson already handles the Python dependency automatically when building extension modules.