Prototype support for PEP 739 (continued) #14657

mgorny · 2025-05-28T18:54:51Z

This is a continuation of work from #13945, see #13945 (comment). For a start, just rebased it against fresh git head.

mgorny · 2025-07-09T17:58:22Z

Okay, I think this is as far as I can go on my own.

Few final points:

I've disabled it on macOS, since — at least judging by the existing code — we need to hack around frameworks, and the relevant data is not available in build-details.json. I don't have a macOS system to work on this properly. I think it's good enough to focus on getting it right for the more general POSIX case first.
The remaining CI failures don't seem to be related to these changes.
I've been able to successfully cross-compile NumPy with these changes.

rgommers

Nice to see it's ready for testing!

You may want to rewrite history already, since it's required in Meson development to have self-contained orthogonal commits before merging.

A few initial comments; I'll try to find time to test this asap!

.github/workflows/cygwin.yml

docs/markdown/Builtin-options.md

rgommers · 2025-07-09T18:59:05Z

mesonbuild/dependencies/python.py

+class PythonBuildConfig:
+    """PEP 739 build-details.json config file."""
+
+    """Schema version currently implemented."""


nit: should be a regular code comment (# ) it looks like. Same comment for 2 lines lower.

I suppose it was meant as a docstring, but according to PEP 258, attribute docstrings go below the assignment, not above. While it was rejected, it seems that tools support this convention and I see it used e.g. in mesonbuild/cargo/manifest.py, so I'll just fix the location.

rgommers · 2025-07-09T19:05:43Z

mesonbuild/dependencies/python.py

-        self.variables = python_holder.info['variables']
+        self.build_config = python_holder.build_config
+
+        if self.build_config:


When I look at the multiple if self.build_config ... else I wondered if it isn't better to combined build_config and info into the same structure in a single place. I bet you considered it though and decided against doing that?

To be honest, I haven't given it much thought, but they certainly do trigger different code paths, and contain slightly different information. I don't think trying to combine them would make the code more readable.

rgommers · 2025-07-09T19:19:35Z

I've been able to successfully cross-compile NumPy with these changes.

Do you happen to have a reusable recipe for this?

rgommers · 2025-07-10T05:45:50Z

This seems to work as expected, here is a set of CI jobs that cross-compiles SciPy from x86-64 to aarch64 in the first job, and then uses an aarch64 runner in the second job to download the built wheel and run the test suite:

scipy/scipy@main...rgommers:scipy:ci-cross
passing CI log (modulo one test failure)

I think there was a script to generate the build-details.json file; couldn't find it so quickly so I hardcoded the paths for now. Overall this was pretty smooth though. I handwrote the json file by taking the example from PEP 739 and just modifying for the target interpreter, took only a minute or two to do that.

mgorny · 2025-07-10T12:19:05Z

I've been able to successfully cross-compile NumPy with these changes.

Do you happen to have a reusable recipe for this?

Not really reusable, no. I've used Gentoo's crossdev to get the needed dependencies, i.e.:

crossdev aarch64-unknown-linux-gnu
aarch64-unknown-linux-gnu-emerge -1v openblas python:3.14

As a result, I've been able to test on a build-details.json from a real Python installation, which amounted to:

`build-details.json`

{
  "schema_version": "1.0",
  "base_prefix": "/usr",
  "base_interpreter": "/usr/bin/python3.14",
  "platform": "linux-aarch64",
  "language": {
    "version": "3.14",
    "version_info": {
      "major": 3,
      "minor": 14,
      "micro": 0,
      "releaselevel": "beta",
      "serial": 3
    }
  },
  "implementation": {
    "name": "cpython",
    "cache_tag": "cpython-314",
    "version": {
      "major": 3,
      "minor": 14,
      "micro": 0,
      "releaselevel": "beta",
      "serial": 3
    },
    "hexversion": 51249331,
    "_multiarch": "aarch64-linux-gnu"
  },
  "abi": {
    "flags": [],
    "extension_suffix": ".cpython-314-aarch64-linux-gnu.so",
    "stable_abi_suffix": ".abi3.so"
  },
  "suffixes": {
    "source": [
      ".py"
    ],
    "bytecode": [
      ".pyc"
    ],
    "extensions": [
      ".cpython-314-x86_64-linux-gnu.so",
      ".abi3.so",
      ".so"
    ]
  },
  "libpython": {
    "dynamic": "/usr/lib/libpython3.14.so",
    "dynamic_stableabi": "/usr/lib/libpython3.so",
    "link_extensions": false
  },
  "c_api": {
    "headers": "/usr/include/python3.14",
    "pkgconfig_path": "/usr/lib/pkgconfig"
  }
}

Created a cross file:

meson --cross-file

[binaries]
c = 'aarch64-unknown-linux-gnu-gcc'
cpp = 'aarch64-unknown-linux-gnu-g++'
pkg-config = 'aarch64-unknown-linux-gnu-pkg-config'

[properties]
sys_root = '/usr/aarch64-unknown-linux-gnu'
pkg_config_libdir = '/usr/aarch64-unknown-linux-gnu/usr/lib/pkgconfig'
needs_exe_wrapper = true
longdouble_format = 'IEEE_QUAD_LE'

[host_machine]
system = 'linux'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'

Merged this branch with #14773, copied mesonbuild/modules/features from NumPy and run:

./meson.py setup --python.build-config /usr/aarch64-unknown-linux-gnu/usr/lib/python3.14/build-details.json --cross-file ~/tmp/meson-cross.txt ../numpy/build ../numpy

rgommers · 2025-07-16T19:17:49Z

I opened mesonbuild/meson-python#779 for the needed meson-python changes that go with this. With this PR and that one, I could remove all the hacks and cross-compile SciPy correctly.

Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Michał Górny <[email protected]>

mgorny · 2025-08-27T16:18:42Z

Gentle ping. Should I do something else here?

jpakkane · 2025-08-27T20:37:23Z

unittests/allplatformstests.py

@@ -3055,6 +3056,121 @@ def test_pkg_config_libdir(self):
        self.wipe()
        self.init(testdir, extra_args=['-Dstart_native=true'], override_envvars=env)

+    @skipIf(is_osx(), 'Not implemented for Darwin yet')
+    @skipIf(is_windows(), 'POSIX only')


Are there estimates on how much work would supporting Windows and Mac be? As in would it need architectural changes or is it merely busywork that someone would just need to do?

I have neither a Windows nor macOS setup to work on that.

The Windows limitation comes from the original PR, and I don't really know the details — particularly, whether it's a limitation at spec level, code level or just test case.

I've added macOS skip due to CI failures. Apparently, Meson's test setup is pretty nonstandard, so I'm not even sure if it's actually a problem with the code or with the CI setup. There's also the following code:

meson/mesonbuild/dependencies/python.py

Lines 330 to 338 in e38545b

# But not Apple, because it's a framework

if self.env.machines.host.is_darwin() and 'PYTHONFRAMEWORKPREFIX' in self.variables:

framework_prefix = self.variables['PYTHONFRAMEWORKPREFIX']

# Add rpath, will be de-duplicated if necessary

if framework_prefix.startswith('/Applications/Xcode.app/'):

self.link_args += ['-Wl,-rpath,' + framework_prefix]

if self.raw_link_args is not None:

# When None, self.link_args is used

self.raw_link_args += ['-Wl,-rpath,' + framework_prefix]

I don't know how that is supposed to interact with PEP 739, and whether we shouldn't actually be fixing Python not to require these hacks (presuming that is even possible, I don't know how Frameworks work).

It's just busywork, and mostly in the test case. The config file keys/values make sense on all platforms, the annoying thing is to construct a correct config file for an unknown interpreter. This test case uses /usr, .so, etc. which isn't correct for macOS and Windows. From Python 3.14, CPython starts shipping this json file with the interpreter itself, which will make testing in the future much easier.

In practice this is primarily useful for cross compiling though, so the main use cases are for Linux et al; cross compiling on macOS and Windows isn't much of a thing in comparison.

and whether we shouldn't actually be fixing Python not to require these hacks (presuming that is even possible, I don't know how Frameworks work).

That is a workaround for the python.pc file shipped by Apple not containing -Wl,-rpath and framework installs are not on the loader search path. The latter is on purpose, for security. The former seems like a bug in the .pc file. It's bad practice to use the macOS system Python for anything as a user or application author, so it's mostly a don't-care I think.

I'd prefer for this to be merged, since it's easier to add to it later. But I can look into the macOS thing if you think that is needed now.

mgorny · 2025-08-28T19:28:07Z

Thanks a lot!

nirbheek · 2025-09-04T02:01:57Z

mesonbuild/dependencies/pkgconfig.py

@@ -41,12 +41,13 @@ def set_program_override(pkg_bin: ExternalProgram, for_machine: MachineChoice) -
        PkgConfigInterface.pkg_bin_per_machine[for_machine] = pkg_bin

    @staticmethod
-    def instance(env: Environment, for_machine: MachineChoice, silent: bool) -> T.Optional[PkgConfigInterface]:
+    def instance(env: Environment, for_machine: MachineChoice, silent: bool,
+                 extra_paths: T.Optional[T.List[str]] = None) -> T.Optional[PkgConfigInterface]:


This change of moving extra_paths into the instantiation breaks detection of Python on platforms where we need to search in a custom pkg-config path, such as on macOS.

You're caching the instances in the class_impl class variable, which means if you change extra_paths it gets ignored. The instances should probably be keyed with a tuple of extra_paths too.

project('testpy', 'c') # Comment this out and the Python dependency below will be found dependency('glib-2.0', required: false) pymod = import('python') python = pymod.find_installation('python3') python_dep = python.dependency(embed: true, include_type: 'system')

I don't have a macOS machine to reproduce. I'm going to try to build an artificial environment to reproduce this — but if you feel like writing a patch, that will probably be faster.

Okay, I didn't really reproduce a failure but I've reproduced the extra_paths mismatch.

I've implemented your suggestion in #15001. I'm not particularly proud of that solution, so if anyone has a better idea…

Add `extra_paths` to cache keys for `PkgConfigInterface` and `PkgConfigCLI` instances, to avoid incorrectly reusing an instance with a different `extra_paths` value, see: mesonbuild#14657 (comment) Signed-off-by: Michał Górny <[email protected]>

This was referenced May 28, 2025

Prototype support for PEP 739 #13945

Draft

Meson doesn't handle limited_api_suffix = None from python_info.py #14673

Open

mgorny force-pushed the pep-739-prototype branch 2 times, most recently from f02c282 to 6c6ad29 Compare June 10, 2025 19:40

mgorny force-pushed the pep-739-prototype branch from 88fa2cc to e716a1f Compare June 24, 2025 13:12

bruchar1 added the modules:python Issues specific to the python module label Jun 25, 2025

mgorny force-pushed the pep-739-prototype branch 2 times, most recently from a587516 to 0559b4a Compare July 8, 2025 17:24

mgorny marked this pull request as ready for review July 9, 2025 17:55

mgorny requested a review from jpakkane as a code owner July 9, 2025 17:55

rgommers reviewed Jul 9, 2025

View reviewed changes

mgorny force-pushed the pep-739-prototype branch 2 times, most recently from 15c0394 to a8bbbb1 Compare July 10, 2025 12:23

rgommers mentioned this pull request Jul 16, 2025

WIP: ENH: implement support for build-details.json (PEP 739) and cross compilation mesonbuild/meson-python#779

Draft

mgorny force-pushed the pep-739-prototype branch from a8bbbb1 to 9071816 Compare July 19, 2025 19:18

python: add a python.build_config option (PEP 739)

1232ec1

Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Michał Górny <[email protected]>

mgorny force-pushed the pep-739-prototype branch from 9071816 to 1232ec1 Compare July 21, 2025 12:47

jpakkane force-pushed the master branch from 00d6953 to 6447832 Compare August 19, 2025 20:51

jpakkane reviewed Aug 27, 2025

View reviewed changes

jpakkane merged commit 59c3dd1 into mesonbuild:master Aug 28, 2025
33 checks passed

mgorny deleted the pep-739-prototype branch August 28, 2025 19:28

nirbheek reviewed Sep 4, 2025

View reviewed changes

mgorny mentioned this pull request Sep 4, 2025

pkgconfig: Fix class cached to be keyed on extra_paths #15001

Open

	# But not Apple, because it's a framework
	if self.env.machines.host.is_darwin() and 'PYTHONFRAMEWORKPREFIX' in self.variables:
	framework_prefix = self.variables['PYTHONFRAMEWORKPREFIX']
	# Add rpath, will be de-duplicated if necessary
	if framework_prefix.startswith('/Applications/Xcode.app/'):
	self.link_args += ['-Wl,-rpath,' + framework_prefix]
	if self.raw_link_args is not None:
	# When None, self.link_args is used
	self.raw_link_args += ['-Wl,-rpath,' + framework_prefix]

Uh oh!

Prototype support for PEP 739 (continued) #14657

Prototype support for PEP 739 (continued) #14657

Uh oh!

Conversation

mgorny commented May 28, 2025

Uh oh!

mgorny commented Jul 9, 2025

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgommers commented Jul 9, 2025

Uh oh!

rgommers commented Jul 10, 2025

Uh oh!

mgorny commented Jul 10, 2025

Uh oh!

rgommers commented Jul 16, 2025

Uh oh!

mgorny commented Aug 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mgorny commented Aug 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!