GH-36411: [C++][Python] Some fixes for sdist build #1

raulcd · 2026-01-15T18:57:42Z

@WillAyd this is a PR against your branch. I thought it would be better to show you first. 👍 first! 😄

I have tested this locally with:

ARCHERY_DEBUG=1 PYARROW_VERSION=23.0.0.dev362 archery docker run python-sdist

The meson size seems 15% bigger than the current one, we might want to investigate if we are including something unnecessary afterwards. I've done a diffoscope between a sdist from main and the sdist from meson and the differences seem fine.

github-actions · 2026-01-15T18:58:06Z

❌ GitHub issue apache#36411 could not be retrieved.

raulcd · 2026-01-15T18:58:14Z

python/pyarrow/meson.build

+# Avoid building Pyarrow if it is a source distribution.
+if not get_option('sdist')


the only change in this file is this if and endif. The rest is just formatting

I wonder if we can change arrow_dep = [] in the top level configuration to arrow_dep = disabler() when building an sdist; that should circumvent the need for having to constantly branch like this I think?

I tried, it fails with the same issue, I could add the changes on this specific file to the existing sdist option check (instead of having two separate checks):

diff --git a/python/meson.build b/python/meson.build index c65a837868..f4622e09af 100644 --- a/python/meson.build +++ b/python/meson.build @@ -18,7 +18,6 @@ project( 'pyarrow', 'cython', - 'cpp', version: run_command( 'python', '-m', @@ -51,8 +50,10 @@ endif # https://github.com/mesonbuild/meson-python/issues/647 if get_option('sdist') - arrow_dep = [] + arrow_dep = disabler() else + add_languages('cpp', native: false) + cc = meson.get_compiler('cpp') arrow_dep = dependency( 'arrow', 'Arrow', @@ -86,6 +87,4 @@ print(incdir) numpy_dep = declare_dependency(include_directories: incdir_numpy) endif -cc = meson.get_compiler('cpp') - subdir('pyarrow')

but the change to avoid building cython (python/pyarrow/meson.build) are also still necessary.

Ah OK. No strong preference here - I think its a pretty minor consideration. Happy to keep as is

WillAyd · 2026-01-15T19:43:10Z

Nice thanks!

The meson size seems 15% bigger than the current one, we might want to investigate if we are including something unnecessary afterwards. I've done a diffoscope between a sdist from main and the sdist from meson and the differences seem fine.

You are referring to the sdist right? And you are saying that you've diffed the files before/after and they are the same?

raulcd · 2026-01-15T19:49:24Z

Yes, I've diffed the sdists there are some differences, for example we must remove possible python cached directories and some other minor things but I'll validate both sdist and wheels exhaustively once I fix the wheels too.

WillAyd · 2026-01-15T21:32:43Z

Ah nice catch. Although I have to say I'm surprised meson-python wouldn't take care of excluding the cache

WillAyd · 2026-01-20T12:28:43Z

ci/scripts/python_wheel_macos_build.sh

+delocate-listdeps ${source_dir}/python/dist/*.whl

+echo "=== (${PYTHON_VERSION}) Bundle shared libraries into wheel ==="
+delocate-wheel -w ${source_dir}/python/repaired_wheels -v ${source_dir}/python/dist/*.whl


This is required for macOS but we don't need repairwheel on linux huh? That seems a bit odd.

I ran into this a lot with Windows and it was called out during reviews, so worth double checking the need for this before going back

Indeed there seems to be something wrong with the manylinux wheels too. We should test those wheels probably on a clean environment where Arrow hasn't been built (and can't be found).
I can't seem to find the libarrow shared libraries on the manylinux wheels but they are included on the existing wheels, I looked at the latest uploaded to PyPI.
I started investigating the following conversations:

How can I bundle a shared library in a wheel with mesonpy? mesonbuild/meson-python#410

Best practice for optionally bundling a library dependency mesonbuild/meson-python#556 (reply in thread)

And from what I can see there it seems like "this shouldn't be meson's concern" but should be tackled with the auditwheel, delocate, delvewheel tools.
With CMake we had thePYARROW_BUNDLE_ARROW_CPP flag and the bundle_arrow_lib function managing that, maybe we should ask on your PR.
BTW I am triggering CI jobs on a WIP PR: apache#48882 because I don't want to "polute" your PR.

Yea that's correct - for building and distributing to another system we need to use any of those tools.

However, building and running on the same system, I would think the "system" libraries would still be locatable at runtime.

I think (as you've already found) Meson really doesn't want to include this in its scope, likely because of all the differences across platforms.

With CMake we had thePYARROW_BUNDLE_ARROW_CPP flag and the bundle_arrow_lib function managing that, maybe we should ask on your PR.

I admittedly struggle to read CMake so I might be overlooking, but this function seems to just place the existing arrow library into the wheel without any name mangling or patching of existing libraries right? If so, I think the pushback we would get is that's anincomplete solution that can easily lead to conflicts with the system, particularly depending on how runtime libraries have already been loaded. I'm also not sure how well that works on Windows.

In either case, I'm no expert, so may not hurt to ask again :-)

I will try to have all wheels green on my PR and validate the contents are the expected ones. I'll manually check against published 23.0.0 wheels. Once this is done I'll share the approach and ask about feedback. There's also something I want to investigate, it feels slightly complex/verbose to have to declare all the -Csetup-args="-Dvar=${PYARROW_WITH_VAR}" \ and wondering about the developer experience when working locally. I'll see if I can investigate that too after I fix the Windows wheels.
Thanks for all the work you did here!

wondering about the developer experience when working locally.

I usually do -Csetup-args="-Dauto_features=enabled" to enable everything, and then explicitly disable things that aren't available on my system (ex: -Csetup-args="-Dcuda=disabled")

The whole -Csetup-args=... I find to be a poor experience, particurlarly since you have to repeat that pattern for every setting. I believe that limitation is enforced by pip:

https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-C

I have done more digging on the manylinux side and we are already calling auditwheel repair that's why it wasn't failing. Before auditwheel repair using auditwheel -v show on the original wheel:

2026-01-21T11:34:59.1785416Z The following external shared libraries are required by the wheel: 2026-01-21T11:34:59.1785484Z { 2026-01-21T11:34:59.1785654Z "libarrow.so.2400": "/tmp/arrow-dist/lib/libarrow.so.2400.0.0", 2026-01-21T11:34:59.1785843Z "libarrow_acero.so.2400": "/tmp/arrow-dist/lib/libarrow_acero.so.2400.0.0", 2026-01-21T11:34:59.1786049Z "libarrow_compute.so.2400": "/tmp/arrow-dist/lib/libarrow_compute.so.2400.0.0", 2026-01-21T11:34:59.1786238Z "libarrow_dataset.so.2400": "/tmp/arrow-dist/lib/libarrow_dataset.so.2400.0.0", 2026-01-21T11:34:59.1786427Z "libarrow_flight.so.2400": "/tmp/arrow-dist/lib/libarrow_flight.so.2400.0.0", 2026-01-21T11:34:59.1786645Z "libarrow_substrait.so.2400": "/tmp/arrow-dist/lib/libarrow_substrait.so.2400.0.0", 2026-01-21T11:34:59.1786812Z "libparquet.so.2400": "/tmp/arrow-dist/lib/libparquet.so.2400.0.0" 2026-01-21T11:34:59.1786880Z }

After auditwheel repair using auditwheel -v show on the repaired wheel:

2026-01-21T11:35:23.1142034Z "libparquet-84e0c9ca.so.2400.0.0": { 2026-01-21T11:35:23.1142407Z "soname": "libparquet-84e0c9ca.so.2400.0.0", 2026-01-21T11:35:23.1142633Z "path": "/tmp/tmps3l07x0n/pyarrow.libs/libparquet-84e0c9ca.so.2400.0.0", 2026-01-21T11:35:23.1142853Z "realpath": "/tmp/tmps3l07x0n/pyarrow.libs/libparquet-84e0c9ca.so.2400.0.0", 2026-01-21T11:35:23.1142929Z "platform": { 2026-01-21T11:35:23.1143027Z "_elf_osabi": "ELFOSABI_SYSV", 2026-01-21T11:35:23.1143106Z "_elf_class": 64, 2026-01-21T11:35:23.1143194Z "_elf_little_endian": true, 2026-01-21T11:35:23.1143283Z "_elf_machine": "EM_X86_64", 2026-01-21T11:35:23.1143403Z "_base_arch": "<Architecture.x86_64: 'x86_64'>", 2026-01-21T11:35:23.1143485Z "_ext_arch": null, 2026-01-21T11:35:23.1143562Z "_error_msg": null 2026-01-21T11:35:23.1143624Z }, 2026-01-21T11:35:23.1143697Z "needed": [ 2026-01-21T11:35:23.1143798Z "libarrow-85ba3dab.so.2400.0.0", 2026-01-21T11:35:23.1143877Z "libdl.so.2", 2026-01-21T11:35:23.1143956Z "libpthread.so.0", 2026-01-21T11:35:23.1144031Z "libstdc++.so.6", 2026-01-21T11:35:23.1144108Z "libm.so.6", 2026-01-21T11:35:23.1144181Z "libgcc_s.so.1", 2026-01-21T11:35:23.1144250Z "libc.so.6", 2026-01-21T11:35:23.1144340Z "ld-linux-x86-64.so.2" 2026-01-21T11:35:23.1144400Z ] 2026-01-21T11:35:23.1144464Z },

I initially thought we were missing the libraries because they are on a different location auditwheel pushes the required shared libraries to pyarrow.libs but bundling with CMake was copying them (installing under pyarrow subfolder) and then we were linking against those and building them in the wheel.

Ah nice find. I wonder if we need to update get_library_dirs for the auditwheel-fixed wheels then to avoid test failures? At a glance, I don't see that as adding pyarrow.libs to the return value (I had to do a similar fix for Windows with delvewheel)

Alternatively there might be a setting for the 3 tools to control where the libraries are getting bundled relative to the package root

…l build to keep investigating

… of C:/arrow-dist

WillAyd · 2026-01-23T14:12:41Z

ci/scripts/python_wheel_windows_build.bat

    -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
    -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
    -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+    -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^


Can you point me to the logs for this one? I'm surprised you'd have to do anything here - does this not get executed within a conda environment?

we use vcpkg for the dependencies on our Windows wheels not conda. This is the previous failure:
https://github.com/ursacomputing/crossbow/actions/runs/21254041726/job/61163356938

Thanks. That's interesting that CMake installs the ArrowComputeConfig.cmake file but then can't find it on a search with the default path.

Checking the logs I see that ArrowComputeConfig.cmake gets installed to C:/Program Files/arrow/lib/cmake/ArrowCompute/ArrowComputeConfig.cmake The CMake Docs suggest that a search location of <prefix>/(lib/<arch>|lib*|share)/cmake/<name>*/ is a Unix convention, so I wonder if its a bug in the Arrow CMake config to install that there in the first place?

Sorry, there has been some trial and error. The initial failure was on:
https://github.com/ursacomputing/crossbow/actions/runs/21252214504/job/61156610086

2026-01-22T15:05:10.1086958Z -- Installing: C:/arrow-dist/lib/cmake/ArrowCompute/ArrowComputeConfig.cmake

We use C:/arrow-dist nowadays. That log I sent you is when I tried installing on system

All tests (per each commit) can be found on this PR and the archery jobs triggered:
apache#48882

From their docs, I think the correct cross-platform path for it should be:

<prefix>/ArrowCompute/lib/cmake/ArrowCompute/ArrowComputeConfig.cmake

oh! I see what you mean now. Sorry wasn't understanding previously. I understand what you mean, at first glance, yes that seems like a possible bug on our installation but I am not sure why Arrow is found then as it is installed following the same convention: Installing: C:/arrow-dist/lib/cmake/Arrow/ArrowConfig.cmake

I think it's Friday and I'm just looking at too many CI failures, as Arrow was found when I installed on system C:/Program Files/arrow/lib/cmake/Arrow/ArrowConfig.cmake but not on C:/arrow-dist/lib/cmake/Arrow/ArrowConfig.cmake.

In the latest log you sent it looks like it fails to find Arrow:

2026-01-22T15:05:27.9376450Z Run-time dependency arrow found: NO (tried pkgconfig and cmake) 2026-01-22T15:05:27.9376779Z Run-time dependency arrow found: NO (tried cmake) 2026-01-22T15:05:27.9376982Z 2026-01-22T15:05:27.9377373Z ..\meson.build:60:16: ERROR: Dependency lookup for Arrow with method 'pkgconfig' failed: Pkg-config for machine host machine not found. Giving up.

But I do see it as found in the first. Strange...

Ah...actually in the first CI logs where Arrow is found it does follow the cross-platform installation pattern:

Installing: C:/Program Files/arrow/lib/cmake/Arrow/ArrowConfig.cmake

I'm assuming the prefix is C:/Program Files and Windows probably doesn't care about the case sensitivity of arrow versus Arrow.

The installation path in the second log does not include the project name in the path, hence why it doesn't find Arrow at all there, I think:

Installing: C:/arrow-dist/lib/cmake/Arrow/ArrowConfig.cmake

CMake is fun :-)

github-actions bot added the awaiting committer review label Jan 15, 2026

raulcd commented Jan 15, 2026

View reviewed changes

github-actions bot added awaiting changes Component: Python awaiting change review and removed awaiting committer review awaiting changes labels Jan 15, 2026

raulcd force-pushed the raulcd-use-meson-python branch from 3c8eb2f to f098f4d Compare January 16, 2026 14:00

raulcd force-pushed the use-meson-python branch from ce85d06 to ddca98c Compare January 20, 2026 08:58

raulcd added 2 commits January 20, 2026 10:02

Some fixes for sdist build

1437209

Fix xlinux wheels

7a410f7

raulcd force-pushed the raulcd-use-meson-python branch from f098f4d to 7a410f7 Compare January 20, 2026 09:02

raulcd added 4 commits January 20, 2026 10:14

Fix macOS wheels

61f895c

Add build requirement to requirements-wheel-build.txt

7f3c548

Do not check for Gandiva as it is off on the macos wheels

7d1fea3

Try using delocate-wheel to add missing bundled shared dependencies

368b2d0

WillAyd reviewed Jan 20, 2026

View reviewed changes

delocate puts shared libraries into .dylibs folder instead of root

855cc69

github-actions bot added awaiting changes and removed awaiting change review labels Jan 20, 2026

Add some debugging info to show wheel details

ab06929

github-actions bot added awaiting change review and removed awaiting changes labels Jan 21, 2026

raulcd added 3 commits January 21, 2026 12:16

Add verbose to see all files

ddd8f49

Minor fix to python_build.bat

5b86e0f

Initial attempt to fix windows wheels

11135bf

Avoid rebuilding wheel because it currently fails and manually instal…

a1a57d8

…l build to keep investigating

raulcd force-pushed the raulcd-use-meson-python branch from 4e69467 to a1a57d8 Compare January 22, 2026 12:50

raulcd added 3 commits January 22, 2026 15:22

Try to find Arrow C++ on ARROW_HOME

d97c36a

Try installing Arrow on default location so meson can find it instead…

c1edb96

… of C:/arrow-dist

Try defining --cmake-prefix-path

728a940

WillAyd reviewed Jan 23, 2026

View reviewed changes

github-actions bot added awaiting changes and removed awaiting change review labels Jan 23, 2026

		# Avoid building Pyarrow if it is a source distribution.
		if not get_option('sdist')

GH-36411: [C++][Python] Some fixes for sdist build #1

Are you sure you want to change the base?

GH-36411: [C++][Python] Some fixes for sdist build #1

Uh oh!

Conversation

raulcd commented Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd commented Jan 15, 2026

Uh oh!

raulcd commented Jan 15, 2026

Uh oh!

WillAyd commented Jan 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raulcd Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

WillAyd Jan 20, 2026 •

edited

Loading

raulcd Jan 20, 2026 •

edited

Loading

WillAyd Jan 20, 2026 •

edited

Loading