Python wheel size for Linux seems big (?)

[eitsupi]

This may be a silly question, but when I considered the smallest Arrow dependency in Python, I was surprised to find that arro3-core has a smaller wheel size (2.5MB) than nanoarrow in Linux (4MB).
I checked and it seems that the size of the nanoarrow wheel is several times larger on Linux (4MB) than it is for macOS and Windows (0.7MB).

Is this intentional or can it be avoided?

[assignUser]

Without checking in detail I have 2 suspicions:

• clang seems to produces smaller binaries on average
• I think Linux wheels bundle more libraries than macros due to system heterogeneity

[paleolimbot]

I think Cython is also an issue with respect to large binary sizes (the generated C code is absolutely wild). With some effort we might be able to get it smaller (or just use arro3, which is great too!). One of the things I think we'd like to do if any of us has the time is rewrite the bindings in nanobind (perhaps coupled with expanding the C++ capabilities of nanoarrow in the process). I don't forsee having time to do that in the near term but would be happy to review if anybody is interested!

I did check to make sure that what we're looking at is binary size, and it appears to be the case (there was a previous issue a while back where I'd been including a bunch of source code which had inflated the size of the wheels considerably).

[WillAyd]

The latest version of Cython has some new functionality that both pandas and scikit-learn have used to reduce their wheel sizes. Might be worth trying this in the near term:

pandas-dev/pandas#61384

[WillAyd]

We can also consider building the pip wheels with -Dbuildtype=minsize instead of release, if we cared for size over speed

[WillAyd]

After some more research I think this might be resolved on main already, by our switch to using meson-python from setuptools.

To illustrate, if I download the pypi wheel and check the top 10 file sizes, I see this:

find . -type f -exec ls -alh {} \; | sort -hr -k5 | head 
-rwxr-xr-x 1 willayd willayd 2.6M Oct  7  2024 ./_schema.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 willayd willayd 2.6M Oct  7  2024 ./_buffer.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 willayd willayd 2.5M Oct  7  2024 ./_array.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 willayd willayd 2.1M Oct  7  2024 ./_ipc_lib.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 willayd willayd 1.6M Oct  7  2024 ./_types.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 willayd willayd 1.5M Oct  7  2024 ./_array_stream.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 willayd willayd 926K Oct  7  2024 ./_device.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 willayd willayd 914K Oct  7  2024 ./_utils.cpython-311-x86_64-linux-gnu.so
-rw-r--r-- 1 willayd willayd 39K Oct  7  2024 ./_buffer.pyx
-rw-r--r-- 1 willayd willayd 37K Oct  7  2024 ./schema.py

But with a local build my shared libraries clock in much smaller:

-rwxrwxr-x 1 willayd willayd 695K Jun  2  2025 ./_ipc_lib.cpython-313-x86_64-linux-gnu.so
-rwxrwxr-x 1 willayd willayd 387K Jun  2  2025 ./_array.cpython-313-x86_64-linux-gnu.so
-rwxrwxr-x 1 willayd willayd 363K Jun  2  2025 ./_schema.cpython-313-x86_64-linux-gnu.so
-rwxrwxr-x 1 willayd willayd 338K Jun  2  2025 ./_buffer.cpython-313-x86_64-linux-gnu.so
-rwxrwxr-x 1 willayd willayd 249K Jun  2  2025 ./_array_stream.cpython-313-x86_64-linux-gnu.so
-rwxrwxr-x 1 willayd willayd 178K Jun  2  2025 ./_device.cpython-313-x86_64-linux-gnu.so
-rwxrwxr-x 1 willayd willayd 164K Jun  2  2025 ./_utils.cpython-313-x86_64-linux-gnu.so
-rwxrwxr-x 1 willayd willayd 149K Jun  2  2025 ./_types.cpython-313-x86_64-linux-gnu.so
-rw-r--r-- 1 willayd willayd 40K Jun  1 04:06 ./schema.py
-rw-r--r-- 1 willayd willayd 23K Jun  1 04:06 ./c_array.py

I don't think auditwheel would drastically change the size of those libraries, so I think we should monitor this as we go through building wheels in the pre-release process

[paleolimbot]

That's a great point...I should have checked the nightlies/I forgot that we hadn't released since that change 🤦

[assignUser]

Nice, good to know!

[paleolimbot]

Closing since I think the meson update fixes this!