How to build multiple wheels?

[c0d1f1ed]

I have a project which has pre-built dynamic libraries; .dll for Windows, .so for Linux, and .dylib for macOS, some for either x86-64 or AArch64. Is it feasible to create multiple .whl distributions from one py -m build --wheel invocation for easy uploading to PyPI?

I've tried calling setuptools.setup() within my setup.py file multiple times. Each time with a different set of libraries listed as package_data. After successfully creating the first wheel for Windows (including the .dll files) the second setup() call starts collecting the files for Linux and as expected it copies .so files but then produces an error about not finding .dll files:

...
copying warp\bin\warp-clang.so -> build\lib.win-amd64-cpython-311\warp\bin
copying warp\bin\warp.so -> build\lib.win-amd64-cpython-311\warp\bin
running build_ext
installing to build\bdist.win-amd64\wheel
running install
running install_lib
copying build\lib.win-amd64-cpython-311\warp\bin\warp-clang.dll -> build\bdist.win-amd64\wheel\.\warp\bin
error: could not create 'build\bdist.win-amd64\wheel\.\warp\bin\warp-clang.dll': No such file or directory

I've tried removing ./build and/or ./warp_lang.egg-info in between the setup() calls, but that causes other errors.

Any advice on how to best go about this? Thanks!

[c0d1f1ed]

As an alternative I've been trying to pass an extra argument from py -m build --wheel into setup.py to indicate which platform to build a wheel for. What almost works is py -m build --wheel --config-setting=--build-option=--platform=manylinux2014_x86_64 by doing:

class warp_bdist_wheel(wheel.bdist_wheel):
    user_options = wheel.bdist_wheel.user_options + [
        ("platform=", "P", "Wheel platform: win_amd64/manylinux2014_x86_64/..."),
    ]

    def initialize_options(self):
        super().initialize_options()
        self.platform = ""

    def finalize_options(self):
        super().finalize_options()
        global platform
        platform = self.platform

    def get_tag(self):
        return "py3", "none", platform

setuptools.setup(
    ...
    package_data=get_libraries_for(platform),
    cmdclass={"bdist_wheel": warp_bdist_wheel},
)

The problem is that get_libraries_for(platform) gets called before we have access to the command line argument in finalize_options().

I could use sys.argv / argparse to get it sooner. Anyone know a cleaner way to handle this?

[webknjaz]

Use pypa/cibuildwheel @ GHA. It'll make you entire matrix of compliant wheels. Just using setuptools is not enough, usually, since it's not allowed to link dynamically against most things and the wheels need to be postprocessed (into manylinux and musllinux wheels, for example).

[c0d1f1ed]

Thanks for the suggestion! Indeed cibuildwheel looks promising, but we use a CI system called TeamCity which doesn't appear to be explicitly supported. Can it be used independently from CI systems?

Just using setuptools is not enough, usually, since it's not allowed to link dynamically against most things...

I'm not trying to use setuptools to build the dynamic libraries themselves. Our project uses LLVM, which can only really be built using CMake. So we're treating building these binaries as separate from building the wheels.

...and the wheels need to be postprocessed (into manylinux and musllinux wheels, for example).

Sorry, I'm not sure I understand this restriction. The way we build our Linux shared library ensures that they works fine across the vast majority of Linux distributions, so that's not a reason for us to move away from setuptools. If it could just have an easier way to pass a platform argument (similar to the cibuildwheel one) from py -m build down to setup.py that would be very helpful, and secondly allowing multiple setup() calls, that would solve all our issues.

Today I actually got everything working, but in a hacky way; by calling python -m build --wheel -C--build-option=-Pwindows && python -m build --wheel -C--build-option=-Plinux && python -m build --wheel -C--build-option=-Pmacos it builds three wheels. But it required manual parsing of -P at the top of setup.py so we can adjust package_data ahead of executing the bdist_wheel command, and we need to clear the /build directory after every wheel creation with an overriding warp_bdist_wheel.run() method (cf. #1871).

• NVIDIA/warp@1f8c238
• NVIDIA/warp@6d2f969