Generating the wheels for a package that needs Boost #989

MonkeyBreaker · 2022-01-10T23:01:11Z

MonkeyBreaker
Jan 10, 2022

Hello,

I am currently trying to generate the wheels for a library that requires boost.
I spend part of my evening trying to figure out what I must do.

To provide more details, in the current CI, I use the github action install-boost to install boost dependency on each runner. I am then able to build and test the packages without any issues.

A different workflow then takes care of generating the wheels, the wheels are generated using [linux, macos, windows]-latest:

      - name: Install boost
        uses: MarkusJx/[email protected]
        id: install-boost
        with:
          boost_version: 1.76.0
          boost_install_dir: ${{ matrix.boost_install_dir }}

      - name: Build wheels
        uses: pypa/[email protected]
        env:
          CIBW_BUILD: "cp36-* cp37-* cp38-* cp39-*"
          # Skip 32 bit architectures, musllinux, and i686
          CIBW_SKIP: "*-win32 *-musllinux_x86_64 *_i686"
          CIBW_BEFORE_BUILD: python -m pip install cmake
          CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
          CIBW_TEST_REQUIRES: pytest
          CIBW_ENVIRONMENT: BOOST_ROOT=${{ steps.install-boost.outputs.BOOST_ROOT }}
          CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=${{ steps.install-boost.outputs.BOOST_ROOT }}
          CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

The current behavior I am observing is as follow:

macos the wheels are correctly generated
linux : BOOST_ROOT is empty (I print the value in my cmake scripts) and I am not even sure that the boost installed on the host will be found in the guest.
windows: BOOST_ROOT is wrong, the backslashes are missing (e.g. D:boost instead of D:\boost), for this one I need to check if maybe my cmake is at fault.

Questions:

Is cibuildwheel the correct approach when the Python package depends on external C++ libraries
If first one is "yes", do you have any example that tries to generate the wheels with external dependencies for windows, linux and mac ?

As additional information, I paste here the printed information in github action when loading cibuildwheels:

For mac:

Run pypa/[email protected]
  with:
    package-dir: .
    output-dir: wheelhouse
  env:
    CIBW_BUILD: cp36-* cp37-* cp38-* cp39-*
    CIBW_SKIP: *-win32 *-musllinux_x86_64 *_i686
    CIBW_BEFORE_BUILD: python -m pip install cmake
    CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
    CIBW_TEST_REQUIRES: pytest
    CIBW_ENVIRONMENT: BOOST_ROOT=/Users/runner/work/boost/boost
    CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=/Users/runner/work/boost/boost
    CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

For linux:

Run pypa/[email protected]
  with:
    package-dir: .
    output-dir: wheelhouse
  env:
    CIBW_BUILD: cp36-* cp37-* cp38-* cp39-*
    CIBW_SKIP: *-win32 *-musllinux_x86_64 *_i686
    CIBW_BEFORE_BUILD: python -m pip install cmake
    CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
    CIBW_TEST_REQUIRES: pytest
    CIBW_ENVIRONMENT: BOOST_ROOT=/home/runner/work/boost/boost
    CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=/home/runner/work/boost/boost
    CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

For windows:

Run pypa/[email protected]
  with:
    package-dir: .
    output-dir: wheelhouse
  env:
    CIBW_BUILD: cp36-* cp37-* cp38-* cp39-*
    CIBW_SKIP: *-win32 *-musllinux_x86_64 *_i686
    CIBW_BEFORE_BUILD: python -m pip install cmake
    CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
    CIBW_TEST_REQUIRES: pytest
    CIBW_ENVIRONMENT: BOOST_ROOT=D:\boost\boost
    CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=D:\boost\boost
    CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

Thank you for any help you can provide me with.

Best,
Julián

Answered by henryiii

Jan 11, 2022

On macOS, you are using the system libraries, so this works. Windows should be too, so this might possibly be a bug with \ being treated as a escape (not sure if it's on our side, or YAML). You might be better off with CIBW_ENVIRONMENT_PASS (and/or using static TOML configuration instead).

On linux, you are inside a docker container (manylinux, specifically), so you need to set it up there (in CIBW_BEFORE_ALL). You will need to use the system library one (very old), or build it yourself, or possibly use a separate build for it somewhere (someone else might know if there is one).

Just to verify, are you sure you need a built boost? A lot of boost is header only, and can easily be used with…

View full answer

henryiii · 2022-01-11T04:11:27Z

henryiii
Jan 11, 2022
Maintainer

On macOS, you are using the system libraries, so this works. Windows should be too, so this might possibly be a bug with \ being treated as a escape (not sure if it's on our side, or YAML). You might be better off with CIBW_ENVIRONMENT_PASS (and/or using static TOML configuration instead).

On linux, you are inside a docker container (manylinux, specifically), so you need to set it up there (in CIBW_BEFORE_ALL). You will need to use the system library one (very old), or build it yourself, or possibly use a separate build for it somewhere (someone else might know if there is one).

Just to verify, are you sure you need a built boost? A lot of boost is header only, and can easily be used without building anything at all - see boost-histogram, for example, which wraps a header only boost library with 6 dependent boost libraries and pybind11; no binaries required, and everything is even packed into the SDist.

4 replies

henryiii Jan 11, 2022
Maintainer

I think a single backslash is not required to be escaped in a normal literal in YAML, but just in case, maybe you could try adding single quotes around the expression?

henryiii Jan 11, 2022
Maintainer

Something like this:

- name: Build wheels
  uses: pypa/[email protected]
  env:
    CIBW_BUILD: "cp36-* cp37-* cp38-* cp39-*"
    BOOST_ROOT: '${{ steps.install-boost.outputs.BOOST_ROOT }}'
    BOOST_ROOT_PIPELINE: '${{ steps.install-boost.outputs.BOOST_ROOT }}'

[tool.cibuidwheel]
# Skip 32 bit architectures, musllinux, and i686
skip = ["*-win32", "*-musllinux_x86_64", "*_i686"]
before-build = "python -m pip install cmake"
test-command = "python -m pytest --rootdir={package} {package}"
test-requires = ["pytest"]

Actually, there's no need for environment-pass, since the path is meaningless on linux and environment-pass doesn't do anything on Windows or macOS, since you are always in the host environment.

MonkeyBreaker Jan 11, 2022
Author

Hi @henryiii,

Thank you for all information !

Just to verify, are you sure you need a built boost? A lot of boost is header only, and can easily be used without building anything at all - see boost-histogram, for example, which wraps a header only boost library with 6 dependent boost libraries and pybind11; no binaries required, and everything is even packed into the SDist.

You are right, I just needs the header files from some boost packages. I will have a good look on how they do this in boost-histogram, thank you.

Maybe it is a good idea that I move everything into a TOML configuration file.

I will keep you update if I make it wok !

MonkeyBreaker Jan 11, 2022
Author

I looked at boost-histogram, they have the dependencies as sub modules, the issue on my repository is that we have submodules that they use multiples part of boost. I think that for making my live easier, I prefer to download the entire header library of boost.

The tips you provide me with helped a lot !

I came with the following solution:

      - name: Install boost
        if: ${{ runner.os == 'Windows' || runner.os == 'macOS' }}
        uses: MarkusJx/[email protected]
        id: install-boost
        with:
          boost_version: 1.76.0
          boost_install_dir: ${{ matrix.boost_install_dir }}

      - name: Build wheels
        uses: pypa/[email protected]
        env:
          # Specify which Python versions to build wheels
          # https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip
          CIBW_BUILD: "cp36-* cp37-* cp38-* cp39-*"
          # Skip 32 bit architectures, musllinux, and i686
          CIBW_SKIP: "*-win32 *-musllinux_x86_64 *_i686"
          CIBW_BEFORE_BUILD: python -m pip install cmake
          CIBW_BEFORE_BUILD_LINUX: echo $BOOST_ROOT
          CIBW_BEFORE_ALL_LINUX: >
            yum install -y wget &&
            cd / &&  mkdir boost && cd boost &&
            wget --no-check-certificate 'https://sourceforge.net/projects/boost/files/boost/1.76.0/boost_1_76_0.tar.bz2' && 
            tar xf boost_1_76_0.tar.bz2 &&
            cd {package}
          CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
          CIBW_TEST_REQUIRES: pytest
          CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT='${{ steps.install-boost.outputs.BOOST_ROOT }}'
          CIBW_ENVIRONMENT_MACOS: BOOST_ROOT='${{ steps.install-boost.outputs.BOOST_ROOT }}'
          CIBW_ENVIRONMENT_LINUX: BOOST_ROOT=/boost
          CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

I don't know if I did a mistake, but I was not able to set BOOST_ROOT without using CIBW_ENVIRONMENT_*`, because it would overwrite the CIBW_ENVIRONMENT_LINUX.

I am still not sure if this solution is the best on the long term, but at least it work for the moment. It also download a 110MB file, I will look if I can find a way to cache it.

I will not migrate to use a configuration file as you suggested.

Thank you @henryiii for your suggestions and help, I don't know if I came with the best solution, but at least it works for now, I can start improving it.

marcmgame · 2023-11-08T18:25:54Z

marcmgame
Nov 8, 2023

I am having the same problem as described above, but in my case I do need to install the Boost library. Is there a solution to use Boost with cibuildwheel? Is there a repo example where this is done?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Generating the wheels for a package that needs Boost #989

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Generating the wheels for a package that needs Boost #989

Uh oh!

MonkeyBreaker Jan 10, 2022

Replies: 2 comments · 4 replies

Uh oh!

Uh oh!

henryiii Jan 11, 2022 Maintainer

Uh oh!

henryiii Jan 11, 2022 Maintainer

Uh oh!

henryiii Jan 11, 2022 Maintainer

Uh oh!

MonkeyBreaker Jan 11, 2022 Author

Uh oh!

MonkeyBreaker Jan 11, 2022 Author

Uh oh!

marcmgame Nov 8, 2023

MonkeyBreaker
Jan 10, 2022

Replies: 2 comments 4 replies

henryiii
Jan 11, 2022
Maintainer

henryiii Jan 11, 2022
Maintainer

henryiii Jan 11, 2022
Maintainer

MonkeyBreaker Jan 11, 2022
Author

MonkeyBreaker Jan 11, 2022
Author

marcmgame
Nov 8, 2023