Use cibuildwheel in the build

[mattip]

We should look at cibuildwheel and possibly removing the complicated scripting subrepos. We actually use very little of multibuild, mainly to build in dockers for linux. Instead, cibuildwheel could do this for us. We also use gfortran compilers from gfortran_install on macos, but the subrepo carries an unused achives directory. We could vendor the single gfortran_utils.sh script file instead.

[ffgan]

@mattip

Hi there. I've recently been trying to use cibuildwheel to accomplish the above-mentioned tasks. Overall, I was able to successfully complete the CI process, with builds passing for both Linux and macOS. See the CI here.

Below is a detailed introduction to the refactoring process.

1. Basic Principles

As mentioned in the current issue, the goal is to remove the related git submodules, namely multibuild and gfortran-install. Therefore, my objective is to directly remove these two repos while ensuring the CI passes and making as few changes as possible to avoid excessive modifications during PR that would increase review difficulty.

I extracted the scripts used from these two repositories and placed them in this repo - there aren't too many. Mainly gfortran_utils.sh and scripts assembled from multibuild, along with significant modifications to the workflow in posix.yml to use cibuildwheel.

The current scripts contain some redundant code and some code that doesn't fully utilize cibuildwheel. This is expected, as these can be gradually cleaned up after stably switching to cibuildwheel.

2. Implementation Process

I first considered dividing by Linux/macOS platforms, supporting Linux first, then considering macOS after completing Linux support. Therefore, we'll discuss two parts:

1. Using cibuildwheel in the current repository to complete Linux builds
2. Based on the completed work above, adding macOS adaptation support

2.1 Building Linux Wheels Using cibuildwheel

Specific modifications can be seen here

I first considered making major changes to posix.yml.

From posix.yml, I observed the following basic processes:

1. Build OpenBLAS and obtain OpenBLAS artifacts
2. Build wheels based on artifacts and the current repo
3. Install and test the built wheels on the current platform

For the first step, for Linux, pypa/manylinux images are used as the build environment. Through volume mounting, the current directory is mounted into the container for building. After building, successfully built OpenBLAS can be seen in libs/*tar.gz. This mainly executes tools/docker_build_wrap.sh to complete the corresponding work. The second step similarly uses pypa/manylinux images, with the corresponding script being tools/build_wheel.sh. The final step performs the relevant tests. Subsequent artifact and wheel uploads are inconsequential.

To use cibuildwheel for building, my main consideration was to simplify posix.yml as much as possible, moving all these scripts out of posix.yml and completing the above work through cibuildwheel's before-build, repair-wheel-command, and test-command. Since the previous script stages were fairly clear, only simple modifications were needed to complete the refactoring. Worth mentioning here is that building OpenBLAS is completed in the before-build stage within the container, followed immediately by wheel building, and the container only exits and is destroyed after completing wheel building and testing. This means building OpenBLAS and building wheels are completed in the same container, rather than the previous approach of building OpenBLAS, exiting the container, then building wheels in a new container, then exit the container.

Simultaneously, I also moved some scripts from multibuild and gfortran into the current repo, removing dependencies on these two submodules.

After completing the above refactoring, only the OpenBLAS submodule dependency remains. At the same time, all wheel builds for the Linux platform can be completed smoothly.

2.2 Adapting macOS Build Support

1. Upgrading Xcode to 16.0

Before starting, I need to mention something off-topic: the current openblas-libs macOS build CI workflow failed, mainly because Xcode 15.4 installation failed. It can be observed that recent CIs for posix.yml all have the same commit 970cb75ec1885e4e750ee6ea13f329bd8bf29b7c, while previous CIs were successful.

Through research, I learned that GitHub Actions runners recently removed support for 15.4, see issue.

Therefore, we need to upgrade Xcode to 16.0. I performed the upgrade here, and the CI results show that upgrading resolves the current openblas-libs macOS build failure issue.

The above CI is for the original openblas-libs repo, not my fork. In my fork, Xcode also needs to be upgraded to complete macOS build support.

2. cibuildwheel Supporting macOS Builds

Previously mentioned, I completed support for Linux builds using cibuildwheel.

Based on this, I switched to another branch macos-build, adding support for macOS builds with cibuildwheel on this branch.

The macOS build process is basically consistent with Linux. However, macOS builds should be executed locally, so container-related configuration is not needed. This is consistent with cibuildwheel - cibuildwheel's macOS builds are also performed locally. cibuildwheel automatically completes Python installation, virtual environment creation, and other operations.

Overall, support for macOS was relatively smooth.

2.3 Wheel Tags

The wheels built by the current repository initially have "py3-none-any" tags, but this isn't correct because OpenBLAS is embedded and built for specific platforms. So there's subsequently a repair wheel operation.

However, since cibuildwheel doesn't support building wheels that are generated as "any", using cibuildwheel without any modifications will result in cibuildwheel: Build failed because a pure Python wheel was generated. after building the wheel but before repairing it. cibuildwheel indicates that if building pure Python wheels, operations like directly build should be used instead of cibuildwheel, see this issue.

To enable openblas-libs to use cibuildwheel, I made modifications to setup.py, making the generated wheels not "any". I also modified the tag processing logic in the current repo to successfully pass cibuildwheel and ultimately pass repair wheel smoothly. The final CI results show the modifications are effective, maintaining consistency with all wheel tags before refactoring.

3. Some Potential Issues

1. Correctness of Refactoring

Regarding whether the actually built wheels are correct, I haven't yet thought of a better method to verify this. Currently, I'm more focused on ensuring the entire CI process runs through, OpenBLAS build shows all tests passed, then wheels with tags consistent with pre-refactoring can be built based on OpenBLAS, then executing wheel installation, and after installation simply checking whether the OpenBLAS wheel can be imported normally. Finally, ensuring both wheels and OpenBLAS artifacts can be uploaded smoothly. These are the operations I've currently confirmed execute successfully.

Since I didn't see more wheel-related tests in the original repository, I'm quite concerned about whether my refactored wheels can maintain consistent behavior with pre-refactored wheels. If you have more methods, please feel free to contact me directly - I'd be happy to consider relevant suggestions.

2. The Role of DOCKER_TEST_IMAGE in posix.yml

I noticed a rather special variable DOCKER_TEST_IMAGE in posix.yml. Based on literal understanding, it should be the image used for testing. In .travis.yml, this might be used as the base image for testing in Travis CI? However, in posix.yml, this image doesn't seem to be directly used. In the action run logs of the current repo, taking this as an example, I searched in raw logs and found this variable's value is multibuild/xenial_x86_64, but didn't find operations like docker pull or docker run for this image.

The only such operations are for quay.io/pypa/manylinux2014_x86_64. Its role is to build OpenBLAS, corresponding to executing /io/tools/docker_build_wrap.sh, then building wheels, corresponding to executing /openblas/tools/build_wheel.sh.

This doesn't involve pulling or using the DOCKER_TEST_IMAGE. Could you tell me where DOCKER_TEST_IMAGE would be used in posix.yml?

After completing the refactoring, I deleted this variable and didn't find any other impact on the CI workflow. So I'm quite puzzled about this variable's role.

I know it's used for testing in multibuild, but it doesn't appear to be used in this repo.

3. Code License Issues

Since I directly/indirectly used most of the code from multibuild and gfortran-install, how should I annotate the relevant licensing issues? Since I'm not a license expert, I don't know if directly raising a PR would violate relevant license declarations. This is not what I expect - I hope to respect the code licenses of the original repos.

4. Other Information

Due to the above issues still existing, I haven't directly raised a PR for this issue, nor have I organized my modifications accordingly. Therefore, those modifications are just a demonstration showing that cibuildwheel can complete Linux/macOS builds for openblas-libs. I also believe sufficient communication can be more conducive to PR merging, so I made this reply in the issue.

If there's anything incorrect in what I said above, please point it out directly - I'm happy to make modifications to resolve this issue as soon as possible.

---

Co-authored by: nijincheng@iscas.ac.cn;

[mattip]

#217 was merged. Still todo:

• convert travis to use cibuildwheel
• use cibuildwheel in the windows builds

[mattip]

Could you tell me where DOCKER_TEST_IMAGE would be used

It is not used anymore. It can be removed.

I'm quite concerned about whether my refactored wheels can maintain consistent behavior

The real test is if the wheels can be used by NumPy/SciPy. This requires building the wheels and then testing them in the downstream user's CI. The price of a bad version is quite low, since NumPy/SciPy pin to a specific version of the wheels. If we really wanted to test the wheels, we could add here

• get an sdist of NumPy
• create a venv
• install the dependecies
• install the 32-bit and 64-bit wheels,
• for each variant
  • spin build --with-scipy-openblas=[64 or 32]
  • spin test

[ffgan]

#217 was merged. Still todo:

• convert travis to use cibuildwheel
• use cibuildwheel in the windows builds

For supporting the Windows series, there aren't too many modifications needed. However, during the modification process, I found several points that are rather troublesome.

1. Regarding Shell Usage

This is slightly troublesome on Windows. On Linux/macOS, we can use bash or zsh and the like. But in Windows CI, even if bash is specified, it may be switched to use pwsh instead. Since pwsh and bash have different syntax, this requires extra attention when writing CI scripts for Windows. At the same time, I noticed that the current project also has a mix of bat scripts. If we use them together on Windows, it will lead to quite confusing behavior. For example, environment variable passing is one such issue.
If possible, could we consider putting Windows-related scripts in a separate folder, distinguishing them from those bash scripts, to improve maintainability?

2. Regarding Script Storage Location

During the refactoring process, I think we should divide the scripts into two folders for storage: one folder for building OpenBLAS, and another folder for building wheels. Before building wheels, we first uniformly build OpenBLAS. Separating them this way can make the project structure much clearer.

3. Legacy Code Refactoring

As mentioned before, in order to minimize the PR review difficulty as much as possible, I only simply modified most parts of the code to adapt to cibuildwheel. There are quite a few operations that should actually be handled by cibuildwheel/pyproject.toml, i.e., modifying some logic of the existing scripts. For example, on unsupported platforms, can we ensure that wheels built from sdist remain usable? That is, without going through CI, if we want to achieve this, we may need to significantly modify the current scripts. Or should we divide them into different CI scripts according to Linux/macOS platforms to improve maintainability?

[rgommers]

Most of those questions are for @mattip to answer, but a few comments:

But in Windows CI, even if bash is specified, it may be switched to use pwsh instead.

shell: bash should be reliable in GitHub Actions, but you need to specify it in every job step that needs it.

For example, on unsupported platforms, can we ensure that wheels built from sdist remain usable?

There isn't, nor should there be, an sdist. Uploading one would definitely not be reliable, and isn't all that useful since these wheels are mostly meant for vendoring by numpy/scipy rather than direct usage.

[ffgan]

There isn't, nor should there be, an sdist. Uploading one would definitely not be reliable, and isn't all that useful since these wheels are mostly meant for vendoring by numpy/scipy rather than direct usage.

Thank you for your reply. What I mean is, if numpy doesn't have a wheel for the relevant supported platform, it will trigger a build from sdist. Does this process depend on our current repo, and will the default local build behavior of our current repo break the usability of the wheel?

[mattip]

We will not upload an sdist to PyPI, so platforms without a binary wheel will not be able to use this repo.