cibuildwheel-related work (#369)

Author: mara004

.github/workflows/cibuildwheel.yaml

@@ -0,0 +1,123 @@
+# SPDX-FileCopyrightText: 2025 geisserml <[email protected]>
+# SPDX-FileCopyrightText: 2025 wojiushixiaobai <[email protected]>
+# SPDX-License-Identifier: Apache-2.0 OR BSD-3-Clause
+
+# NOTE: This workflow is currently written with a dynamic matrix.
+# Another option would be to extract a reusable "build one" workflow and declare an individual job for each target here.
+
+name: Build with cibuildwheel
+on:
+  workflow_dispatch:
+    inputs:
+      cibw_py_ver:
+        default: 'cp38'
+        type: string
+      linux_main:
+        default: true
+        type: boolean
+      linux_ibm:
+        default: true
+        type: boolean
+      linux_emulated:
+        default: false
+        type: boolean
+      linux_musl:
+        default: true
+        type: boolean
+
+permissions: {}
+
+jobs:
+  
+  prepare_matrix:
+    name: Determine build matrix
+    runs-on: ubuntu-latest
+    outputs:
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+    
+    steps:
+      - name: Run python script that outputs the build matrix
+        id: set-matrix
+        shell: python
+        env:
+          LINUX_MAIN:     ${{ inputs.linux_main     && 1 || 0 }}
+          LINUX_IBM:      ${{ inputs.linux_ibm      && 1 || 0 }}
+          LINUX_EMULATED: ${{ inputs.linux_emulated && 1 || 0 }}
+          LINUX_MUSL:     ${{ inputs.linux_musl     && 1 || 0 }}
+        run: |
+          import os, sys, json
+          
+          LINUX_MAIN     = bool(int( os.environ["LINUX_MAIN"] ))
+          LINUX_IBM      = bool(int( os.environ["LINUX_IBM"] ))
+          LINUX_EMULATED = bool(int( os.environ["LINUX_EMULATED"] ))
+          LINUX_MUSL     = bool(int( os.environ["LINUX_MUSL"] ))
+          
+          matrix = []
+          images = ["manylinux"]
+          if LINUX_MUSL:
+              images.append("musllinux")
+          
+          def job(image, os, arch, emulated=False):
+              matrix.append(dict(
+                  image=image, os=os, arch=arch, emulated=emulated
+              ))
+          
+          def linux_job(os, arch, emulated=False, images=images):
+              for image in images:
+                  job(os, arch, image, emulated)
+          
+          if LINUX_MAIN:
+              linux_job("ubuntu-24.04", "x86_64")
+              linux_job("ubuntu-24.04-arm", "aarch64")
+          if LINUX_IBM:
+              # XXX will become native as soon as we get access to IBM's self-hosted runners
+              linux_job("ubuntu-24.04", "ppc64le", True)  # False
+              linux_job("ubuntu-24.04", "s390x", True)  # False
+          if LINUX_EMULATED:
+              linux_job("ubuntu-24.04", "loongarch64", True)
+              linux_job("ubuntu-24.04", "riscv64", True)
+              if LINUX_MUSL:
+                  # pdfium-binaries don't currently build armv7l for musl (but they do for glibc)
+                  linux_job("ubuntu-24.04", "armv7l", True, images=("musllinux", ))
+          
+          matrix_json = json.dumps(matrix)
+          print(matrix_json, file=sys.stderr)
+          with open(os.environ["GITHUB_OUTPUT"], 'a') as output_fh:
+              print(f"matrix={matrix_json}", file=output_fh)
+  
+  build_wheels:
+    name: Build ${{ matrix.arch }} ${{ matrix.image }} on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    needs: prepare_matrix
+    
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJSON(needs.prepare_matrix.outputs.matrix) }}
+    
+    steps:
+      
+      - name: Check out the repo
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      
+      - name: Set up QEMU
+        if: ${{ matrix.emulated }}
+        uses: docker/setup-qemu-action@v3
+      
+      # Reminder: most configuration is in pyproject.toml so we can use TOML overrides
+      - name: Build wheels
+        uses: pypdfium2-team/[email protected]
+        env:
+          # Will be tagged as not python specific by our setup.py. inputs.cibw_py_ver only controls the version used at build time. Could also use `*`, then cibuildwheel would build with the oldest supported version, and walk through the others but skip because a compatible wheel is around already.
+          CIBW_BUILD: "${{ inputs.cibw_py_ver }}-${{ matrix.image }}_${{ matrix.arch }}"
+          CIBW_ARCHS: ${{ matrix.arch }}
+        with:
+          output-dir: wheelhouse
+      
+      - name: Upload artifact
+        uses: actions/upload-artifact@v4
+        with:
+          path: ./wheelhouse/*.whl
+          name: cibw-${{ matrix.image }}-${{ matrix.arch }}

.github/workflows/conda.yaml

@@ -15,7 +15,7 @@ on:
         default: 'latest'
         type: string
       new_only:
-        # only with package == "raw", ignored otherwise (actually the default should be false in that case, but I don't know if GH supports dynamic defaults depending on other inputs)
+        # only with package == "raw", ignored otherwise (actually the default should be false in that case, but don't know if GH supports dynamic defaults depending on other inputs)
         default: true
         type: boolean
       test:
@@ -95,6 +95,7 @@ jobs:
       fail-fast: false
       matrix:
         # NOTE On GH actions, macOS <=13 is Intel, whereas macOS >=14 will be ARM64
+        # Can't test 'windows-11-arm' because setup-miniconda doesn't support it AOTW
         os: ['ubuntu-latest', 'ubuntu-24.04-arm', 'macos-13', 'macos-latest', 'windows-latest']
         py: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
 

.github/workflows/main.yaml

@@ -118,8 +118,16 @@ jobs:
       fail-fast: false
       matrix:
         # NOTE On GH actions, macOS <=13 is Intel, whereas macOS >=14 will be ARM64
-        os: ['ubuntu-latest', 'ubuntu-24.04-arm', 'macos-13', 'macos-latest', 'windows-latest']
+        os: ['ubuntu-latest', 'ubuntu-24.04-arm', 'macos-13', 'macos-latest', 'windows-latest', 'windows-11-arm']
         py: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
+        exclude:
+          # not supported by setup-python action
+          - os: windows-11-arm
+            py: '3.8'
+          - os: windows-11-arm
+            py: '3.9'
+          - os: windows-11-arm
+            py: '3.10'
         include:
           - os: ubuntu-latest
             wheel: dist/*manylinux_*_x86_64*.whl
@@ -131,13 +139,13 @@ jobs:
             wheel: dist/*macosx_*_arm64*.whl
           - os: windows-latest
             wheel: dist/*win_amd64.whl
+          - os: windows-11-arm
+            wheel: dist/*win_arm64.whl
 
     runs-on: ${{ matrix.os }}
 
     steps:
 
-      - uses: extractions/setup-just@v3
-      
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
@@ -172,7 +180,7 @@ jobs:
           WHEEL: ${{ matrix.wheel }}
 
       - name: Run Test Suite
-        run: just test
+        run: python3 -m pytest tests/
 
 
   publish:

.github/workflows/test_release.yaml

@@ -21,15 +21,21 @@ jobs:
       fail-fast: false
       matrix:
         # NOTE On GH actions, macOS <=13 is Intel, whereas macOS >=14 will be ARM64
-        os: ['ubuntu-latest', 'ubuntu-24.04-arm', 'macos-13', 'macos-latest', 'windows-latest']
+        os: ['ubuntu-latest', 'ubuntu-24.04-arm', 'macos-13', 'macos-latest', 'windows-latest', 'windows-11-arm']
         py: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
+        exclude:
+          # not supported by setup-python action
+          - os: windows-11-arm
+            py: '3.8'
+          - os: windows-11-arm
+            py: '3.9'
+          - os: windows-11-arm
+            py: '3.10'
 
     runs-on: ${{ matrix.os }}
 
     steps:
 
-      - uses: extractions/setup-just@v3
-      
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
@@ -54,4 +60,4 @@ jobs:
         run: python3 -m pip install -U -r req/converters.txt -r req/test.txt
 
       - name: Run tests
-        run: just test
+        run: python3 -m pytest tests/

.github/workflows/test_setup.yaml

@@ -18,15 +18,19 @@ jobs:
       fail-fast: false
       matrix:
         # NOTE On GH actions, macOS <=13 is Intel, whereas macOS >=14 will be ARM64
-        os: ['ubuntu-latest', 'ubuntu-24.04-arm', 'macos-13', 'macos-latest', 'windows-latest']
+        os: ['ubuntu-latest', 'ubuntu-24.04-arm', 'macos-13', 'macos-latest', 'windows-latest', 'windows-11-arm']
         py: ['3.9', '3.10', '3.11', '3.12', '3.13']
+        exclude:
+          # not supported by setup-python action
+          - os: windows-11-arm
+            py: '3.9'
+          - os: windows-11-arm
+            py: '3.10'
 
     runs-on: ${{ matrix.os }}
 
     steps:
 
-      - uses: extractions/setup-just@v3
-      
       # AOTW, the slsa-verifier GH action does not support anything but Ubuntu x86_64.
       - name: slsa-verifier
         if: ${{ startsWith(matrix.os, 'ubuntu') && !endsWith(matrix.os, '-arm') }}
@@ -61,7 +65,7 @@ jobs:
         run: python3 -m pip install -v --no-build-isolation -e .
 
       - name: Build docs
-        run: just docs-build
+        run: python3 -m sphinx -b html docs/source docs/build/html
 
       - name: Run test suite
-        run: just test
+        run: python3 -m pytest tests/

.github/workflows/test_sourcebuild.yaml

@@ -19,8 +19,9 @@ jobs:
       fail-fast: false
       matrix:
         # On GH actions, macOS <=13 is Intel, whereas macOS >=14 will be ARM64
-        # Google's toolchain doesn't seem to run on Linux arm64 natively. The toolchain-free build (or cross-compilation from x86_64) should work, though.
-        os: ['ubuntu-latest', 'macos-13', 'macos-latest', 'windows-latest']  # 'ubuntu-24.04-arm'
+        # Google's toolchain doesn't seem to run on Linux/Windows arm64 natively. The toolchain-free build (or cross-compilation from x86_64) should work, though.
+        # 'ubuntu-24.04-arm', 'windows-11-arm'
+        os: ['ubuntu-latest', 'macos-13', 'macos-latest', 'windows-latest']
         build_mode: ['toolchained']
         include:
           - os: 'ubuntu-latest'

README.md

@@ -142,7 +142,7 @@ This project comes with two scripts to automate the build process: `build_toolch
 - `build_toolchained` is based on the build instructions in pdfium's Readme, and uses Google's toolchain (this means foreign binaries and sysroots). This results in a heavy checkout process that may take a lot of time and space. By default, this script will use vendored libraries, but you can also pass `--use-syslibs` to try to use system libraries. An advantage of the toolchain is its powerful cross-compilation support (including symbol reversioning).
 - `build_native` is an attempt to address some shortcomings of the toolchained build (mainly a bloated checkout process, and lack of portability). It is tailored towards native compilation, and uses system tools and libraries (including the system's GCC compiler), which must be installed by the caller beforehand. This script should theoretically work on arbitrary Linux architectures. As a drawback, this process is not supported or even documented upstream, so it might be hard to maintain.
 
-You can also set `PDFIUM_PLATFORM` to `sourcebuild-native` or `sourcebuild-toolchained` to trigger either build script through setup.
+You can also set `PDFIUM_PLATFORM` to `sourcebuild-native` or `sourcebuild-toolchained` to trigger either build script through setup, and pass command-line flags with `$BUILD_PARAMS`.
 However, for simplicity, both scripts/subtargets share just `sourcebuild` as staging directory.
 
 Dependencies:
@@ -160,7 +160,7 @@ PDFIUM_PLATFORM="sourcebuild" python -m pip install -v .
 Or for the native build, on Ubuntu 24.04, you could do e.g.:
 ```bash
 # Install dependencies
-sudo apt-get install generate-ninja ninja-build libfreetype-dev liblcms2-dev libjpeg-dev libopenjp2-7-dev libpng-dev zlib1g-dev libicu-dev libtiff-dev libglib2.0-dev
+sudo apt-get install generate-ninja ninja-build libfreetype-dev liblcms2-dev libjpeg-dev libopenjp2-7-dev libpng-dev libtiff-dev zlib1g-dev libicu-dev libglib2.0-dev
 ```
 ```bash
 # Build with GCC
@@ -180,9 +180,50 @@ python ./setupsrc/pypdfium2_setup/build_native.py --compiler clang
 PDFIUM_PLATFORM="sourcebuild" python -m pip install -v .
 ```
 
+Note, on *some* platforms, you might also need symlinks for GCC, e.g.:
+```bash
+PREFIX=$(python ./utils/get_gcc_prefix.py)  # in pypdfium2 dir
+GCC_DIR="/usr"  # or e.g. /opt/rh/gcc-toolset-14/root
+sudo ln -s $GCC_DIR/bin/gcc $GCC_DIR/bin/$PREFIX-gcc
+sudo ln -s $GCC_DIR/bin/g++ $GCC_DIR/bin/$PREFIX-g++
+sudo ln -s $GCC_DIR/bin/nm $GCC_DIR/bin/$PREFIX-nm
+sudo ln -s $GCC_DIR/bin/readelf $GCC_DIR/bin/$PREFIX-readelf
+```
+
 > [!TIP]
 > By default, the build scripts will create separate DLLs for vendored dependency libraries (e.g. `abseil`). However, if you want to bundle everything into a single DLL, pass `--single-lib`.
 
+> [!NOTE]
+> The native sourcebuild currently supports Linux (or similar).
+> macOS and Windows are not handled, as we do not have access to these systems, and working over CI did not turn out feasible – use the toolchain-based build for now.
+> Community help / pull requests to extend platform support would be welcome.
+
+##### cibuildwheel
+
+The native sourcebuild can be run through cibuildwheel. For targets configured in our [`pyproject.toml`](./pyproject.toml), the basic invocation is as simple as p.ex.
+```bash
+CIBW_BUILD="cp311-manylinux_x86_64" cibuildwheel
+```
+
+See also our [cibuildwheel workflow](.github/workflows/cibuildwheel.yaml).
+For more options, see the [upstream documentation](https://cibuildwheel.pypa.io/en/stable/options).
+
+Note that, for Linux, cibuildwheel requires Docker. On the author's version of Fedora, it can be installed as follows:
+```bash
+sudo dnf in moby-engine  # this provides the docker command
+sudo systemctl start docker
+sudo systemctl enable docker
+sudo usermod -aG docker $USER
+# then reboot (re-login might also suffice)
+```
+For other ways of installing Docker, refer to the cibuildwheel docs ([Setup](https://cibuildwheel.pypa.io/en/stable/setup/), [Platforms](https://cibuildwheel.pypa.io/en/stable/platforms/)) and the links therein.
+
+> [!WARNING]
+> cibuildwheel copies the project directory into a container, not taking `.gitignore` rules into account.
+> Thus, it is advisable to make a fresh checkout of pypdfium2 before running cibuildwheel.
+> In particular, a toolchained checkout of pdfium within pypdfium2 is problematic, and will cause a halt on the `Copying project into container...` step.
+> For development, make sure the fresh checkout is in sync with the working copy.
+
 ##### Android (Termux)
 
 The native build may also work on Android with Termux in principle.
@@ -310,7 +351,7 @@ Disclaimer: As it is hard to keep up with constantly evolving setup code, it is
     + If unset or `auto`, the host platform is detected and a corresponding binary will be selected.
     + If an explicit platform identifier (e.g. `linux_x64`, `darwin_arm64`, ...), binaries for the requested platform will be used.[^platform_ids]
     + If `system-search`, look for and bind against system-provided pdfium instead of embedding a binary. If just `system`, consume existing bindings from `data/system/`.
-    + If `sourcebuild`, binary and bindings will be taken from `data/sourcebuild/`, assuming a prior run of the native or toolchained build scripts. `sourcebuild-native` or `sourcebuild-toolchained` can also be used to trigger either build through setup. However, triggering on the caller side is preferred as this allows to pass custom options.
+    + If `sourcebuild`, binary and bindings will be taken from `data/sourcebuild/`, assuming a prior run of the native or toolchained build scripts. `sourcebuild-native` or `sourcebuild-toolchained` can also be used to trigger either build through setup (use `$BUILD_PARAMS` to pass custom options).
     + If `sdist`, no platform-specific files will be included, so as to create a source distribution.
 
 * `$PYPDFIUM_MODULES=[raw,helpers]` defines the modules to include. Metadata adapts dynamically.
@@ -954,7 +995,6 @@ Additionally, one doc build can also be hosted on [GitHub Pages](https://pypdfiu
 It is implemented with a CI workflow, which is supposed to be triggered automatically on release.
 This provides us with full control over build env and used commands, whereas RTD may be less liberal in this regard.
 
-
 ### Testing
 
 pypdfium2 contains a small test suite to verify the library's functionality. It is written with [pytest](https://github.com/pytest-dev/pytest/):
@@ -984,10 +1024,28 @@ find . -name '*.pdf' -exec bash -c "echo \"{}\" && pypdfium2 toc \"{}\"" \;
 
 [^testing_corpora]: For instance, one could use the testing corpora of open-source PDF libraries (pdfium, pikepdf/ocrmypdf, mupdf/ghostscript, tika/pdfbox, pdfjs, ...)
 
+### Adding a new workflow
+
+When writing a new workflow, it is usually desirable to test in a branch first before merging into main.
+However, new workflows from branches cannot be dispatched from the GitHub Actions panel yet. That's why you'll want to use the [`gh`](https://cli.github.com/) command-line tool, as follows:
+```bash
+gh workflow run $WORKFLOW_NAME.yaml --ref $MY_BRANCH
+```
+If inputs are needed, JSON can be used
+```bash
+echo '{"my_json_info":1, "my_var":"hello"}' | gh workflow run $WORKFLOW_NAME.yaml --ref $MY_BRANCH --json
+# real-world example
+echo '{"cibw_py_ver":"cp38", "linux_main":"true", "linux_ibm":"false", "linux_emulated":"false", "linux_musl":"true"}' | gh workflow run cibuildwheel.yaml --ref cibuildwheel --json
+```
+You should pass the complete set of fields here, defaults might not be recognized with this form of dispatch.
+
+> [!IMPORTANT]
+> You need to be in the pypdfium2 directory for this to work. Otherwise, the request will be silently ignored.
+
 ### Release workflow
 
 The release process is fully automated using Python scripts and scheduled release workflows.
-You may also trigger the workflow manually using the GitHub Actions panel or the [`gh`](https://cli.github.com/) command-line tool.
+You may also trigger the workflow manually from the GitHub Actions panel or similar.
 
 Python release scripts are located in the folder `setupsrc/pypdfium2_setup`, along with custom setup code:
 * `update.py` downloads binaries.
@@ -1038,10 +1096,10 @@ If something went wrong with commit or tag, you can still revert the changes:
 # perform an interactive rebase to change history (substitute $N_COMMITS with the number of commits to drop or modify)
 git rebase -i HEAD~$N_COMMITS
 git push --force
-# delete local tag (substitute $TAGNAME accordingly)
-git tag -d $TAGNAME
-# delete remote tag
+# delete remote tag (substitute $TAGNAME accordingly)
 git push --delete origin $TAGNAME
+# delete local tag
+git tag -d $TAGNAME
 ```
 Faulty PyPI releases may be yanked using the web interface.