Ability to install Python dependencies from system instead of PyPI during test phase?

[mara004]

I'm trying to build a C extension package for some exotic architectures, and run its test suite within cibuildwheel.
The test suite in turn has binary extension dependencies (pillow and numpy), which are not available on PyPI for the archs in question.

However, I think they are provided by the container image, and could be installed with the system package manager.
Now, I tried doing that with CIBW_BEFORE_TEST, but found that cibuildwheel sets up a venv for testing, where system python packages are not recognized.

Could an option be added to cibuildwheel to pass --system-site-packages when creating the venv?
Or should I manually set the PYTHONPATH or something?

[joerick]

I think I'd probably try setting PYTHONPATH in test-environment. I'm guessing you're using a custom container image, so you probably know the exact paths where the system packages are installed.

If you're building multiple versions of python, you'll need to use TOML overrides to set a different path for each python version.

[mara004]

Thanks for the pointers, I'll give that a try.

It's not actually a custom image, just cibuildwheel-managed ppc64le and s390x so far, but I should be able to figure out the paths.
Also tried riscv64, but didn't work because one of our build dependencies (gn) is not available in the riscv64 container.
For loongarch64 I plan to use https://github.com/loong64/cibuildwheel/tree/loong64-v3.1.3 which probably uses a custom image.

On the other hand, I'm already wondering if we wouldn't be better off doing something custom with cross-compilation, as emulation is so slow...

[mara004]

OK, I can confirm export PYTHONPATH="${PYTHONPATH}:/usr/lib64/python3.6/site-packages" does the trick with a current manylinux_2_28 container.

[mara004]

Oh no, in fact it doesn't work. I just forgot to add the && so commands got lost and the test suite wasn't actually executed.
The failure is logical: system packages are built against python 3.6, whereas cibuildwheel only supports python 3.8 and above, but the python version must match up for C extension modules from numpy and PIL to work.
For the record, here is the failure:
https://github.com/pypdfium2-team/pypdfium2/actions/runs/16995896979/job/48186393271

[henryiii]

You would need a custom testing image for each Python version, I believe. We don't have a way to set custom testing images at all currently; I think at that point you could just do the testing in a separate step after cibuildwheel runs.

[mara004]

I think at that point you could just do the testing in a separate step after cibuildwheel runs.

True, but then we'll have to set up test container & emulation on our own, which defeats some of cibuildwheel's convenience...