BLD, CI: Use cibuildwheel to build Emscripten/Pyodide wheels, push nightlies to Anaconda.org (#58647)

* BLD: Add note about keeping jobs in sync

* BLD, CI: Upload Emscripten wheels nightly to Anaconda

* Add configuration for `cibuildwheel`-Pyodide

* Use unreleased `cibuildwheel` in wheels CI job

* Temporarily move config from TOML to env vars

* Rename job, to match update comment

* Try out Pyodide 0.26.1

* Move Pyodide configuration to `pyproject.toml`

* Use cibuildwheel v2.19 + clean up workflow

* Skip a test that uses subprocesses

* Match tests args with other Pyodide tests; use `not single_cpu`

* Bump to cibuildwheel version 2.19.1

* Don't add `cp312` in job name

Co-authored-by: Matthew Roeschke <[email protected]>

* Don't use separate job for Pyodide wheels

* Fix matrix inclusion

* Add separate step, set `CIBW_PLATFORM` to Pyodide

* Add condition for non-Pyodide jobs

* Use just one step, inherit `CIBW_PLATFORM` if not set

Co-Authored-By: Thomas Li <[email protected]>

* Remove condition that skips the step

Co-Authored-By: Thomas Li <[email protected]>

---------

Co-authored-by: Matthew Roeschke <[email protected]>
Co-authored-by: Thomas Li <[email protected]>

Author: 3 people

.github/workflows/unit-tests.yml

@@ -391,6 +391,7 @@ jobs:
         env:
           PYTHON_GIL: 0
 
+  # NOTE: this job must be kept in sync with the Pyodide build job in wheels.yml
   emscripten:
     # Note: the Python version, Emscripten toolchain version are determined
     # by the Pyodide version. The appropriate versions can be found in the

.github/workflows/wheels.yml

@@ -100,6 +100,13 @@ jobs:
         - [windows-2022, win_amd64]
         # TODO: support PyPy?
         python: [["cp310", "3.10"], ["cp311", "3.11"], ["cp312", "3.12"]]
+
+        # Build Pyodide wheels and upload them to Anaconda.org
+        # NOTE: this job is similar to the one in unit-tests.yml except for the fact
+        # that it uses cibuildwheel instead of a standard Pyodide xbuildenv setup.
+        include:
+          - buildplat: [ubuntu-22.04, pyodide_wasm32]
+            python: ["cp312", "3.12"]
     env:
       IS_PUSH: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
       IS_SCHEDULE_DISPATCH: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
@@ -146,6 +153,7 @@ jobs:
         env:
           CIBW_PRERELEASE_PYTHONS: True
           CIBW_BUILD: ${{ matrix.python[0] }}-${{ matrix.buildplat[1] }}
+          CIBW_PLATFORM: ${{ matrix.buildplat[1] == 'pyodide_wasm32' && 'pyodide' || 'auto' }}
 
       - name: Set up Python
         uses: mamba-org/setup-micromamba@v1

pandas/tests/test_common.py

@@ -7,6 +7,8 @@
 import numpy as np
 import pytest
 
+from pandas.compat import WASM
+
 import pandas as pd
 from pandas import Series
 import pandas._testing as tm
@@ -233,6 +235,7 @@ def test_temp_setattr(with_exception):
     assert ser.name == "first"
 
 
+@pytest.mark.skipif(WASM, reason="Can't start subprocesses in WASM")
 @pytest.mark.single_cpu
 def test_str_size():
     # GH#21758

pyproject.toml

@@ -177,6 +177,16 @@ test-command = ""
 select = "*-macosx*"
 environment = {CFLAGS="-g0"}
 
+[[tool.cibuildwheel.overrides]]
+select = "*pyodide*"
+test-requires = "pytest>=7.3.2 hypothesis>=6.46.1"
+# Pyodide repairs wheels on its own, using auditwheel-emscripten
+repair-wheel-command = ""
+test-command = """
+  PANDAS_CI='1' python -c 'import pandas as pd; \
+  pd.test(extra_args=["-m not clipboard and not single_cpu and not slow and not network and not db", "--no-strict-data-files"]);' \
+  """
+
 [tool.ruff]
 line-length = 88
 target-version = "py310"