Support Pyodide 0.28 and bump to Python >=3.13

.github/workflows/main.yml

@@ -9,6 +9,9 @@ on:
 permissions:
   contents: read
 
+env:
+  FORCE_COLOR: 3
+
 jobs:
   test:
     runs-on: ubuntu-latest
@@ -20,10 +23,10 @@ jobs:
           fetch-tags: true
           persist-credentials: false
 
-      - name: Set up Python 3.10
+      - name: Set up Python 3.12
         uses: actions/setup-python@42375524e23c412d93fb67b49958b491fce71c38 # v5.4.0
         with:
-          python-version: "3.10"
+          python-version: "3.12"
 
       - name: Install dependencies
         run: |
@@ -35,15 +38,14 @@ jobs:
 
       - uses: actions/setup-node@1d0ff469b7ec7b3cb9d8673fde0c81c44821de2a # v4.2.0
         with:
-          node-version: "20.6.1"
+          node-version: "22"
 
       - name: Install the pyodide npm package
-        run: |
-          npm install pyodide@0.24.1
+        run: npm install pyodide@0.27.3
 
       - name: Test package + examples
         run: |
-          pytest --cov=pyodide_pack -n 2 --cov-report=xml
+          pytest --cov=. -n auto --cov-report=xml
 
       - name: Upload coverage
         uses: codecov/codecov-action@0565863a31f2c772f9f0395002a31e3f06189574 # v5.4.0

.readthedocs.yml

@@ -12,6 +12,6 @@ python:
       path: .
 
 build:
-  os: ubuntu-20.04
+  os: ubuntu-22.04
   tools:
-    python: "3.11"
+    python: "3.12"

CHANGELOG.md

@@ -12,8 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
    `pyproject.toml` files
    [#35](https://github.com/pyodide/pyodide-pack/pull/35)
 
-
- - Add `pyidide minify` command to minify the Python packages with AST rewrites by,
+ - Add `pyodide minify` command to minify the Python packages with AST rewrites by,
    removing comments and docstrings
    [#23](https://github.com/pyodide/pyodide-pack/pull/23)
 
@@ -31,8 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
- - Added support for Pyodide 0.24.0. This is now the minimal supported version of Pyodide.
-   [#26](https://github.com/pyodide/pyodide-pack/pull/26)
+ - Added support for Pyodide 0.27.3. This is now the minimal supported version of Pyodide.
+   [#57](https://github.com/pyodide/pyodide-pack/pull/57)
 
 ## Fixed
 

README.md

@@ -15,16 +15,16 @@ Each of these approaches have different tradeoffs, and can be used separately or
 
 ## Install
 
-Pyodide-pack requires Python 3.10+ and can be installed via pip:
+`pyodide-pack` requires Python >=3.12 and can be installed via `pip`:
 
 ```
 pip install pyodide-pack
 ```
 
-(optionally) For elimation of unused modules via runtime detection, run NodeJS  needs to be installed together with Pyodide 0.24.0+:
+(optionally) For elimation of unused modules via runtime detection, run Node.js, needs to be installed together with Pyodide 0.27.3 and later:
 
 ```bash
-npm install pyodide@">=0.24.0"
+npm install pyodide@">=0.27.3"
 ```
 
 ## Usage

docs/conf.py

@@ -26,7 +26,7 @@
     "sphinx_autodoc_typehints",
 ]
 intersphinx_mapping = {
-    "python": ("https://docs.python.org/3.10", None),
+    "python": ("https://docs.python.org/3.12", None),
     "pyodide": ("https://pyodide.org/en/stable/", None),
 }
 

docs/module-elimination-at-runtime.md

@@ -17,61 +17,64 @@
 
 2. Create the package bundle,
 
-   ```bash
-   pyodide pack examples/pandas/app.py
-   ```
-   which would produce the following output
+    ```bash
+    pyodide pack examples/pandas/app.py
+    ```
+
+    which would produce the following output:
+
+    ```
+    Running pyodide-pack on examples/pandas/app.py
+
+    Note: unless otherwise specified all sizes are given for gzip compressed files to be representative of CDN compression.
+
+    Loaded requirements from: examples/pandas/requirements.txt
+    Running the input code in Node.js to detect used modules..
+
+    [...]
+
+    Done input code execution in 6.2 s
+
+    Using stdlib (554 files) with a total size of 2.29 MB.
+    Detected 5 dependencies with a total size of 9.34 MB  (uncompressed: 37.62 MB)
+    In total 487 files and 0 dynamic libraries were accessed.
+    Total initial size (stdlib + dependencies): 11.63 MB
+
+
+                                            Packing..
+    ┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
+    ┃ No ┃ Package                        ┃ All files ┃ .so libs ┃   Size (MB) ┃ Reduction ┃
+    ┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
+    │  0 │ stdlib                         │ 554 → 168 │          │ 2.29 → 0.46 │    79.8 % │
+    │  1 │ numpy-2.0.2-cp312-cp312-pyodi… │  338 → 96 │  19 → 12 │ 3.05 → 2.16 │    29.3 % │
+    │  2 │ pandas-2.2.3-cp312-cp312-pyod… │ 396 → 295 │  44 → 42 │ 5.61 → 4.52 │    19.5 % │
+    │  3 │ python_dateutil-2.9.0.post0-p… │   25 → 14 │    0 → 0 │ 0.23 → 0.18 │    21.6 % │
+    │  4 │ pytz-2024.1-py2.py3-none-any.… │   615 → 5 │    0 → 0 │ 0.43 → 0.01 │    97.8 % │
+    │  5 │ six-1.16.0-py2.py3-none-any.w… │     6 → 1 │    0 → 0 │ 0.01 → 0.01 │    41.8 % │
+    └────┴────────────────────────────────┴───────────┴──────────┴─────────────┴───────────┘
+    Wrote pyodide-package-bundle.zip with 6.99 MB (25.2% reduction)
+
+    Spawning webserver at http://127.0.0.1:52009 (see logs in /tmp/tmpx0ktv9fw/http-server.log)
+    Running the input code in Node.js to validate bundle..
+
+            Validating and benchmarking the output bundle..
+    ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
+    ┃ Step                 ┃ Load time (s) ┃ Fraction of load time ┃
+    ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
+    │ loadPyodide          │          0.82 │                34.6 % │
+    │ fetch_unpack_archive │          0.14 │                 5.9 % │
+    │ load_dynamic_libs    │          0.11 │                 4.6 % │
+    │ import_run_app       │          1.30 │                54.9 % │
+    │ TOTAL                │          2.36 │                 100 % │
+    └──────────────────────┴───────────────┴───────────────────────┘
+
+    Total output size (stdlib + packages): 7.45 MB (35.9% reduction)
+
+    Bundle validation successful.
+    ```
 
-   ```
-   Running pyodide-pack on examples/pandas/app.py
-
-   Note: unless otherwise specified all sizes are given for gzip compressed files to
-   be representative of CDN compression.
-
-   Loaded requirements from: examples/pandas/requirements.txt
-   Running the input code in Node.js to detect used modules..
-
-   [...]
-   Done input code execution in 3.8 s
-
-   Using stdlib (547 files) with a total size of 2.25 MB.
-   Detected 5 dependencies with a total size of 8.92 MB  (uncompressed: 35.46 MB)
-   In total 487 files and 0 dynamic libraries were accessed.
-   Total initial size (stdlib + dependencies): 11.17 MB
-
-
-                                          Packing..
-   ┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
-   ┃ No ┃ Package                        ┃ All files ┃ .so libs ┃   Size (MB) ┃ Reduction ┃
-   ┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
-   │  0 │ stdlib                         │ 547 → 151 │          │ 2.25 → 0.75 │    66.7 % │
-   │  1 │ numpy-1.25.2-cp311-cp311-emsc… │ 430 → 111 │   19 → 0 │ 3.06 → 2.36 │    23.0 % │
-   │  2 │ pandas-1.5.3-cp311-cp311-emsc… │ 462 → 292 │   42 → 0 │ 5.17 → 4.64 │    10.3 % │
-   │  3 │ python_dateutil-2.8.2-py2.py3… │   25 → 15 │    0 → 0 │ 0.24 → 0.22 │     9.4 % │
-   │  4 │ pytz-2023.3-py2.py3-none-any.… │   614 → 5 │    0 → 0 │ 0.43 → 0.02 │    96.1 % │
-   │  5 │ six-1.16.0-py2.py3-none-any.w… │     6 → 1 │    0 → 0 │ 0.01 → 0.01 │    18.5 % │
-   └────┴────────────────────────────────┴───────────┴──────────┴─────────────┴───────────┘
-   Wrote pyodide-package-bundle.zip with 7.37 MB (17.4% reduction)
-
-   Spawning webserver at http://127.0.0.1:52009 (see logs in /tmp/tmpx0ktv9fw/http-server.log)
-   Running the input code in Node.js to validate bundle..
-
-           Validating and benchmarking the output bundle..
-   ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
-   ┃ Step                 ┃ Load time (s) ┃ Fraction of load time ┃
-   ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
-   │ loadPyodide          │          1.34 │                36.1 % │
-   │ fetch_unpack_archive │          0.27 │                 7.4 % │
-   │ load_dynamic_libs    │          0.00 │                 0.1 % │
-   │ import_run_app       │          2.10 │                56.5 % │
-   │ TOTAL                │          3.72 │                 100 % │
-   └──────────────────────┴───────────────┴───────────────────────┘
-
-   Total output size (stdlib + packages): 8.12 MB (27.3% reduction)
-
-   Bundle validation successful.
-   ```
 3. Load your Python web application with,
+
    ```js
    let pyodide = await loadPyodide({fullStdLib: false});
 

examples/netcdf4/app.py

@@ -0,0 +1,9 @@
+import netCDF4 as nc
+import numpy as np
+
+dataset = nc.Dataset("memory.nc", "w", diskless=True, persist=False)
+dataset.createDimension("x", 3)
+var = dataset.createVariable("data", np.float32, ("x",))
+var[:] = np.array([1.0, 2.0, 3.0])
+print(f"Data: {var[:]}")
+dataset.close()

examples/netcdf4/pyproject.toml

@@ -0,0 +1,5 @@
+[tool.pyodide_pack]
+requires = [
+    "numpy",
+    "netcdf4",
+]

examples/scikit-learn/pyproject.toml

@@ -1,5 +1,4 @@
 [tool.pyodide_pack]
 requires = [
-    "numpy",
     "scikit-learn"
 ]

pyodide_pack/ast_rewrite.py

@@ -54,7 +54,7 @@ def _strip_module_docstring(tree: ast.Module) -> ast.Module:
     if (
         tree.body
         and isinstance(expr := tree.body[0], ast.Expr)
-        and isinstance(expr.value, ast.Str | ast.Constant)
+        and isinstance(expr.value, ast.Constant)
         and isinstance(expr.value.value, str)
     ):
         tree.body.pop(0)

pyodide_pack/config.py

@@ -1,16 +1,11 @@
 from __future__ import annotations
 
+import tomllib
 from pathlib import Path
 from typing import Any
 
 from pydantic import BaseModel, ConfigDict
 
-try:
-    import tomllib
-except ImportError:
-    # Python <3.11
-    import tomli as tomllib  # type: ignore[no-redef]
-
 
 def _find_pyproject_toml(input_path: Path) -> Path | None:
     """Find a `pyproject.toml` in any of the parent dirs"""

pyodide_pack/js/discovery.js

@@ -99,7 +99,11 @@ async function main() {
 	await micropip.install({{packages}});
   }
 
+  // Explicitly import the cp437 encoding to ensure it's included in the bundle.
+  // It's required for zipfiles to work as decoding can rely on it, see:
+  // https://github.com/python/cpython/blob/0b05ead877f909b7efe712db758012d9dbece7ce/Lib/zipfile/__init__.py#L1457
   await pyodide.runPythonAsync(`
+import encodings.cp437
 {{ code }}
 `);
   // Run code used in the loader

pyodide_pack/loader/pyodide_pack_loader.py

@@ -3,8 +3,34 @@
 
 async def setup():
     """Load dynamic libraries in the pyodide-pack bundle"""
-    from pyodide_js import _module
+    try:
+        from pyodide_js import _module
 
-    for paths in Path("/bundle-so-list.txt").read_text().splitlines():
-        path, is_shared = paths.split(",")
-        await _module.API.loadDynlib(path, bool(is_shared))
+        # We need to handle duplicate shared libraries carefully. Many packages
+        # that depend on static/shared libraries like h5py and netcdf4 may include
+        # them (such as libhdf5_hl.so); loading them more than once, especially one
+        # by one instead of their dependent order may cause errors.
+
+        loaded_libs = set()
+        so_list_path = Path("/bundle-so-list.txt")
+        if not so_list_path.exists():
+            print(f"Warning: {so_list_path} not found")
+            return
+
+        for line in so_list_path.read_text().splitlines():
+            try:
+                path, is_shared = line.split(",")
+
+                if path in loaded_libs:
+                    continue
+
+                lib_basename = Path(path).name
+                if any(lib_basename == Path(loaded).name for loaded in loaded_libs):
+                    continue
+
+                await _module.API.loadDynlib(path, bool(is_shared))
+                loaded_libs.add(path)
+            except Exception as e:
+                print(f"Warning: Failed to load {path}: {str(e)}")
+    except Exception as e:
+        print(f"Error in setup: {str(e)}")