Skip to content

gh-126187 Add emscripten.py script to automate emscripten build #126190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 9, 2024
Merged

gh-126187 Add emscripten.py script to automate emscripten build #126190

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1b662d7
gh-126187 Add emscripten.py script to automate emscripten build
hoodmane Oct 3, 2024
d0fc9ab
Add a news entry
hoodmane Oct 30, 2024
7eec30d
Add readme, move emscripten.py into subfolder
hoodmane Nov 7, 2024
4ac6836
Remove fix for endianness detection
hoodmane Nov 7, 2024
ac9f92a
Apply suggestions from code review
hoodmane Nov 7, 2024
4e67746
Address review comments
hoodmane Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Build/2024-10-30-17-47-15.gh-issue-126187.0jFCZB.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Introduced ``Tools/wasm/emscripten.py`` to simplify doing Emscripten builds.
164 changes: 28 additions & 136 deletions Tools/wasm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Python WebAssembly (WASM) build

**WASI support is [tier 2](https://peps.python.org/pep-0011/#tier-2).**
**Emscripten is NOT officially supported as of Python 3.13.**
**Emscripten support is [tier 3](https://peps.python.org/pep-0011/#tier-3).**

This directory contains configuration and helpers to facilitate cross
compilation of CPython to WebAssembly (WASM). Python supports Emscripten
Expand All @@ -27,154 +27,57 @@ It comes with a reduced and preloaded stdlib without tests and threading
support. The ``Emscripten/node`` target has threading enabled and can
access the file system directly.

Cross compiling to the wasm32-emscripten platform needs the
[Emscripten](https://emscripten.org/) SDK and a build Python interpreter.
Emscripten 3.1.19 or newer are recommended. All commands below are relative
to a repository checkout.
To cross compile to the ``wasm32-emscripten`` platform you need
[the Emscripten compiler toolchain](https://emscripten.org/),
a Python interpreter, and an install of Node version 18 or newer. Emscripten
3.1.42 or newer are recommended. All commands below are relative to a checkout
of the Python repository.

#### Toolchain
#### Install [the Emscripten compiler toolchain](https://emscripten.org/docs/getting_started/downloads.html)

##### Container image

Christian Heimes maintains a container image with Emscripten SDK, Python
build dependencies, WASI-SDK, wasmtime, and several additional tools.

From within your local CPython repo clone, run one of the following commands:

```
# Fedora, RHEL, CentOS
podman run --rm -ti -v $(pwd):/python-wasm/cpython:Z -w /python-wasm/cpython quay.io/tiran/cpythonbuild:emsdk3

# other
docker run --rm -ti -v $(pwd):/python-wasm/cpython -w /python-wasm/cpython quay.io/tiran/cpythonbuild:emsdk3
You can install the Emscripten toolchain as follows:
```shell
git clone https://github.com/emscripten-core/emsdk.git --depth 1
./emsdk/emsdk install 3.1.68
./emsdk/emsdk activate 3.1.68
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to nominate 3.1.68 specifically, rather than latest? I'm a little wary of putting a "magic number" into documentation unless that number is a minimum (like 3.1.42 above) or a pinned version of some significance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I'll put latest for now and when we add actual CI I'll switch it to the specific version used in CI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of interest - how stable/backwards compatible is "latest"? Is the emscripten version analogous to the manylinux tag/macOS dev version for compatibility purposes? Or is this a situation where we'll pick a specific emscripten version for all versions of a Python release (i.e., all 3.14.x releases are Emscripten 3.1.68)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we care about abi compatibility the safe way to go is to fix a version.

```

##### Manually

###### Install [Emscripten SDK](https://emscripten.org/docs/getting_started/downloads.html)

**NOTE**: Follow the on-screen instructions how to add the SDK to ``PATH``.

To add the Emscripten compiler to your path:
```shell
git clone https://github.com/emscripten-core/emsdk.git /opt/emsdk
/opt/emsdk/emsdk install latest
/opt/emsdk/emsdk activate latest
source ./emsdk/emsdk_env.sh
```
This adds `emcc` and `emconfigure` to your path.

###### Optionally: enable ccache for EMSDK
##### Optionally: enable ccache for EMSDK

The ``EM_COMPILER_WRAPPER`` must be set after the EMSDK environment is
sourced. Otherwise the source script removes the environment variable.

```
. /opt/emsdk/emsdk_env.sh
EM_COMPILER_WRAPPER=ccache
```

###### Optionally: pre-build and cache static libraries

Emscripten SDK provides static builds of core libraries without PIC
(position-independent code). Python builds with ``dlopen`` support require
PIC. To populate the build cache, run:

```shell
. /opt/emsdk/emsdk_env.sh
embuilder build zlib bzip2 MINIMAL_PIC
embuilder --pic build zlib bzip2 MINIMAL_PIC
export EM_COMPILER_WRAPPER=ccache
```


### Compile and build Python interpreter

From within the container, run the following command:

```shell
./Tools/wasm/wasm_build.py build
```

The command is roughly equivalent to:

```shell
mkdir -p builddir/build
pushd builddir/build
../../configure -C
make -j$(nproc)
popd
```

#### Cross-compile to wasm32-emscripten for browser

```shell
./Tools/wasm/wasm_build.py emscripten-browser
```

The command is roughly equivalent to:

You can use `python Tools/wasm/emscripten` to compile and build targetting
Emscripten. You can do everything at once with:
```shell
mkdir -p builddir/emscripten-browser
pushd builddir/emscripten-browser

CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
emconfigure ../../configure -C \
--host=wasm32-unknown-emscripten \
--build=$(../../config.guess) \
--with-emscripten-target=browser \
--with-build-python=$(pwd)/../build/python

emmake make -j$(nproc)
popd
python Tools/wasm/emscripten build
```

Serve `python.html` with a local webserver and open the file in a browser.
Python comes with a minimal web server script that sets necessary HTTP
headers like COOP, COEP, and mimetypes. Run the script outside the container
and from the root of the CPython checkout.

or you can break it out into four separate steps:
```shell
./Tools/wasm/wasm_webserver.py
python Tools/wasm/emscripten configure-build-python
python Tools/wasm/emscripten make-build-python
python Tools/wasm/emscripten configure-host
python Tools/wasm/emscripten make-host
```

and open http://localhost:8000/builddir/emscripten-browser/python.html . This
directory structure enables the *C/C++ DevTools Support (DWARF)* to load C
and header files with debug builds.


#### Cross compile to wasm32-emscripten for node

Extra arguments to the configure steps are passed along to configure. For
instance, to do a debug build, you can use:
```shell
./Tools/wasm/wasm_build.py emscripten-node-dl
python Tools/wasm/emscripten build --with-py-debug
```

The command is roughly equivalent to:

```shell
mkdir -p builddir/emscripten-node-dl
pushd builddir/emscripten-node-dl

CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
emconfigure ../../configure -C \
--host=wasm32-unknown-emscripten \
--build=$(../../config.guess) \
--with-emscripten-target=node \
--enable-wasm-dynamic-linking \
--with-build-python=$(pwd)/../build/python

emmake make -j$(nproc)
popd
```

```shell
node --experimental-wasm-threads --experimental-wasm-bulk-memory --experimental-wasm-bigint builddir/emscripten-node-dl/python.js
```

(``--experimental-wasm-bigint`` is not needed with recent NodeJS versions)

### Limitations and issues

Emscripten before 3.1.8 has known bugs that can cause memory corruption and
resource leaks. 3.1.8 contains several fixes for bugs in date and time
functions.

#### Network stack

- Python's socket module does not work with Emscripten's emulated POSIX
Expand Down Expand Up @@ -241,8 +144,6 @@ functions.
[gh-90548](https://github.com/python/cpython/issues/90548).
- Python's object allocator ``obmalloc`` is disabled by default.
- ``ensurepip`` is not available.
- Some ``ctypes`` features like ``c_longlong`` and ``c_longdouble`` may need
NodeJS option ``--experimental-wasm-bigint``.

#### In the browser

Expand All @@ -263,15 +164,6 @@ Node builds use ``NODERAWFS``.
- Node RawFS allows direct access to the host file system without need to
perform ``FS.mount()`` call.

### wasm64-emscripten

- wasm64 requires recent NodeJS and ``--experimental-wasm-memory64``.
- ``EM_JS`` functions must return ``BigInt()``.
- ``Py_BuildValue()`` format strings must match size of types. Confusing 32
and 64 bits types leads to memory corruption, see
[gh-95876](https://github.com/python/cpython/issues/95876) and
[gh-95878](https://github.com/python/cpython/issues/95878).

### Hosting Python WASM builds

The simple REPL terminal uses SharedArrayBuffer. For security reasons
Expand Down
Loading
Loading