Skip to content

Commit 3af15f7

Browse files
bulenty584pre-commit-ci[bot]ryanking13
authored
Add install-emscripten CLI command (#243)
* Add install-emscripten CLI command - Add new 'pyodide xbuildenv install-emscripten' command - Add clone_emscripten() method to clone emsdk repository - Add install_emscripten() method to install and activate Emscripten SDK - Add tests for new functionality * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Emsdk installation comments fixed in xbuildenv.py and unneeded tests in test_install_emscripten.py removed as well * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * TestInstallEmscripten class removed from test_install_emscripten.py and some basic reformatting * Following @ryanking's comments, 1. Changed default version to `get_build_flag("PYODIDE_EMSCRIPTEN_VERSION")` 2. Updated `clone_emscripten` to `_clone_emscripten` 3. Removed and edited redundant tests in both new test files 4. Updated patch path in `xbuildenv.py` and added new command to CI workflow * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed integration test command to use xbuildenv and activate emsdk Removed the get_build_flag command from xbuildenv cli and added default behavior as none * 1. emsdk activation directory changed in main.yml 2. emsdk caching job removed in main.yml 3. install-emscripten help message also edited * 1. Fixed unittests to follow new code changes 2. Fixed pyodide-root to pyodide_root in integration tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated Python version to 3.13 for xbuildenv and changed back lines 94-96 in for clearer test conditions * Update .github/workflows/main.yml * Remove emsdk dependency check Removed emsdk check from the Makefile. * Restore emsdk activation for integration tests Reintroduced emsdk activation in test runs and ensured environment setup for pytest. * Update Python version to 3.13 in workflow * Change Python version from 3.13 to 3.12 for unittest Updated Python version in GitHub Actions workflow. * Update NumPy version to 2.2.5 in integration test * Change workflow OS to only use Ubuntu Updated OS configuration for workflow to use only Ubuntu. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Gyeongjae Choi <[email protected]>
1 parent 16d9bfc commit 3af15f7

File tree

8 files changed

+686
-22
lines changed

8 files changed

+686
-22
lines changed

.github/workflows/main.yml

Lines changed: 9 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
- name: Setup Python
3737
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
3838
with:
39-
python-version: "3.12"
39+
python-version: "3.12" # TODO: update to 3.13
4040

4141
- name: Set up Node.js
4242
uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
@@ -107,7 +107,8 @@ jobs:
107107
{ name: test-src, installer: uv },
108108
{ name: test-integration-marker, installer: pip }, # installer doesn't matter
109109
]
110-
os: [ubuntu-latest, macos-latest]
110+
# os: [ubuntu-latest, macos-latest] # FIXME: https://github.com/oracle/graal/issues/11855
111+
os: [ubuntu-latest]
111112
pyodide-version: [stable]
112113
include:
113114
# Run no-isolation tests and Pyodide minimum version testing only
@@ -131,7 +132,7 @@ jobs:
131132
- name: Setup Python
132133
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
133134
with:
134-
python-version: "3.12"
135+
python-version: "3.13"
135136

136137
- name: Set up Node.js
137138
uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
@@ -154,33 +155,26 @@ jobs:
154155
fi
155156
echo EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version) >> $GITHUB_ENV
156157
157-
- name: Cache emsdk
158-
uses: actions/cache@v4
159-
with:
160-
path: ${{ env.EMSDK_CACHE_FOLDER }}
161-
key: ${{ env.EMSDK_CACHE_NUMBER }}-${{ env.EMSCRIPTEN_VERSION }}-${{ runner.os }}
162-
163158
- name: Install Emscripten
164-
uses: mymindstorm/setup-emsdk@6ab9eb1bda2574c4ddb79809fc9247783eaf9021 # v14
165-
with:
166-
version: ${{ env.EMSCRIPTEN_VERSION }}
167-
actions-cache-folder: ${{env.EMSDK_CACHE_FOLDER}}
159+
run: pyodide xbuildenv install-emscripten
168160

169161
- name: Get number of cores on the runner
170162
id: get-cores
171163
run: echo "CORES=$(nproc)" >> $GITHUB_OUTPUT
172164

173165
- name: Run tests marked with integration
174166
if: matrix.task.name == 'test-integration-marker'
175-
run: pytest --junitxml=test-results/junit.xml --cov=pyodide-build pyodide_build -m "integration"
167+
run: |
168+
source $(pyodide config get pyodide_root)/../../emsdk/emsdk_env.sh
169+
pytest --junitxml=test-results/junit.xml --cov=pyodide-build pyodide_build -m "integration"
176170
177171
- name: Run the recipe integration tests (${{ matrix.task.name }})
178172
if: matrix.task.name != 'test-integration-marker'
179173
env:
180174
PYODIDE_JOBS: ${{ steps.get-cores.outputs.CORES }}
181175
working-directory: integration_tests
182176
run: |
183-
177+
source $(pyodide config get pyodide_root)/../../emsdk/emsdk_env.sh
184178
# https://github.com/pyodide/pyodide-build/issues/147
185179
# disable package with scikit-build-core
186180
if [[ ${{ matrix.os }} == "macos-latest" ]]; then

integration_tests/Makefile

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,6 @@ check:
4747
@echo "... Checking dependencies"
4848

4949
@which pyodide > /dev/null || (echo "pyodide-build is not installed"; exit 1)
50-
@which emsdk > /dev/null || (echo "emscripten is not installed"; exit 1)
5150

5251
@echo "... Passed"
5352

integration_tests/src/numpy.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
set -e
44

5-
VERSION="2.0.2"
5+
VERSION="2.2.5"
66
URL="https://files.pythonhosted.org/packages/source/n/numpy/numpy-${VERSION}.tar.gz"
77

88
wget $URL

pyodide_build/cli/xbuildenv.py

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
import typer
44

5-
from pyodide_build.build_env import local_versions
5+
from pyodide_build.build_env import get_build_flag, local_versions
66
from pyodide_build.common import default_xbuildenv_path
77
from pyodide_build.views import MetadataView
88
from pyodide_build.xbuildenv import CrossBuildEnvManager
@@ -205,3 +205,31 @@ def _search(
205205
print(MetadataView.to_json(views))
206206
else:
207207
print(MetadataView.to_table(views))
208+
209+
210+
@app.command("install-emscripten")
211+
def _install_emscripten(
212+
version: str = typer.Option(
213+
None,
214+
help="Emscripten version corresponding to the target Pyodide version",
215+
),
216+
path: Path = typer.Option(DEFAULT_PATH, help="Pyodide cross-env path"),
217+
) -> None:
218+
"""
219+
Install Emscripten SDK into the cross-build environment.
220+
221+
This command clones the emsdk repository, installs and activates the specified
222+
Emscripten version, and applies Pyodide-specific patches.
223+
"""
224+
check_xbuildenv_root(path)
225+
manager = CrossBuildEnvManager(path)
226+
227+
if version is None:
228+
version = get_build_flag("PYODIDE_EMSCRIPTEN_VERSION")
229+
230+
print("Installing emsdk...")
231+
232+
emsdk_dir = manager.install_emscripten(version)
233+
234+
print("Installing emsdk complete.")
235+
print(f"Use `source {emsdk_dir}/emsdk_env.sh` to set up the environment.")
Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
"""Tests for install-emscripten CLI command"""
2+
3+
import subprocess
4+
from pathlib import Path
5+
6+
from typer.testing import CliRunner
7+
8+
from pyodide_build.cli import xbuildenv
9+
10+
runner = CliRunner()
11+
12+
13+
def test_install_emscripten_no_xbuildenv(tmp_path):
14+
"""Test that install-emscripten fails when no xbuildenv exists"""
15+
envpath = Path(tmp_path) / ".xbuildenv"
16+
17+
result = runner.invoke(
18+
xbuildenv.app,
19+
[
20+
"install-emscripten",
21+
"--path",
22+
str(envpath),
23+
],
24+
)
25+
26+
assert result.exit_code != 0, result.stdout
27+
assert "Cross-build environment not found" in result.stdout, result.stdout
28+
29+
30+
def test_install_emscripten_default_version(tmp_path, monkeypatch):
31+
"""Test installing Emscripten with default version"""
32+
envpath = Path(tmp_path) / ".xbuildenv"
33+
envpath.mkdir()
34+
35+
monkeypatch.setattr(
36+
"pyodide_build.cli.xbuildenv.get_build_flag",
37+
lambda name: "3.1.46",
38+
)
39+
40+
called = {}
41+
42+
def fake_install(self, version):
43+
called["version"] = version
44+
return self.env_dir / "emsdk"
45+
46+
monkeypatch.setattr(
47+
"pyodide_build.xbuildenv.CrossBuildEnvManager.install_emscripten",
48+
fake_install,
49+
)
50+
51+
result = runner.invoke(
52+
xbuildenv.app,
53+
[
54+
"install-emscripten",
55+
"--path",
56+
str(envpath),
57+
],
58+
)
59+
60+
assert result.exit_code == 0, result.stdout
61+
assert "Installing emsdk..." in result.stdout, result.stdout
62+
assert "Installing emsdk complete." in result.stdout, result.stdout
63+
assert "Use `source" in result.stdout, result.stdout
64+
assert "emsdk_env.sh` to set up the environment." in result.stdout, result.stdout
65+
assert called["version"] == "3.1.46"
66+
67+
68+
def test_install_emscripten_specific_version(tmp_path, monkeypatch):
69+
"""Test installing Emscripten with a specific version"""
70+
envpath = Path(tmp_path) / ".xbuildenv"
71+
envpath.mkdir()
72+
73+
called = {}
74+
75+
def fake_install(self, version):
76+
called["version"] = version
77+
return self.env_dir / "emsdk"
78+
79+
monkeypatch.setattr(
80+
"pyodide_build.xbuildenv.CrossBuildEnvManager.install_emscripten",
81+
fake_install,
82+
)
83+
84+
emscripten_version = "3.1.46"
85+
result = runner.invoke(
86+
xbuildenv.app,
87+
[
88+
"install-emscripten",
89+
"--version",
90+
emscripten_version,
91+
"--path",
92+
str(envpath),
93+
],
94+
)
95+
96+
assert result.exit_code == 0, result.stdout
97+
assert "Installing emsdk..." in result.stdout, result.stdout
98+
assert "Installing emsdk complete." in result.stdout, result.stdout
99+
assert called["version"] == emscripten_version
100+
101+
102+
def test_install_emscripten_with_existing_emsdk(tmp_path, monkeypatch):
103+
"""Test installing Emscripten when emsdk already exists (should pull updates)"""
104+
envpath = Path(tmp_path) / ".xbuildenv"
105+
envpath.mkdir()
106+
107+
existing_emsdk = envpath / "emsdk"
108+
existing_emsdk.mkdir()
109+
110+
monkeypatch.setattr(
111+
"pyodide_build.cli.xbuildenv.get_build_flag",
112+
lambda name: "latest",
113+
)
114+
115+
def fake_install(self, version):
116+
assert version == "latest"
117+
assert existing_emsdk.exists()
118+
return existing_emsdk
119+
120+
monkeypatch.setattr(
121+
"pyodide_build.xbuildenv.CrossBuildEnvManager.install_emscripten",
122+
fake_install,
123+
)
124+
125+
result = runner.invoke(
126+
xbuildenv.app,
127+
[
128+
"install-emscripten",
129+
"--path",
130+
str(envpath),
131+
],
132+
)
133+
134+
assert result.exit_code == 0, result.stdout
135+
assert "Installing emsdk..." in result.stdout, result.stdout
136+
assert "Installing emsdk complete." in result.stdout, result.stdout
137+
assert str(existing_emsdk / "emsdk_env.sh") in result.stdout
138+
139+
140+
def test_install_emscripten_git_failure(tmp_path, monkeypatch):
141+
"""Test handling of git clone failure"""
142+
envpath = Path(tmp_path) / ".xbuildenv"
143+
envpath.mkdir()
144+
145+
monkeypatch.setattr(
146+
"pyodide_build.xbuildenv.CrossBuildEnvManager.install_emscripten",
147+
lambda self, version: (_ for _ in ()).throw(
148+
subprocess.CalledProcessError(1, "git clone")
149+
),
150+
)
151+
152+
result = runner.invoke(
153+
xbuildenv.app,
154+
[
155+
"install-emscripten",
156+
"--path",
157+
str(envpath),
158+
],
159+
)
160+
161+
# Should fail due to git clone error
162+
assert result.exit_code != 0
163+
assert isinstance(result.exception, subprocess.CalledProcessError)
164+
165+
166+
def test_install_emscripten_emsdk_install_failure(tmp_path, monkeypatch):
167+
"""Test handling of emsdk install command failure"""
168+
envpath = Path(tmp_path) / ".xbuildenv"
169+
envpath.mkdir()
170+
171+
monkeypatch.setattr(
172+
"pyodide_build.xbuildenv.CrossBuildEnvManager.install_emscripten",
173+
lambda self, version: (_ for _ in ()).throw(
174+
subprocess.CalledProcessError(1, "./emsdk install")
175+
),
176+
)
177+
178+
result = runner.invoke(
179+
xbuildenv.app,
180+
[
181+
"install-emscripten",
182+
"--path",
183+
str(envpath),
184+
],
185+
)
186+
187+
# Should fail due to emsdk install error
188+
assert result.exit_code != 0
189+
assert isinstance(result.exception, subprocess.CalledProcessError)
190+
191+
192+
def test_install_emscripten_output_format(tmp_path, monkeypatch):
193+
"""Test that the output message format is correct"""
194+
envpath = Path(tmp_path) / ".xbuildenv"
195+
envpath.mkdir()
196+
197+
monkeypatch.setattr(
198+
"pyodide_build.cli.xbuildenv.get_build_flag",
199+
lambda name: "latest",
200+
)
201+
202+
expected_path = envpath / "emsdk"
203+
204+
monkeypatch.setattr(
205+
"pyodide_build.xbuildenv.CrossBuildEnvManager.install_emscripten",
206+
lambda self, version: expected_path,
207+
)
208+
209+
result = runner.invoke(
210+
xbuildenv.app,
211+
[
212+
"install-emscripten",
213+
"--path",
214+
str(envpath),
215+
],
216+
)
217+
218+
assert result.exit_code == 0, result.stdout
219+
220+
# Verify output format - check for key messages (logger adds extra lines)
221+
assert "Installing emsdk..." in result.stdout
222+
assert "Installing emsdk complete." in result.stdout
223+
assert "Use `source" in result.stdout
224+
assert "emsdk_env.sh` to set up the environment." in result.stdout

pyodide_build/tests/test_cli_xbuildenv.py

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ def test_xbuildenv_install(tmp_path, mock_xbuildenv_url):
9393
assert result.exit_code == 0, result.stdout
9494
assert "Downloading Pyodide cross-build environment" in result.stdout, result.stdout
9595
assert "Installing Pyodide cross-build environment" in result.stdout, result.stdout
96+
assert str(envpath.resolve()) in result.stdout, result.stdout
9697
assert (envpath / "xbuildenv").is_symlink()
9798
assert (envpath / "xbuildenv").resolve().exists()
9899

@@ -121,8 +122,10 @@ def test_xbuildenv_install_version(tmp_path, fake_xbuildenv_releases_compatible)
121122
os.environ.pop(CROSS_BUILD_ENV_METADATA_URL_ENV_VAR, None)
122123

123124
assert result.exit_code == 0, result.stdout
124-
assert "Downloading Pyodide cross-build environment" in result.stdout, result.stdout
125-
assert "Installing Pyodide cross-build environment" in result.stdout, result.stdout
125+
assert "Pyodide cross-build environment installed at" in result.stdout, (
126+
result.stdout
127+
)
128+
assert str(envpath.resolve()) in result.stdout, result.stdout
126129
assert (envpath / "xbuildenv").is_symlink()
127130
assert (envpath / "xbuildenv").resolve().exists()
128131
assert (envpath / "0.1.0").exists()
@@ -166,8 +169,10 @@ def test_xbuildenv_install_force_install(
166169
)
167170

168171
assert result.exit_code == 0, result.stdout
169-
assert "Downloading Pyodide cross-build environment" in result.stdout, result.stdout
170-
assert "Installing Pyodide cross-build environment" in result.stdout, result.stdout
172+
assert "Pyodide cross-build environment installed at" in result.stdout, (
173+
result.stdout
174+
)
175+
assert str(envpath.resolve()) in result.stdout, result.stdout
171176
assert (envpath / "xbuildenv").is_symlink()
172177
assert (envpath / "xbuildenv").resolve().exists()
173178
assert (envpath / "0.1.0").exists()

0 commit comments

Comments
 (0)