Skip to content

Commit 5c3eb38

Browse files
ravenac95ravenac95
authored
chore: wasm dev docs (#4941)
* chore: handle renamed marimo build script * chore: add wasm dev environment docs * inline doc * clean up * updates * chore: adds docs on how to add new packages * more updates
1 parent 9855e5d commit 5c3eb38

File tree

2 files changed

+236
-1
lines changed

2 files changed

+236
-1
lines changed

‎.github/workflows/deploy-marimo-wasm.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ env:
99
on:
1010
# Allows you to run this workflow manually from the Actions tab
1111
workflow_dispatch:
12+
schedule:
13+
# Hack for now since we can't trigger off events in the marimo fork. Deploys every hour
14+
- cron: '0 * * * *'
1215

1316
jobs:
1417
build-deploy:
@@ -45,7 +48,7 @@ jobs:
4548
run: pixi run make fe
4649

4750
- name: Build the dist directory for marimo
48-
run: pixi run bash scripts/build_production_marimo_wasm.sh marimo_dist marimo.opensource.observer
51+
run: pixi run bash scripts/build_marimo_static_dist.sh marimo_dist marimo.opensource.observer
4952

5053
- name: Install wrangler
5154
run: pnpm add -g wrangler

‎apps/docs/docs/guides/wasm.md‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
---
2+
title: Developing in the WASM Environment
3+
sidebar_position: 1
4+
---
5+
6+
In order to make wasm notebooks a possibility we utilize marimo's ability to run
7+
inside of the browser. For OSO, we've added some special configurations that are
8+
unique to the OSO platform. In order to support this, we created a tool that
9+
allows us to easily test changes in the python environment that is loaded into
10+
marimo. Additionally, it allows us to inject our own WASM controller so that we
11+
can provide a tailored experience.
12+
13+
The primary tool for all of this is called `wasm-builder` and must currently be
14+
loaded from OSO's [marimo fork](https://github.com/opensource-observer/marimo).
15+
This is a proxy for testing the wasm version of marimo and would only be useful
16+
in that regard. The proxy allows us the following features while testing:
17+
18+
- Create and serve an updated wasm compatible marimo python wheel based on
19+
changes in the local environment
20+
- By default, marimo's wasm environment links to a deployed version of
21+
marimo. Using this proxy allows completely localized development.
22+
- Create an up to date pyodide lock file for the wasm frontend
23+
- The pyodide lock file, much like a `uv.lock` or `package-lock.json` file,
24+
ensures that the same versions of packages are used each time. This is
25+
important for stability and reproducibility. This file normally requires a
26+
bit of manual work to generate. The `wasm-builder` tool automatically
27+
regenerates this as needed as the local marimo python code is changed.
28+
- Support developing _any_ python code for use in wasm
29+
- In addition to marimo's based code required to run the wasm notebook, we
30+
can also generate updated wheel files for any uv based python project
31+
(this is extensible to other package management solutions in the future).
32+
33+
## Background
34+
35+
Marimo's wasm environment is a bit complex and requires a bit of knowledge of
36+
their codebase and some general understanding about python packaging.
37+
Additionally, it's important to understand how we use marimo and how we inject
38+
custom behavior.
39+
40+
### Pyodide
41+
42+
Pyodide is the wasm version of CPython, allowing you to run Python code in the
43+
browser. It is a key component of the marimo wasm environment. Marimo sets this
44+
up in what it calls a `WasmController`. This `WasmController` can be injected at
45+
runtime as long as there is a `/wasm/controller.js` endpoint available on the
46+
server hosting the wasm notebook. We currently provide our own in our marimo
47+
fork at `frontend/src/oso-extensions/wasm/controller.tsx`.
48+
49+
### Python Wheel
50+
51+
You will see "python wheels" referenced in pyodide and wasm quite a bit. A
52+
python wheel is a distribution format for Python packages. It is a binary
53+
package format that is actually a zip of all the files needed for a given
54+
execution platform. A pure python package will have `none` as the platform.
55+
Python wheel files look like:
56+
`<package_name>-<version>-<python_version>-<abi>-<platform>.whl`.
57+
58+
## Prerequisites
59+
60+
Please ensure the following are installed:
61+
62+
- [pixi](https://pixi.sh)
63+
- Needed for the marimo code
64+
- [docker](https://www.docker.com/)
65+
66+
## Starting the development environment
67+
68+
Since `wasm-builder` is a proxy you will also need a proxied service to be
69+
running. That proxied service, in this case, should be the marimo frontend. To
70+
this, first clone OSO's marimo fork:
71+
72+
```bash
73+
git clone https://github.com/opensource-observer/marimo.git
74+
cd marimo
75+
```
76+
77+
Additionally, if you're on a macos with Apple Silicon, you may need to run the
78+
following command:
79+
80+
```bash
81+
pixi workspace platform add linux-aarch64
82+
```
83+
84+
Next, in one terminal start the frontend:
85+
86+
```bash
87+
pixi run hatch shell # Start a shell in the pixi environment
88+
make fe # build the frontend
89+
cd frontend
90+
# Start the frontend. You will want to set PYODIDE=true so that you can force the use of the
91+
# pyodide backend. Vite sometimes needs a bit more memory with the marimo frontend build.
92+
# Hence the `NODE_OPTIONS` setting.
93+
PYODIDE=true NODE_OPTIONS=--max-old-space-size=6144 pnpm vite --config oso.viteconfig.mts
94+
```
95+
96+
:::Note
97+
If you would like to remove `react-scan` from the frontend dev server you should
98+
ensure that `NODE_ENV != "development"`. You can set this to something like
99+
`NODE_ENV=test`
100+
:::
101+
102+
The frontend listens on port 3000 by default. In another terminal, you can start
103+
the wasm-builder proxy (this is from the root of the fork):
104+
105+
```bash
106+
pixi run hatch shell # Ensure you're in the pixi environment
107+
cd packages/wasm-builder
108+
pnpm start
109+
```
110+
111+
The server will start listening on port 6008 by default.
112+
113+
:::Note
114+
If you happen to be developing, using a remote development setup you will want
115+
to make sure you set the `PUBLIC_PACKAGES_HOST` to the correct host for your
116+
remote setup.
117+
:::
118+
119+
To access the notebook now, you can navigate to `http://localhost:6008/notebook` in your
120+
browser. The `/notebook` endpoint is specific to OSO's wasm environment.
121+
122+
:::
123+
Note: If you access the frontend without the proxy, you will not be
124+
able to have a working wasm environment.
125+
:::
126+
127+
## Setting up pyoso and oso_semantic in the development environment
128+
129+
As part of OSO's marimo wasm environment, we load both pyoso and oso_semantic
130+
into the pyodide environment. It is likely that you will want to develop against
131+
these in the wasm environment. To do so, you need to add the following to the
132+
`.env` in the `wasm-builder`'s main directory:
133+
134+
```bash
135+
OTHER_UV_PACKAGES_TO_INCLUDE='[{"name": "pyoso", "projectDir": "/path/to/oso/warehouse/pyoso", "outputDir": "/path/to/oso/dist"},{"name": "oso_semantic", "projectDir": "/path/to/oso/warehouse/oso_semantic", "outputDir": "/path/to/oso/dist"} ]'
136+
```
137+
138+
In the this configuration you simply need to replace all the occurrences of the
139+
string `/path/to/oso` with the path on your system to the oso repository. This
140+
is supposed to be the path to where you store the `oso` monorepo and then the
141+
`/dist` subdirectory within (which is where the built artifacts will be placed
142+
by uv).
143+
144+
## Adding additional packages to the wasm environment
145+
146+
The previous section about pyoso and oso_semantic is a specific example of how
147+
to add additional packages to the wasm environment. To add a brand new package,
148+
you will need to update the `OTHER_UV_PACKAGES_TO_INCLUDE` variable in the
149+
`.env` file with the new package's information.
150+
151+
The `OTHER_UV_PACKAGES_TO_INCLUDE` variable is a JSON array of objects, where each object
152+
contains the following fields:
153+
154+
- `name`: The name of the package.
155+
- `projectDir`: The path to the package's source code.
156+
- `outputDir`: The path to the directory where the built artifacts will be placed.
157+
158+
By default, all packages in the `OTHER_UV_PACKAGES_TO_INCLUDE` json object are
159+
loaded in the pyodide environment. If, however, you'd like to add these into the
160+
production build you'll need to change our fork of the wasm controller to inject
161+
the correct packages.
162+
163+
Within OSO's marimo fork, you'd update the file
164+
`frontend/src/oso-extensions/wasm/controller.tsx`. The section that looks like this:
165+
166+
```tsx
167+
private async loadNotebookDeps(code: string, foundPackages: Set<string>) {
168+
const pyodide = this.requirePyodide;
169+
170+
foundPackages.add("pyoso>=0.6.4")
171+
```
172+
173+
Adds, `pyoso>=0.6.4` into the environment. To add any additional packages just
174+
add them to the `foundPackages` set, like so:
175+
176+
```tsx
177+
private async loadNotebookDeps(code: string, foundPackages: Set<string>) {
178+
const pyodide = this.requirePyodide;
179+
180+
foundPackages.add("pyoso>=0.6.4")
181+
foundPackages.add("mypackages")
182+
```
183+
184+
You can specify any version that you require.
185+
186+
:::Note
187+
As each package will add to the load time of the wasm notebook, we would suggest
188+
to try to limit the number of preloaded packages as much as possible.
189+
:::
190+
191+
## Manually testing the build
192+
193+
Inevitably, you will want to manually test the build for wasm. To do so you can
194+
use the build scripts that we have in the marimo repository. The syntax for this
195+
is as follows:
196+
197+
```bash
198+
bash scripts/build_marimo_static_dist.sh <build-dir> <host> [<port>]
199+
```
200+
201+
The `build-dir` and `host` are both required, while the `port` is optional and
202+
defaults to `443`. As we are testing, we likely need to set both the `host` and
203+
`port` arguments. For the host variable, you will want to ensure that it points
204+
to the correct host your browser will be accessing when you test this. Assuming
205+
you're developing this locally, it's likely `127.0.0.1`. For the port, choose
206+
any port you like for this example, we will use `6008`.
207+
208+
In addition, we will want to ensure that the `NODE_ENV` is explicitly set to
209+
something that is not `production`. Production builds will ignore additional uv
210+
packages (as it's expected that we are building from things on pypi).
211+
212+
So finally, to run the build into a directory `.static_wasm`:
213+
214+
```bash
215+
cd /path/to/marimo
216+
pixi run hatch shell
217+
NODE_ENV=test bash scripts/build_marimo_static_dist.sh .static_wasm 127.0.0.1 6008
218+
```
219+
220+
Then to serve this, you can use a simple python server:
221+
222+
```bash
223+
cd .static_wasm
224+
python3 -m http.server 6008
225+
```
226+
227+
Now you can go to your browser at http://localhost:6008/notebook.html.
228+
229+
:::Note
230+
Python simple server does support reference html files without `.html`. This
231+
would be different behavior than something deployed on our production setup.
232+
:::

0 commit comments

Comments
 (0)