Marimo build, test, deploy workflow (#22)

* wip: make marimo html-wasm example

* feat: build all marimo notebooks and push to GH pages

* docs: update README

* chore: just pin to major versions

* feat: add integration test for plant explorer

* add playwright to wasm env
* add pytest fixture which spins up a test server
* futz around with playwright selectors to interact with different UI elements & look at charts
* update GHA + export script to accommodate testing changes
* update README :)
* fix plant-explorer which didn't have a parquet engine imported -_-

* docs: updates from PR review

Author: jdangerx

.github/workflows/deploy-marimo-pages.yml

@@ -2,41 +2,37 @@
 name: Deploy Marimo Pages
 
 on:
+  workflow_dispatch:
   push:
     branches:
       - main
     paths:
-      - "marimo/**"
+      - "wasm/marimo/**"
+      - "scripts/export_marimo_notebooks.py"
+      - "pixi.toml"
+      - "pixi.lock"
       - ".github/workflows/deploy-marimo-pages.yml"
-  workflow_dispatch:
-
 jobs:
   build:
     runs-on: ubuntu-latest
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v6
 
-      - name: Set up pixi
-        uses: prefix-dev/setup-pixi@v0.9.0
+      - name: Install pixi
+        uses: prefix-dev/setup-pixi@v0.9.3
         with:
-          cache: true
+          environments: wasm
 
       - name: Ensure docs directory exists
         run: mkdir -p docs/
 
       - name: Export Marimo notebooks
-        run: |
-          pixi run marimo export html-wasm marimo/sec10k-data-review.py -o docs/sec10k-data-review.html --mode edit
-
-      - name: Verify generated files
-        run: |
-          echo "Generated files:"
-          find docs -type f -name "*.html" -exec ls -la {} \;
+        run: pixi run -e wasm export-wasm-marimo
 
       - name: Upload Pages Artifact
-        uses: actions/upload-pages-artifact@v3
+        uses: actions/upload-pages-artifact@v4
         with:
           path: docs/
 

.github/workflows/integration-tests.yml

@@ -0,0 +1,31 @@
+---
+name: Integration Tests
+
+on:
+  pull_request:
+    paths:
+      - "wasm/marimo/**"
+      - "scripts/export_marimo_notebooks.py"
+      - "pixi.toml"
+      - "pixi.lock"
+      - ".github/workflows/integration-tests.yml"
+      - "tests/playwright/**"
+
+jobs:
+  wasm-marimo-playwright:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install pixi
+        uses: prefix-dev/setup-pixi@v0.9.3
+        with:
+          environments: wasm
+
+      - name: Install Playwright browser
+        run: pixi run -e wasm playwright install --with-deps chromium
+
+      - name: Run Playwright tests against exported notebooks
+        run: pixi run -e wasm test-wasm-marimo

.gitignore

@@ -3,3 +3,5 @@
 **/*.swp
 dask-worker-space
 **/__pycache__/
+docs/
+tests/playwright/_build/

README.md

@@ -5,16 +5,44 @@ This repository contains a collection of
 and software distributed by [Catalyst Cooperative](https://catalyst.coop)'s
 [Public Utility Data Liberation (PUDL) project](https://github.com/catalyst-cooperative/pudl).
 
-The notebooks in the top-level directory of the repository are mirrored from the
-[Kaggle
-notebooks](https://www.kaggle.com/datasets/catalystcooperative/pudl-project/data).
-**Any changes made to these notebooks directly in this repository will be overwritten.**
+We have multiple categories of notebooks in this repository:
 
+| Category | File location | Description | Live view | Local dev env |
+| --- | --- | --- | --- | --- |
+| [Kaggle notebooks](#kaggle-notebooks) | repository root | Mirrored from the [Kaggle notebooks](https://www.kaggle.com/code/catalystcooperative/). Any changes made to these notebooks directly in this repository will be overwritten. | [Kaggle](https://www.kaggle.com/code/catalystcooperative/) | `pixi run -e kaggle jupyter lab` |
+| [WebAssembly Marimo notebooks](#webassembly-marimo-notebooks) | `wasm/marimo` | Marimo notebooks designed for exporting as interactive browser dashboards. | [PUDL Data Viewer](https://data.catalyst.coop/secret-gallery) | `pixi run -e wasm marimo edit` |
 
-## Run Jupyter Notebooks on Kaggle
+## General development notes
 
-The easiest way to get up and running with the Jupyter examples and a fresh copy of all the
-PUDL data is on [Kaggle](https://www.kaggle.com).
+We use `pixi` to manage the Python environment. You can install it according to
+[their instructions](https://pixi.sh/latest/installation).
+
+We use multiple `pixi` environments to manage the various quirks of the
+different execution environments of our notebooks: we currently have `kaggle`
+and `wasm` defined. We also have `dev` defined for repository-infrastructure
+related tools.
+
+Speaking of which - you probably want to install the pre-commit hooks if you're
+planning on making changes to the notebooks, the repository infrastructure, or
+anything else that might require you to commit something.
+
+```
+$ pixi run -e dev pre-commit install
+```
+
+## Kaggle notebooks
+
+These are notebooks which explore how to access PUDL and showcase some potential
+analyses you can do. They are intended to serve as jumping-off points for your
+own code.
+
+Critically, these are mirrored from Kaggle, and updates in this repository will
+be overwritten from Kaggle.
+
+### Running on Kaggle
+
+The easiest way to get up and running with a fresh copy of all the PUDL data is
+on [Kaggle](https://www.kaggle.com).
 
 Kaggle offers substantial free computing resources and convenient data storage, so you
 can start playing with the PUDL data without needing to set up any software or download
@@ -26,24 +54,15 @@ any data.
 - [03 EIA-930 Sanity Checks](https://www.kaggle.com/code/catalystcooperative/03-eia-930-sanity-checks)
 - [04 Renewable Generation Profiles](https://www.kaggle.com/code/catalystcooperative/04-renewable-generation-profiles)
 - [05 FERC-714 Electricity Demand Forecast Biases](https://www.kaggle.com/code/catalystcooperative/05-ferc-714-electricity-demand-forecast-biases)
+- [06 PUDL Imputed Electricity Demand](https://www.kaggle.com/code/catalystcooperative/06-pudl-imputed-electricity-demand)
+- [07 FERC EQR Access](www.kaggle.com/code/catalystcooperative/07-ferc-eqr-access)
 
 You'll find the [PUDL data dictionary](https://catalystcoop-pudl.readthedocs.io/en/latest/data_dictionaries/pudl_db.html)
 helpful for interpreting the data.
 
-## Run Kaggle Notebooks Locally
+### Running locally
 
-You can also work with these notebooks and the PUDL data locally.
-
-We use `pixi` to manage the Python environment. You can install it according to
-[their instructions](https://pixi.sh/latest/installation).
-
-Then install the project environment including development dependencies:
-
-```
-$ pixi install -e kaggle
-```
-
-To run the Jupyter notebooks, run:
+To run these notebooks locally, run:
 
 ```
 $ pixi run -e kaggle jupyter lab
@@ -58,15 +77,13 @@ You can find detailed instructions for how to download the data in our [data
 access
 documentation](https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#quick-reference).
 
-## Developer notes
+#### Updating the dependencies
 
-To install the pre-commit hooks, run:
+The Kaggle Python environment is an eternally shifting patchwork of
+dependencies. We try to keep the Pixi environment up to date by occasionally
+running a notebook on Kaggle that just checks dependency versions.
 
-```
-$ pixi run -e dev pre-commit install
-```
-
-To update the Kaggle dependency pins:
+To update the Kaggle dependency pins yourself:
 
 1. Create a Kaggle API token and put it in your env as `KAGGLE_API_TOKEN`.
 2. Run the sync script:
@@ -78,6 +95,65 @@ $ pixi run -e dev sync-kaggle-deps
    dependencies to `--exclude` in the `sync-kaggle-deps` pixi task.
 
 
+## WebAssembly Marimo notebooks
+
+These are designed to be exported to HTML and viewed in a web browser as an
+interactive dashboard or visualization.
+
+### Viewing in-browser
+
+These are currently viewable on the [PUDL Data
+Viewer](https://data.catalyst.coop/secret-gallery). Click through and play
+around!
+
+### Running locally
+
+Run `marimo edit` to start editing:
+
+```
+$ pixi run -e wasm marimo edit wasm/marimo
+```
+
+If you want to export the notebooks to WebAssembly and see if they work as
+standalone HTML pages:
+
+```
+$ pixi run -e wasm export-wasm-marimo --serve
+```
+
+Then go to `localhost:8000/<your-notebook-name>.html` to see it work!
+
+This uses [`pyodide`](https://pyodide.org/) to run Python in your browser, which
+comes with a few quirks:
+
+* Marimo is currently (as of 2026-02-27) pinned to version `0.27.7` - which
+  means the [libraries that are
+  available](https://pyodide.org/en/0.27.7/usage/packages-in-pyodide.html) in
+  `pyodide` are pretty old. To minimize shock while going from `marimo edit` to
+  the exported version, we pin dependencies based on those versions.
+* You have to explicitly import `fastparquet` or `pyarrow` for
+  `pandas.read_parquet` to actually work. `fastparquet` is 3-10x faster.
+
+See the [Marimo docs](https://docs.marimo.io/guides/wasm/) for more details.
+
+
+Finally, if you want to add automated tests with `playwright`:
+
+First, install a browser driver for `playwright`:
+
+```
+$ pixi run -e wasm playwright install --with-deps chromium
+```
+
+Then, run the tests:
+
+```
+$ pixi run -e wasm test-wasm-marimo
+```
+
+You can add more tests in `tests/playwright/...`
+
+
 ## Stalk us on the Internet
 
 - <https://catalyst.coop>