Creating an installable pudl-examples package

[zaneselvans]

Recently we've run into a couple of situations where having an external package and python environment that relies on PUDL data, but not the PUDL development environment might be useful. See:

• Create final demand imputation output notebook pudl-examples#10
• Improve usability of Kaggle notebooks / datasets #4381

Internal vs. External PUDL Usage

• We (Catalyst) are frequently working from within the pudl-dev conda environment, which contains all of the dependencies required to process the data that's part of PUDL.
• Users of the PUDL data typically do not use this python environment, and access the data products we publish using whatever software tools (including python packages) they're most comfortable with.
• We've accumulated a few routines for doing data visualization, or helping ourselves access the data, either within the PUDL repo or in our example notebooks.
• However, we can't share those easily. The ones that are part of PUDL require the whole pudl-dev environment to be installed, which is a lot. The ones in our example notebooks aren't installable at all.
• In the case of the EIA-930 / FERC-714 imputation visualizations, this currently means we're cutting and pasting the routines that exist within PUDL into the notebook, which duplicates code, ensuring something will get out of sync, and introduces complexity into the notebook.
• This separation also means that we don't have the same experience of using PUDL data as our users do. In addition to the visualization functions, we have access to intermediate Dagster assets that aren't published, we also have helper functions that can read the data and fix data types. So we may be missing pain points that users face.

Depending on Kaggle

• Kaggle is great because it provides a zero-setup way for users to start playing with PUDL data, without having to think about python environments or packages at all, and with enough memory and CPU resources that they can explore pretty much any of our data without worrying about crashing the machine.
• However, putting all the example code into the notebooks on Kaggle limits the portability and reusability of any examples we come up with. By tailoring the notebooks to run on Kaggle specifically, we also ensure that they probably don't run elsewhere, at least not without some futzing.
• With the notebooks pointing at the Kaggle dataset, they have a big startup lag, because Kaggle has to copy all 30GB over to the notebook workspace before you can run anything. Pointing the notebooks at S3 directly might be more ergonomic.

Potential benefits of a stand-alone catalystcoop.pudl-examples package:

• Users (and we!) could install the package and use these shared routines in contexts besides Kaggle / pudl-dev
• We could create a simpler pudl-prod (or just pudl) python environment that doesn't have all of the extra data viz packages, which aren't actually required for PUDL to do its data processing work.
• However, we could still install pudl-examples within the pudl-dev environment and have access to whatever utility functions we put there for day-to-day development.
• Avoid cutting-and-pasting code from pudl-dev into notebooks / code duplication.
• Avoid cluttering notebooks with big visualization function definitions.
• Could specify the libraries that pudl-examples depends on, independent of whatever is bundled in the Kaggle Python Docker image, and then add just a single extra dependency to our Kaggle notebooks (pudl-examples) which would cause any of those extra dependencies to be installed, if they aren't already part of the Kaggle environment.
• We could use these visualization routines to generate static outputs on a regular basis within the repo using GitHub Actions, e.g. every night at a time after when we expect the nightly builds to have run, both to test that the routines still work with the most recent data outputs, and to have fresh visualizations to look at. These could be visual diagnostics of the build outputs for our use, or could highlight interesting energy system details in an easily shareable format that's automatically kept up to date.
• If we also refactor our example notebooks to rely on data from S3 instead of Kaggle, they could be run in other contexts without modification -- e.g. in browser with JupyterLite / Pyodide.
• We could re-use the same pudl-examples functions to build Marimo notebooks / PUDL data dashboards.
• The repo / package would provide a more holistic example of how to programmatically work with PUDL data.
• This change would let us stop packaging the main PUDL repo for distribution on PyPI/conda-forge while still providing users with a PUDL-data aware python package they can install and play with.
• Would provide another opportunity to learn about new Python environment & packaging management tools (uv/pixi).

Potential downsides / difficulties

• Need to make sure that the pudl-examples dependencies are compatible with the Kaggle environment. E.g. by running tests / notebooks inside the Kaggle python docker container.
• Pointing notebooks at S3 rather than PUDL Kaggle dataset (and removing the dataset dependency) will mean the notebooks don't show up on the Kaggle PUDL dataset anymore. They'll show up on the Catalyst page, but we would need to change how we link to them / promote them.
• Some initial work to get the packaging, docs, tests, and CI set up
• Additional ongoing maintenance of the repo to ensure dependencies get updated, everything keeps working.
• With helper/dataviz functions in a package instead of the notebooks themselves, they'll be a bit more obscured from users.

Implementation Tasks

• Set up a package within the pudl-exmaples repo
  • Decide whether pixi (conda) is required or if uv will work.
  • If we want to use geopandas and the gdal/osgeo libraries, it's likely we'll need pixi
• Move visualization functions from example notebooks and the PUDL repository into that package
• Refactor example notebooks to rely on data in S3 instead of the Kaggle PUDL dataset
• Switch FERC DBF/XBRL extraction to producing DuckDB (or another format that can be queried remotely) rather than SQLite
• How should versioning work?
  • main branch always points at nightly PUDL build outputs. We expect this to break sometimes and need to be fixed.
  • package releases mirror PUDL release versions, point at stable versioned build outputs e.g. v2025.7.0
  • stable branch always points at stable PUDL build outputs?

Continuous Integration / Testing

• Try to run all the notebooks in CI -- do they have enough resources?
• Set up a docs build & deployment (is there an easy way to embed executable python / dataviz in the docs?)
• Create a test suite for the various functions included in the package
• Create a PyPi / conda-forge package distribution job
• Set up pre-commit hooks for the repo & make sure that pre-commit.ci works