Takeaways from the PyPSA workshop

[gnawin]

Hi all!

I participated in a PyPSA workshop and has some takeaways to share. The full information is here slides.

TL;DR

• PyPSA-Eur uses an automated workflow by snakemake, i.e., with one line of command, it automatically downloads the data, prepares the data in the format that PyPSA needs, reads the configurations for the run and modifies the data accordingly, create the model, and solve the model.
• There is only one configuration file (think of methods, time horizon, countries, some of the techno-economic data etc.) for each run where you only change the things you want, the rest keeps default.
• This can be a starting point for Tulipa: what do we want? do we want to (partially) adopt this?

Overview

• PyPSA is a framework with no data, but PyPSA-Eur includes data in the workflow.
• Linopy, developed by the PyPSA team, is a counterpart of Pyomo which caused performance issues. Julia JuMP was considered, but it was decided that it is easier for the users to not do workflow in Python and optimization in Julia.
• For multi-horizon, both myopic (rolling horizon, most commonly used due to performance) and perfect foresight (too large to solve) are available. Shall/Can we add roling horizon also for multi-year?

Workflow

• PyPSA-Eur has command-line automated workflow
• Data processing pipeline using snakemake (there is also Apache Airflow)
• Data is automatically downloaded. Occasionally there is problems, so recommend to save and label data locally.
• Workflow
  • Build shapes
  • Build base network (OpenStreetMap transmission lines, amongst others)
  • Simplify and cluster network (to 380 kV, 128 (most commonly used) regions). Node clustering is based on demands. Accordingly, the networks are also clustered, but no complex algorithm used for the networks. You can change the spatial resolution, e.g., NL with higher resolution. But there is an effect of that, check https://doi.org/10.1016/j.apenergy.2021.116726
  • Profiles (historical weather data) and availablity renewable and hydro using atlite (atlite: Convert weather data to energy systems data — atlite 0.4 documentation)
  • Build power plants using powerplantmatching (Welcome to powerplantmatching’s documentation! — powerplantmatching 0.7.0.post10+gb21da6f documentation) collecting data from various source, Global Energy Monitor is highly recommended.
  • techno-economic assumptions, prioritizing Danish Energy Agency (all given in 2020 values)
  • General remark: it is not straightforward to bring in own data, there is only high-level options for running sensitivity studies. If you want to use (part of) your own data, use PyPSA and build your own workflow instead.
  • Time aggregation is interesting for the audience, using tsam.

Performance

• All sectors, 128 nodes, 3h, need around ~250 GB of RAM -> solving time is magic, can be 2 days
• GPU-based solver copt Cardinal Optimizer (COPT)-Cardinal Operations (world-class performance, comparable to Gurobi, but not very stable yet). Algorithm different from simplex and barrier.
• Preparing data and creating the model takes only 1% of total time, the rest is solving. Say a model that takes 2 days to solve, preparing the model takes only minutes. The memory use for model preparation is not higher then for solving.
• In preparing the data, what takes most time is to calculate the renewabel potential and profiles (lots of calculations).
• It gets everything in pandas dataframes, and sometimes netCDF files (like compressed dataframes).
• Then the model is built using xarrays, which is very performant. This is their silver bullet for performance. Using xarray can directly create constraints on multiple indices without loops, similar to broadcasting.
• It is good practice to add a transport loss even though there are more constraints, or using a quadratic cost term in the obj, because it is easier for the solver to distinguish.

Configuration

• There is a default configuation file. For each run, you specify a configuration file with only the information that deviates from the default.
• Results are stored in one netcdf file, a container which you can load and explore results interactively. Or you can export to csv files.

PyPSA

• You can use some generic functions to build your model, e.g., components such as buses, and configurations such as costs. You first build an empty model, then add things manually. You can also do import/export using formats such as csv, netCDF, and HDF5. You only put the data you need, and the rest will be defaults.
• Data is in memory using pandas.DataFrame
• Units, if different from the defaults, need to be manually converted.

Sector-couping

• Boundaries between energy and material models blur: transport, heating, industry, industrial feedstcoks, biomass…in terms of networks, there are co2, liquid hydrocarbons
• Distribution bottlenecks are modeled.
• Flow chart
• Can't do stochastic optimization, but check Designing a sector-coupled European energy system robust to 60 years of historical weather data.
• Some retrofitting options are available, but not from gas storage to H2 storage.
• Validation is done by comparing modeling results with historical data.
• There are global constraints that can be applied to "groups", but they don't call them "groups", instead, different labels such as "fuel types" or "carriers" are used for some group constraints.

Snakemake (based on LLM results)

Tor larger, multi-step workflows, Snakemake removes a lot of manual work by:

• Tracking dependencies → Only re-runs when needed.
• Managing parallel execution → Runs independent tasks simultaneously.
• Handling environments → Ensures reproducibility.
• Logging and benchmarking → Automatically tracks execution details.
• Can be used for scenario management (YAML, JSON, TOML)

Should You Use Snakemake for Your Julia Project?

Yes, if:

• You have multiple Julia scripts with dependencies.

• You want to automate and parallelize execution.

• You work on large-scale computations (HPC, cloud, large datasets).

• You need consistent package environments (Pkg.activate()).

No, if:

• You have just one Julia script with no dependencies.
• You are fine with manually running scripts.

[clizbe]

What's interesting is the amount of data scraping + processing they're doing. If they could open that up and make it customizable and shareable, that'd be awesome!

[gnawin]

To me, the most valuable thing is to have seen what an end product (workflow) looks like, and what are the steps to get there. Ofc, we don't have to adopt their workflow, but just to open up discussions.

Option 1:

For each type of data we need,
Step 1: locate the common data sources, import them, and convert to your data format,
Step 2: locate the user-specific data sources and import them (or import directly), and convert to your data format.

They use pandas dataframes, and we use tables.

Option 2:

Import all data given a designated formate (e.g., csv)

Some hints that we may follow:

• Build base network (OpenStreetMap transmission lines, amongst others)
• Simplify and cluster network (to 380 kV, 128 (most commonly used) regions). Node clustering is based on demands. Accordingly, the networks are also clustered, but no complex algorithm used for the networks. You can change the spatial resolution, e.g., NL with higher resolution. But there is an effect of that, check https://doi.org/10.1016/j.apenergy.2021.116726

Get something from the sources and using our upcoming NetworkReduction.jl package?

• Profiles (historical weather data) and availablity renewable and hydro using atlite (atlite: Convert weather data to energy systems data — atlite 0.4 documentation)

Get this and process it using TulipaProfileFitting.jl?

• Build power plants using powerplantmatching (Welcome to powerplantmatching’s documentation! — powerplantmatching 0.7.0.post10+gb21da6f documentation) collecting data from various source, Global Energy Monitor is highly recommended.

Use this and incorporate our own data?

• techno-economic assumptions, prioritizing Danish Energy Agency (all given in 2020 values)

Get techno-economic data from DEA, and incorporate our own data?