environment.yaml update

.gitignore

@@ -1,13 +1,40 @@
-# Snakemake internals
-.snakemake*
-__pycache__
-
-# Directory structure
-networks/*
-resources/*
-results/*
-
-benchmarks/*
-logs/*
-
-data/*
+# Snakemake internals
+.snakemake*
+__pycache__
+
+~$*
+**/.vscode/
+
+archive/*
+benchmarks/*
+cutouts/*
+docs/build/*
+data/archive/*
+data/bundle/*
+data/weather/*
+envs/archive/*
+logs/*
+networks/*
+notebooks/archive/*
+notebooks/private/*
+pre_processing/archive/*
+pre_processing/resource_processing/archive/*
+pre_processing/resource_processing/private/*
+pre_processing/resource_processing/availability/*
+pre_processing/resource_processing/timeseries_data/*
+resources/*
+results/*
+scenarios/archive/*
+scenarios/private/*
+scripts/archive/*
+scripts/private/*
+
+
+
+
+
+
+
+
+
+

.readthedocs.yaml

@@ -0,0 +1,16 @@
+version: 2
+
+build:
+  os: "ubuntu-20.04"
+  tools:
+    python: "3.8"
+sphinx:
+  fail_on_warning: false
+  configuration: docs/conf.py
+formats:
+  - pdf
+  - epub
+python:
+  # Install our python package before building the docs
+  install:
+    - requirements: docs/requirements.txt

.vscode/launch.json

@@ -0,0 +1,17 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Python: Current File",
+            "type": "python",
+            "request": "launch",
+            "program": "${file}",
+            "console": "integratedTerminal",
+            "justMyCode": true,
+            "cwd": "${fileDirname}"
+        }
+    ]
+}

.vscode/settings.json

@@ -0,0 +1,8 @@
+{
+    "editor.fontSize": 18,
+    "markdown.preview.fontSize": 18,
+    "terminal.integrated.fontSize": 18,
+    "scm.inputFontSize": 18,
+    "cSpell.language": "en-GB",
+    "taipyStudio.gUI.elementsFilePaths": []
+}

README.md

@@ -1,76 +1,78 @@
-# PyPSA-ZA
-
-[PyPSA](https://pypsa.org/) model of the South African electricity system at the level of ESKOM's supply regions.
-
-![Visualisation of optimal capacities and costs in the least cost scenario](imgs/network_csir-moderate_redz_E_LC_p_nom_ext.png)
-
-The model is described and evaluated in the paper [PyPSA-ZA: Investment and operation co-optimization of integrating wind and solar in South Africa at high spatial and temporal detail](https://arxiv.org/abs/1710.11199), 2017, [arXiv:1710.11199](https://arxiv.org/abs/1710.11199).
-
-This repository contains the scripts to automatically reproduce the analysis.
-
-## Instructions
-
-To build and solve the model, a computer with about 20GB of memory with a strong
-interior-point solver supported by the modelling library
-[PYOMO](https://github.com/Pyomo/pyomo) like Gurobi or CPLEX are required.
-
-We recommend as preparatory steps (the path before the `%` sign denotes the
-directory in which the commands following the `%` should be entered):
-
-1. cloning the repository using `git` (**to a directory without any spaces in the path**)
-   ```shell
-   /some/other/path % cd /some/path/without/spaces
-   /some/path/without/spaces % git clone https://github.com/FRESNA/pypsa-za.git
-   ```
-
-2. installing the necessary python dependencies using conda (from within the `pypsa-za` directory)
-   ```shell
-   .../pypsa-za % conda env create -f environment.yaml
-   .../pypsa-za % source activate pypsa-za  # or conda activate pypsa-za on windows
-   ```
-
-3. getting the separate [data bundle](https://vfs.fias.science/d/f204668ef2/files/?p=/pypsa-za-bundle.7z&dl=1) (see also [Data dependencies] below) and unpacking it in `data`
-   ```shell
-   .../data % wget "https://vfs.fias.science/d/f204668ef2/files/?dl=1&p=/pypsa-za-bundle.7z"
-   .../data % 7z x pypsa-za-bundle.7z
-   ```
-
-All results and scenario comparisons are reproduced using the workflow
-management system `snakemake`
-```shell
-.../pypsa-za % snakemake
-[... will take about a week on a recent computer with all scenarios ...]
-```
-
-`snakemake` will first compute several intermediate data files in the directory
-`resources`, then prepare unsolved networks in `networks`, solve them and save
-the resulting networks in `results/version-0.x/networks` and finally render the
-main plots into `results/version-0.5/plots`.
-
-Instead of computing all scenarios (defined by the product of all wildcards in
-the `scenario` config section), `snakemake` also allows to compute only a
-specific scenario like `csir-aggressive_redz_E_LC`:
-```shell
-.../pypsa-za % snakemake results/version-0.5/plots/network_csir-aggressive_redz_E_LC_p_nom
-```
-
-## Data dependencies
-
-For ease of installation and reproduction we provide a bundle
-[`pypsa-za-bundle.7z`](https://vfs.fias.science/d/f204668ef2/files/?p=/pypsa-za-bundle.7z&dl=1)
-with the necessary data files:
-
-| File                                               | Citation                                                                                                                                                                                                       |
-|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| South_Africa_100m_Population                       | WorldPop, South Africa 100m Population (2013). [doi:10.5258/soton/wp00246](https://doi.org/10.5258/soton/wp00246)                                                                                             |
-| Supply area normalised power feed-in for PV.xlsx   | D. S. Bofinger, B. Zimmermann, A.-K. Gerlach, D. T. Bischof-Niemz, C. Mushwana, [Wind and Solar PV Resource Aggregation Study for South Africa](https://www.csir.co.za/csir-energy-centre-documents). (2016). |
-| Supply area normalised power feed-in for Wind.xlsx | same as above                                                                                                                                                                                                 |
-| EIA_hydro_generation_2011_2014.csv                 | U.S. EIA, [Hydroelectricity Net Generation ZA and MZ 2011-2014](http://tinyurl.com/EIA-hydro-gen-ZA-MZ-2011-2014) (2017).                                                                                     |
-| Existing Power Stations SA.xlsx                    | Compiled by CSIR from [Eskom Holdings](https://www.eskom.co.za/) (Jan 2017) and RSA DOE, [IRP2016](http://www.energy.gov.za/IRP/2016/Draft-IRP-2016-Assumptions-Base-Case-and-Observations-Revision1.pdf)     |
-| Power_corridors                                    | RSA DEA, [REDZs Strategic Transmission Corridors](https://egis.environment.gov.za/) (Apr 2017)                                                                                                                |
-| REDZ_DEA_Unpublished_Draft_2015                    | RSA DEA, [Wind and Solar PV Energy Strategic Environmental Assessment REDZ Database](https://egis.environment.gov.za/) (Mar 2017)                                                                             |
-| SACAD_OR_2017_Q2                                   | RSA DEA, [South Africa Conservation Areas Database (SACAD)](https://egis.environment.gov.za/) (Jun 2017)                                                                                                      |
-| SAPAD_OR_2017_Q2                                   | RSA DEA, [South Africa Protected Areas Database (SAPAD)](https://egis.environment.gov.za/) (Jun 2017)                                                                                                         |
-| SystemEnergy2009_13.csv                            | Eskom, System Energy 2009-13 Hourly, available from Eskom on request                                                                                                                                          |
-| SALandCover_OriginalUTM35North_2013_GTI_72Classes  | GEOTERRAIMAGE (South Africa), [2013-14 South African National Land-Cover Dataset](https://egis.environment.gov.za/data_egis/node/109) (2017)                                                                  |
-
+<!--
+SPDX-FileCopyrightText:  PyPSA-RSA, PyPSA-Eur and PyPSA-Earth Authors
+
+SPDX-License-Identifier: AGPL-3.0-or-later
+-->
+
+# Development Status: **Under development**
+
+[![Documentation Status](https://readthedocs.org/projects/pypsa-earth/badge/?version=latest)](https://pypsa-rsa.readthedocs.io/en/latest/?badge=latest)
+[![License: AGPL v3](https://img.shields.io/badge/License-AGPLv3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
+[![Discord](https://img.shields.io/discord/911692131440148490?logo=discord)](https://discord.gg/bsdHkHwujt)
+[![Google Drive](https://img.shields.io/badge/Google%20Drive-4285F4?style=flat&logo=googledrive&logoColor=white)](https://drive.google.com/drive/folders/17f54zTMEfeFZhNByXxLkf9qcZRdhng03)
+
+# PyPSA-RSA: An open Optimisation Model of the South African Power System
+The accelerating development of open-source energy system modelling tools in recent years has now reached the point where it opens up a credible alternative approach to closed source exclusivity. An ever increasing number of studies are demonstrating that it is possible to produce analysis of a high quality using open-source energy system models, whilst building a wider participating modelling community. This builds confidence in results by enabling more effective peer review of work and therefore more effective feedback loops. It also builds a consistent stream of new talent entering the space to ensure that energy sector analytical capacity can be retained and steadily expanded.
+
+This model makes use of freely available and open data which encourages the open exchange of model data developments and eases the comparison of model results. It provides a full, automated software pipeline to assemble the optimisation model from the original datasets, which enables easy replacement and improvement of the individual parts. Running the model requires a wide range on input datasets. Users are required to access the following datasets from the original sources:
+
+- [GIS shape files for supply regions](https://www.ntcsa.co.za/wp-content/uploads/2024/06/GCCA-2025-GIS.zip)
+- [GIS data for existing Eskom transmission lines](https://www.ntcsa.co.za/wp-content/uploads/2024/06/Shapefiles-1.zip)
+- [GIS data for population and GVA](http://stepsatest.csir.co.za/socio_econ.html)
+
+Custom data generated for this model will need to be downloaded from [Google Drive](https://drive.google.com/drive/folders/17f54zTMEfeFZhNByXxLkf9qcZRdhng03)
+
+# Spatial resolution
+PyPSA-RSA has been designed to conduct capacity expansion planning and resource adequacy studies at differing spatial and temporal resolutions. Five different spatial resolutions are available in the model, and custom GIS shpae files can also be utilsied:
+
+- ``1-supply``: A single node for the entire South Africa.
+- ``10-supply``: 10 nodes based on the [GCCA 2025 Eskom Transmission Supply Regions](https://www.eskom.co.za/eskom-divisions/tx/gcca/).
+- ``27-supply``: 27 nodes based on the Eskom 27 supply regions as per the original PyPSA-ZA model.
+- ``34-supply``: 34 nodes based on the CLNs level in the [GCCA 2025 Eskom Local Supply Regions](https://www.eskom.co.za/eskom-divisions/tx/gcca/).
+- ``159-supply``: 159 nodes based on the [GCCA 2025 Eskom Transmission MTS Regions](https://www.eskom.co.za/eskom-divisions/tx/gcca/).
+
+![Spatial resolutions](docs/img/pypsa-rsa_spatial2.png)
+
+PyPSA-RSA can be solved for a single year, or for multiple years, with perfect foresight. Multi-horizon capacity expansion planning is compuationally intensive, and therefore the spatial resolution will typically need to be reduced to ``1-supply`` or ``10-supply`` depending on the number of years modelled. By defualt PyPSA-RSA uses full chronology (8760h per year), but the number of snapshots can be reduced through the use of time-series 
+segmentation through the open-source [Time Series Aggregation Module (TSAM)]( https://github.com/FZJ-IEK3-VSA/tsam/). 
+
+# Resources to get started
+PyPSA-RSA is built upon the fundamental components in the [PyPSA library](https://pypsa.org/). Before starting to use this model it is highly recommended to explore the [PyPSA documentation](https://pypsa.readthedocs.io/en/latest/index.html). More detailed documentation can be found under our [readthedocs](https://pypsa-za.readthedocs.io/en/latest/). A two-part video series on using PyPSA and PyPSA-RSA can be found under [link](https://meridianeconomics.co.za/our-publications/pypsa-rsa-workshop1/). Please feel free to join our [Discord channel](https://discord.gg/bsdHkHwujt) to ask questions. 
+
+PyPSA-RSA assembles the mathematical equations that need to be solved to generate a result. Whilst open-source solvers such as [HiGHS](https://highs.dev/) and [CBC](https://github.com/coin-or/Cbc) can be used to solve smaller problems, a commercial solver is likely to be required for multi-horizon capacity expansion models. Suitable options include:
+- [Gurobi](https://www.gurobi.com/) 
+- [IBM CPLEX](https://www.ibm.com/products/ilog-cplex-optimization-studio/cplex-optimizer)
+- [FICO Xpress](https://www.fico.com/en/products/fico-xpress-optimization)
+- [COPT](https://www.shanshu.ai/copt)
+- [MindOpt](https://opt.alibabacloud.com/)
+
+RAM requirements will vary based on the complexity of the model. Multi-horizon capacity expansion planning models with full time chronology will likely require between 65-128GB of RAM. 
+
+# Installation
+1. Open your terminal at a location where you want to install pypsa-rsa. Type the following in your terminal to download the package from GitHub:
+    ```bash
+        .../some/path/without/spaces % git clone https://github.com/MeridianEconomics/pypsa-rsa.git
+    ```
+2. The python package requirements are curated in the `envs/environment.yaml` file.
+   The environment can be installed using:
+
+    ```bash
+        .../pypsa-rsa % conda env create -f envs/environment.yaml
+    ```
+
+   If the above takes longer than 30min, you might want to try mamba for faster installation:
+
+    ```bash
+        (base) conda install -c conda-forge mamba
+
+        .../pypsa-rsa % mamba env create -f envs/environment.yaml
+    ```
+
+3. For running the optimization one has to install the solver. We can recommend the open source HiGHs solver which installation manual is given [here](https://github.com/PyPSA/PyPSA/blob/633669d3f940ea256fb0a2313c7a499cbe0122a5/pypsa/linopt.py#L608-L632).
+
+# Documentation
+The documentation is available here: [documentation](https://pypsa-rsa.readthedocs.io/en/latest/?badge=latest).
+
+# Developers
+New collaborators are welcome! This project is currently maintained by [Meridian Economics]( https://meridianeconomics.co.za/). Previous versions were developed within the Energy Centre at the [Council for Scientific and Industrial Research (CSIR)](https://www.csir.co.za/) as part of the [CoNDyNet project](https://fias.institute/en/projects/condynet/), which is supported by the [German Federal Ministry of Education and Research](https://www.bmbf.de/bmbf/en/home/home_node.html) under grant no. 03SF0472C. Credits to Jonas Hörsch and Joanne Calitz who developed the original [PyPSA-ZA model](https://arxiv.org/pdf/1710.11199.pdf), [Meridian Economics](http://meridianeconomics.co.za) who extended the PyPSA-ZA model.PyPSA-RSA is relies on a number of functions from the [PyPSA-Eur](https://github.com/PyPSA/pypsa-eur) and [PyPSA-Meets-Earth](https://github.com/pypsa-meets-earth/pypsa-earth). 
+