Merge pull request #447 from PyPSA/develop

v0.4.0 Release

Author: ktehranchi

.pre-commit-config.yaml

@@ -51,6 +51,7 @@ repos:
   - id: docformatter
     args: ["--in-place", "--make-summary-multi-line", "--pre-summary-newline"]
 
+
 - repo: https://github.com/keewis/blackdoc
   rev: v0.3.9
   hooks:
@@ -60,4 +61,15 @@ repos:
   rev: 24.10.0
   hooks:
   - id: black
+    language_version: python3.11
+    args: ["--line-length", "120"]
   - id: black-jupyter
+    language_version: python3.11
+    args: ["--line-length", "120"]
+
+
+- repo: https://github.com/pycqa/flake8
+  rev: 7.1.1
+  hooks:
+  - id: flake8
+    args: [--select=F841]    # F841 is the error code for unused variables

docs/source/_static/networks/FERC1000.png

@@ -6,40 +6,43 @@ Users can clone the repository using HTTPS, SSH, or GitHub CLI. See [GitHub docs
 
 ### Using HTTPS
 
-```bash
-$ git clone https://github.com/PyPSA/pypsa-usa.git
+```console
+git clone https://github.com/PyPSA/pypsa-usa.git
 ```
 
 ### Using SSH-Key
 
 If it your first time cloning a **repository through ssh**, you will need to set up your git with an ssh-key by following these [directions](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
 
-```bash
-$ git clone [email protected]:PyPSA/pypsa-usa.git
+```console
+git clone [email protected]:PyPSA/pypsa-usa.git
 ```
 
 ## Step 2. Initialize Configuration files
 
 From the command line, run the script `init_pypsa_usa.sh` to copy configuration file
 templates into the `workflow/config` folder.
 
-```bash
-$ bash init_pypsa_usa.sh
+```console
+bash init_pypsa_usa.sh
 ```
 
-## Step 3: Create Conda Environment
+## Step 3: Create Mamba Environment
 
 PyPSA-USA uses conda/mamba to manage project dependencies. You can download and install mamba following the [instructions](https://mamba.readthedocs.io/en/latest/mamba-installation.html). Follow links for mambaforge installation. There are two ways to install mamba, the first (recommended) method will start with a fresh install, meaning if you have previously installed conda environments, you will need to recreate these conda envs. If you already have conda installed and do not wish to install mamba, you can follow the same set of instructions replacing any `mamba` with `conda`
 
 Once mamba is installed, use the environment file within your git repository to activate the `pypsa-usa` conda environment. This step can take ~10-20 minutes. After creating the mamba environment, you will only need to activate it before running the snakemake workflow.
 
-```bash
-$ mamba env create -f workflow/envs/environment.yaml
-$ mamba activate pypsa-usa
+```console
+mamba env create -f workflow/envs/environment.yaml
+mamba activate pypsa-usa
 ```
 
 You also have the option to use miniconda. Download [Miniconda](https://docs.conda.io/en/latest/miniconda.html) following their [instructions](https://docs.conda.io/en/latest/miniconda.html).
 
+```{seealso}
+If you are planning to develop PyPSA-USA, please see our [contribution guidelines](./contributing.md#code-contributions) for installing additional dependencies
+```
 
 ## Step 4: Install a Solver
 

docs/source/about-install.md

@@ -30,7 +30,7 @@ The diagram below illustrates the workflow of PyPSA-USA, highlighting how the da
 
 PyPSA-USA is organized to facilitate easy navigation and efficient execution. Below is the folder structure of the project. Each folder serves a specific purpose, from environment setup to data processing and storage. After running the Snakemake file for the first time, your directory will be built and populated with the necessary data files.
 
-```bash
+```console
 ├── .gitignore
 ├── README.md
 ├── LICENSE.md

docs/source/about-introduction.md

@@ -16,7 +16,7 @@ You can find more information on each configuration setting on the [configuratio
 
 To run the workflow, `cd` into the `workflow` directory and run the `snakemake` from your terminal with your selection of config file:
 
-```bash
+```console
 snakemake -j1 --configfile config/config.default.yaml
 ```
 
@@ -30,7 +30,7 @@ Next, identify the name of the configuration file you would like to run by editi
 
 To run, open a terminal within a login node of your cluster and run the script included in the `workflow` directory:
 
-```bash
+```console
 bash run_slurm.sh
 ```
 
@@ -45,7 +45,7 @@ Result plots and images are automatically built in the `workflow/results` folder
 
 To force the execution of a portion of the workflow up to a given rule, cd to the `workflow` directory and run:
 
-```bash
+```console
 snakemake -j4 -R build_shapes  --until build_base_network
 ```
 where `build_shapes` is forced to run, and `build_base_network` is the last rule you would like to run.

docs/source/about-usage.md

@@ -215,7 +215,6 @@ Sector coupling studies are all under active development. More info to come!
 (clustering_cf)=
 ## `clustering`
 
-When clustering `aggregation_zones` defines the region boundaries which will be respected through the clustering process; State boarders, balancing authority regions, or REeDs shapes. This feature is important for imposing constraints (`opts`) which are defined over specific regions. For example, the data included in the model on interface transfer capacities are prepared for REeDs shapes but not states and BA regions. Moving forward we plan to use REeDs shapes as our default however we will maintain States and BA regions as well.
 
 Each clustering and interconnection option will have a different number of minimum nodes which can be clustered to, an error will be thrown in `cluster_network` notifying you of that number if you have selected a value too low.
 

docs/source/config-configuration.md

@@ -0,0 +1,80 @@
+# Spatial Configuration
+
+## Configuring Spatial Scope
+
+PyPSA-USA allows for flexible configuration of the spatial scope of your energy model, enabling you to define the geographical area and level of detail for your simulations. The **spatial scope** is determined by the `interconnect`, `model_topology` configuration settings.
+
+- `{interconnect}` is used to select which of the three asynchronous interconnection to model. You can select `western`, `eastern`, `texas`, or `usa` to model the entire US.
+- After selecting your `{interconnect}`, you can specify `model_topology: include:` to filter individual states or balancing authorities to be selected in your model.
+
+### Example: Modeling California
+
+To create a model that includes only California, you can specify the relevant ReEDS zone IDs (p8-11) as shown below. This will limit the spatial scope to the specified regions within California. Your interconnect could be set to `western` or `usa`.
+
+```yaml
+model_topology:
+   include:
+      reeds_zone: ['p8', 'p9', 'p10', 'p11']
+```
+
+Alternatively, you can use the reeds_state: 'CA' option to achieve the same result by specifying the entire state.
+
+## Configuring Resource Resolution
+PyPSA-USA allows you to independently configure the resolution of resource zones from the transmission network. You can control this using the simpl and clusters parameters in the configuration file.
+
+For example, if you want a transmission network with 10 nodes and a resource model with 100 nodes, you would configure it as follows:
+
+```yaml
+scenario:
+   clusters: [10m]
+   simpl: [100]
+```
+
+This setup results in a model with 10 transmission nodes and 100 distinct resource zones, allowing for more granular modeling of renewable resource distribution while keeping the transmission network simplified.
+
+## Configuring Transmission Resolution
+
+### Transmission Network Selection
+You can specify the transmission network you want to use by setting the `model_topology: transmission_network:` option. There are two available options:
+
+- 'reeds': The ReEDS NARIS networks.
+- 'tamu': The synthetic BE-TAMU nodal network.
+
+When selecting between the three ReEDS NARIS networks, you will need to also specify the `model_topology: topological_boundaries:`. Currently you can set either `county` or `reeds_zone`. To use the FERC 1000 regions, you will need to use the custom network topologies described in the example below.
+
+### Transmission Network Resolution
+
+IF you are using the TAMU/BE network, you can flexibly set an arbitrary number of clusters between the min and max number of nodes. If using a ReEDS NARIS network, you need to specify the minimum number of clusters (nodes) for your modeled interconnection. The number of nodes for each zone is **detailed in the table below**.
+
+If you're working with custom configurations, PyPSA-USA will notify you during the cluster_network stage, indicating the correct number of nodes to set in the clusters configuration.
+
+```{eval-rst}
+.. csv-table::
+   :header-rows: 1
+   :widths: 22,22,33
+   :file: datatables/transmission_nodes.csv
+```
+
+#### Example: Meshed ReEDS NARIS WECC Topology
+
+If you would like to mesh the three ReEDS NARIS networks you can do so by using the `model_topology: aggregate:` option. For instance, to create a model where California is represented at a county level resolution, but Non-CA WECC areas are represented at the FERC 1000 level, you would configure the model as follows:
+
+
+```yaml
+scenario:
+   interconnect: [western]
+   clusters: [87]
+   simpl: [380] # can be set differently based on number of resource zones you'd like to keep
+
+
+model_topology:
+  transmission_network: 'reeds'
+  topological_boundaries: 'county'
+  interface_transmission_limits: false
+  include:
+    # nothing specified here since we are modeling the entire WECC
+  aggregate:
+    trans_grp: ['NorthernGrid_South', 'NorthernGrid_West', 'NorthernGrid_East', 'WestConnect_North','WestConnect_South']
+```
+
+This configuration will copper plate the Non-CA regions listed under `trans_grp`, effectively creating a copper-plate network where resources can be clustered and shared across the region. Using these custom aggregation requires information on the region memberships which you can find in `workflow/repo_data/ReEDS_Constraints/membership.csv`.

docs/source/config-spatial.md

@@ -48,35 +48,28 @@ The `{ll}` wildcard specifies what limits on
 line expansion are set for the optimisation model.
 It is handled in the rule :mod:`prepare_network`.
 
-The wildcard, in general, consists of two parts:
+We reccomend using the line volume limit for constraining
+transission expansion. Use ``lv`` (for setting a limit on line volume)
 
-    1. The first part can be
-       ``v`` (for setting a limit on line volume) or
-       ``c`` (for setting a limit on line cost)
+After ``lv`` you can specify two type of limits:
 
-    2. The second part can be
        ``opt`` or a float bigger than one (e.g. 1.25).
 
        (a) If ``opt`` is chosen line expansion is optimised
-           according to its capital cost
-           (where the choice ``v`` only considers overhead costs for HVDC transmission lines, while
-           ``c`` uses more accurate costs distinguishing between
-           overhead and underwater sections and including inverter pairs).
+           according to its capital cost.
 
        (b) ``v1.25`` will limit the total volume of line expansion
            to 25 % of currently installed capacities weighted by
            individual line lengths; investment costs are neglected.
 
-       (c) ``c1.25`` will allow to build a transmission network that
-           costs no more than 25 % more than the current system.
 
 (opts)=
 ## The `{opts}` wildcard
 
 The `{opts}` wildcard is used for electricity-only studies. It triggers
 optional constraints, which are activated in either :mod:`prepare_network` or
 the :mod:`solve_network` step. It may hold multiple triggers separated by `-`,
-i.e. `Co2L-3H` contains the `Co2L` trigger and the `3H` switch.
+i.e. `REM-3H` contains the `REM` regional emissions limit trigger and the `3H` switch.
 
 The REM, SAFER, RPS can be defined using either the reeds zone name 'p##"
 the state code (eg, TX, CA, MT), pypsa-usa interconnect name (western, eastern, texas, usa),

docs/source/config-wildcards.md

@@ -6,8 +6,7 @@ feature,str," {'solar+onwind-time', 'solar+onwind-cap', 'solar-time', 'solar-cap
 cluster_network:,,,
 algorithm,str,{'kmeans'},
 feature,str," {'solar+onwind-time', 'solar+onwind-cap', 'solar-time', 'solar-cap', 'solar+offwind-cap'}",For HAC clustering.
-aggregation_zones,str,"{'balancing_area', 'state', 'reeds_zone'}",Boundaries of GIS shapes that are to be respected in clustering. Retain if you would like to analyze expansion within a given zone.
 aggregation_strategies:,,,
 table --> {key},str,"{'mean','max','min',etc}","Specifiy the method of aggregating fields within the generators, buses tables. "
 focus_weights:,,,
-region_name',float,,Specify the proportion of nodes to be attributed to a given zone in the form (California: 0.5) for half of all nodes to be located in California
+region_name',float,,Specify the proportion of nodes to be attributed to a given zone in the form (California: 0.5) for half of all nodes to be located in California. Only used with TAMU Network

docs/source/configtables/clustering.csv

@@ -1,17 +1,13 @@
 ,Unit,Values,Description
 conventional_carriers,--,"Any subset of {nuclear, oil, OCGT, CCGT, coal, geothermal, biomass}","List of conventional power plants to include in the model from ``resources/powerplants.csv``. If an included carrier is also listed in ``extendable_carriers``, the capacity is taken as a lower bound."
 renewable_carriers,--,"Any subset of {solar, onwind, offwind-ac, offwind-dc, hydro}",List of renewable generators to include in the model.
-voltage_simplified,kV,int,Voltage level to simplify network to in rule ``simplify_network``
-gaslimit,MWhth,float or false,Global gas usage limit (Set False for development)
-co2limit,:math:`t_{CO_2-eq}/a`,float,Cap on total annual system carbon dioxide emissions
-co2base,:math:`t_{CO_2-eq}/a`,float,Reference value of total annual system carbon dioxide emissions if relative emission reduction target is specified in ``{opts}`` wildcard.
 retirement, --,One of ``economic`` or ``technical``,"Sets the retirement method for converntional generators. If ``technical`` all generators ``p_nom_min`` are set to ``p_nom`` to prevent selling off of the asset. Retirements are then tracked in post-proccessing. If ``economic`` existing plants have their ``p_nom_min`` set as ``0``,  ``p_nom_max`` set to ``p_nom``,  and capital costs set to fixed costs. Generators with ``p_nom`` are then added to handle capacity expansion."""
 ,,,
 operational_reserve:,,,Settings for reserve requirements following `GenX <https://genxproject.github.io/GenX.jl/stable/Model_Reference/core/#Operational-Reserves>`_
---activate,bool,true or false,Whether to take operational reserve requirements into account during optimisation
---epsilon_load,--,float,share of total load
---epsilon_vres,--,float,share of total renewable supply
---contingency,MW,float,fixed reserve capacity
+#NAME?,bool,true or false,Whether to take operational reserve requirements into account during optimisation
+#NAME?,--,float,share of total load
+#NAME?,--,float,share of total renewable supply
+#NAME?,MW,float,fixed reserve capacity
 ,,,
 max_hours:,,,
 battery,h,float,Maximum state of charge capacity of the battery in terms of hours at full output capacity ``p_nom``. Cf. `PyPSA documentation <https://pypsa.readthedocs.io/en/latest/components.html#storage-unit>`_.
@@ -23,14 +19,14 @@ Store,--,Any subset of {``battery``},Adds extendable storage units (battery and/
 Links,--,Any subset of {},Adds extendable linksat every connection where there are lines or HVDC links without capacity limits and with zero initial capacity. Hydrogen pipelines require hydrogen storage to be modelled as ``Store``.
 ,,,
 demand:,,,
---profile,--,"One of {``efs``, ``eia``}",Datasource for electrical load data. ``EFS`` pulls future state level electrical demand data. ``EIA`` pulls historical balancing level electrical demand dataa.
---scale,--,"One of {``efs``, ``aeo``}, or a float",(UNDER DEVELOPMENT) Used to scale the demand profile data. ``AEO`` will scale according to demand projections from the Annual Energy Outlook. ``EFS`` will scale according to growth rates from the Electrification Futures Study. Or can sclae according to a user defined value.
---disaggregation,--,One of {``pop``},Method to dissagreagate load data. ``pop`` will dissagregate based on population from Breakthrough Energy.
+#NAME?,--,"One of {``efs``, ``eia``}",Datasource for electrical load data. ``EFS`` pulls future state level electrical demand data. ``EIA`` pulls historical balancing level electrical demand dataa.
+#NAME?,--,"One of {``efs``, ``aeo``}, or a float",(UNDER DEVELOPMENT) Used to scale the demand profile data. ``AEO`` will scale according to demand projections from the Annual Energy Outlook. ``EFS`` will scale according to growth rates from the Electrification Futures Study. Or can sclae according to a user defined value.
+#NAME?,--,One of {``pop``},Method to dissagreagate load data. ``pop`` will dissagregate based on population from Breakthrough Energy.
 scenario:,,,
---efs_case,--,"One of {``reference``, ``medium``, ``high``}",(UNDER DEVELOPMENT) Extracts EFS data according to level of adoption
---efs_speed,--,"One of {``slow``, ``moderate``, ``fast``}",(UNDER DEVELOPMENT) Extracts EFS data according to speed of electrification
---aeo,--,One of the AEO scenarios `here <https://www.eia.gov/outlooks/aeo/data/browser/>`_,(UNDER DEVELOPMENT) Scales future demand according to the AEO scenario
+#NAME?,--,"One of {``reference``, ``medium``, ``high``}",(UNDER DEVELOPMENT) Extracts EFS data according to level of adoption
+#NAME?,--,"One of {``slow``, ``moderate``, ``fast``}",(UNDER DEVELOPMENT) Extracts EFS data according to speed of electrification
+#NAME?,--,One of the AEO scenarios `here <https://www.eia.gov/outlooks/aeo/data/browser/>`_,(UNDER DEVELOPMENT) Scales future demand according to the AEO scenario
 ,,,
 autarky,,,
---enable,bool,``true`` or ``false``,Require each node to be autarkic by removing all lines and links.
---by_country,bool,``true`` or ``false``,Require each region to be autarkic by removing all cross-border lines and links. ``electricity: autarky`` must be enabled.
+#NAME?,bool,``true`` or ``false``,Require each node to be autarkic by removing all lines and links.
+#NAME?,bool,``true`` or ``false``,Require each region to be autarkic by removing all cross-border lines and links. ``electricity: autarky`` must be enabled.