Merge branch 'master' into fix-fn-consistency

Author: tgilon

.pre-commit-config.yaml

@@ -14,7 +14,7 @@ repos:
 # Run ruff to lint and format
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
-  rev: v0.9.4
+  rev: v0.9.9
   hooks:
     # Run the linter.
   - id: ruff

Snakefile

@@ -2,6 +2,8 @@
 #
 # SPDX-License-Identifier: GPL-3.0-only
 
+import pathlib
+
 
 configfile: "config.yaml"
 
@@ -32,6 +34,8 @@ rule compile_cost_assumptions:
         mem=500,
     conda:
         "environment.yaml"
+    log:
+        pathlib.Path("logs", "compile_cost_assumptions.log"),
     script:
         "scripts/compile_cost_assumptions.py"
 
@@ -40,18 +44,22 @@ rule compile_cost_assumptions_usa:
     input:
         cost_files_to_modify=expand("outputs/costs_{year}.csv", year=config["years"]),
         nrel_atb_input_files=expand(
-            "inputs/atb_e_{year}.parquet",
+            "inputs/US/atb_e_{year}.parquet",
             year=config["nrel_atb"]["nrel_atb_input_years"],
         ),
-        nrel_atb_input_discount_rate="inputs/discount_rates_usa.csv",
-        nrel_atb_input_fuel_costs="inputs/fuel_costs_usa.csv",
+        nrel_atb_manual_input_usa="inputs/US/manual_input_usa.csv",
+        eur_inflation_rate="inputs/prc_hicp_aind__custom_9928419_spreadsheet.xlsx",
+        nrel_atb_input_discount_rate="inputs/US/discount_rates_usa.csv",
+        nrel_atb_input_fuel_costs="inputs/US/fuel_costs_usa.csv",
     output:
         expand("outputs/US/costs_{year}.csv", year=config["years"]),
     threads: 1
     resources:
         mem=500,
     conda:
         "environment.yaml"
+    log:
+        pathlib.Path("logs", "compile_cost_assumptions_usa.log"),
     script:
         "scripts/compile_cost_assumptions_usa.py"
 

docs/release_notes.rst

@@ -16,10 +16,19 @@ Upcoming Release
 ..   The features listed below are not released yet, but will be part of the next release! 
 ..   To use the features already you have to use the ``master`` branch.
 
+* Include unit test for compile_cost_assumptions_usa.py (https://github.com/PyPSA/technology-data/pull/170)
+
+* US specific folder for NREL/ATB data (https://github.com/PyPSA/technology-data/pull/172)
+
 * Include unit test execution and compile_cost_assumptions_usa.py in ci.yaml (https://github.com/PyPSA/technology-data/pull/174)
+
 * Align `snakemake` version and the related `mock_snakemake` to PyPSA-Eur (https://github.com/PyPSA/technology-data/pull/177)
 * Improve filename consistency in the sources (https://github.com/PyPSA/technology-data/pull/178)
 
+* Include further unit tests for compile_cost_assumptions_usa.py (https://github.com/PyPSA/technology-data/pull/182)
+
+* Updates the documentation with compile_cost_assumptions_usa.py (https://github.com/PyPSA/technology-data/pull/186)
+
 `v0.11.0 <https://github.com/PyPSA/technology-data/releases/tag/v0.11.0>`__ (24th January 2025)
 =======================================================================================
 

docs/structure.rst

@@ -11,37 +11,61 @@ Structure of the repository
 
 This repository has the following structure:
 
--  **inputs**: input data in format .csv or .xlsx
+-  **inputs**: input data in format .csv, .xlsx or .parquet
 
 -  **outputs**: technology data saved as ``costs_{year}.csv`` format for defined years. In the output ``costs_{year}.csv`` are specified
 
-	* technology (e.g. 'onwind')
-	* parameter (e.g. FOM)
-	* value (e.g. 1.18)
-	* unit (e.g. %/year)
-	* source (e.g. DEA, excel_file_name.xlsx)
-	* further description (specific assumptions, sheet name if data from a multi-sheet Excel file)
+    * technology (e.g. 'onwind')
+    * parameter (e.g. FOM)
+    * value (e.g. 1.18)
+    * unit (e.g. %/year)
+    * source (e.g. DEA, excel_file_name.xlsx)
+    * further description (specific assumptions, sheet name if data from a multi-sheet Excel file)
+    * currency_year (year used for adjusting economic values to reflect current purchasing power)
+
+-  **outputs/US**: US-specific technology data saved as ``US/costs_{year}.csv`` format for defined years. In the outputs ``US/costs_{year}.csv`` are specified
+
+    * technology (e.g. 'onwind')
+    * parameter (e.g. FOM)
+    * value (e.g. 1.18)
+    * unit (e.g. %/year)
+    * source (e.g. DEA, excel_file_name.xlsx)
+    * further description (specific assumptions, sheet name if data from a multi-sheet Excel file)
+    * currency_year (year used for adjusting economic values to reflect current purchasing power)
+    * financial_case (financial assumptions for the definition of the cost of capital)
+    * scenario (technology innovation scenario)
 
 -  **config**:
 
 	.. literalinclude:: ../config.yaml
 	   :language: yaml
-	   :lines: 4-24
-
- the following parameters can be set in the ``config.yaml``
-
-	* years : numpy array of all the years of which an output costs csv should be created
-	* rate_inflation : inflation rate (currently: rate_inflation=0.02)
-	* solar_utility_from_vartiaien : Bool (True/False) if solar utility data is taken from DEA or Vartiaien
-	* solar_rooftop_from_etip : Bool (True/False) if solar rooftop data is taken from DEA or ETIP
-	* h2_from_budischak : Bool (True/False) if fuel cell and electrolyzer efficiencies are taken from DEA or Budischak
-	* offwind_no_gridcosts : Bool (True/False) if offshore wind grid connection costs should be removed (they are calculated seperately in PyPSA-Eur)
+	   :lines: 4-40
+
+ The following parameters can be set in the ``config.yaml``
+
+    * years : numpy array of all the years of which an output costs csv should be created
+    * nrel_atb_input_years : list of years that define the source files
+    * nrel_atb_columns_to_keep : list of columns to use from the NREL/ATB source files
+    * nrel_atb_core_metric_parameter_to_keep : list of parameters to use from the NREL/ATB source files
+    * nrel_atb_technology_to_remove : list of technologies that should be excluded from NREL/ATB
+    * nrel_atb_source_link : source url for the NREL/ATB data used
+    * expectation : tech data uncertainty, possible options [None, "optimist", "pessimist"]
+    * eur_year : year for EUR outputs
+    * solar_utility_from_vartiaien : Bool (True/False) if solar utility data is taken from DEA or Vartiaien
+    * solar_rooftop_from_etip : Bool (True/False) if solar rooftop data is taken from DEA or ETIP
+    * h2_from_budischak : Bool (True/False) add fuel cell/electrolysis efficiencies from Budischak
+    * ewg_home_battery: Bool (True/False) add home battery data derived from DEA data and EWG study
+    * add_data: Bool (True/False) add storage data mainly from PNNL
+    * approx_beyond_2030: ["geometric_series"] or ["same_as_2030"]
+    * offwind_no_gridcosts : Bool (True/False) if offshore wind grid connection costs should be removed (they are calculated seperately in PyPSA-Eur)
+    * salinity : estimated in PSU (Practical Salinity Unit) = kg/m^3
+    * ndigits : number of significant digits
 
 -  **scripts** :
-
-	* :mod:`compile_cost_assumptions.py`  converts input data from multiple sources to ``cost_{year}.csv`` for chosen year. Interpolates data for missing years or calculates the costs at a certain year based on the inflation rate. Technology data from the `Danish Energy Agency <https://github.com/PyPSA/technology-data>`_ are preferred. If data are missing from all sources, these are taken from the old PyPSA cost assumptions (with a printed warning).
-	* :mod:`convert_pdf_fraunhofer_to_dataframe.py` converts table from Fraunhofer ISE report in pdf to csv format for input data. Script can be modified to convert other .pdf sources to .csv format
-	* :mod:`retrieve_data_from_dea.py` downloads up-to-date technology data from DEA website and saves it in the **input** folder. Optional, also retrieves the documentation of the data into the folder **docu**
+    * :mod:`compile_cost_assumptions.py` converts input data from multiple sources to ``cost_{year}.csv`` for chosen year. Interpolates data for missing years or calculates the costs at a certain year based on the inflation rate. Technology data from the `Danish Energy Agency <https://github.com/PyPSA/technology-data>`_ are preferred. If data are missing from all sources, these are taken from the old PyPSA cost assumptions (with a printed warning).
+    * :mod:`compile_cost_assumptions_usa.py` converts input data from NREL/ATB to ``US/cost_{year}.csv`` for chosen year. It starts from the cost assumptions files produced by `compile_cost_assumptions.py`. All technology-parameter pairs present in the NREL/ATB input data are updated. Those not present in NREL/ATB are left untouched.
+    * :mod:`convert_pdf_fraunhofer_to_dataframe.py` converts table from Fraunhofer ISE report in pdf to csv format for input data. Script can be modified to convert other .pdf sources to .csv format
+    * :mod:`retrieve_data_from_dea.py` downloads up-to-date technology data from DEA website and saves it in the **input** folder. Optional, also retrieves the documentation of the data into the folder **docu**
 
 -  **docu**: reports, paper, additional information about the input data, format .pdf