Configurable cutout directory by FabianHofmann · Pull Request #1515 · PyPSA/pypsa-eur

FabianHofmann · 2025-01-26T14:57:27Z

Closes #1504

Checklist

I tested my contribution locally and it works as intended.
Code and workflow changes are sufficiently documented.
Changed dependencies are added to envs/environment.yaml.
Changes in configuration options are added in config/config.default.yaml.
Changes in configuration options are documented in doc/configtables/*.csv.
Sources of newly added data are documented in doc/data_sources.rst.
A release note doc/release_notes.rst is added.

github-actions · 2025-01-26T15:00:50Z

Validator Report

I am the Validator. Download all artifacts here.
I'll be back and edit this comment for each new commit.

❗ Run failed!

Download 'logs' artifact to see more details.

master failed in: no_logs_found
cutout-dir failed in: no_logs_found

Model Metrics

Benchmarks

Comparing cutout-dir (e776e6e) with master (a12d724).
Branch is 5 commits ahead and 0 commits behind.
Last updated on 2025-02-06 06:50:45 CET.

* Revert "Transition from stores to storage units for LTES, introducing energy-…" (PyPSA#1520) This reverts commit 0e6a8f8. * use pypi `pycipopt` (PyPSA#1524) * Update pinned environment files for all platforms (PyPSA#1523) Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> * fix: ensure correct dtype of "reverse" column (PyPSA#1525) * fix: ensure correct dtype of "reverse" column * apply boolean dtype to reverse column in perfect-foresight * reinsert objective check * fix: improve selection of bidirectionals * fix: relax condition on sanitizing "reversed" column * revert some unneeded changes * debug: print dataframes in summary (to be reverted) * filter empty dfs in make summary * use highs-simplex instead of highs-default for ci --------- Co-authored-by: Lukas Trippe <lkstrp@pm.me> * fix(add_electricity): Fix nuclear p_max_pu time-series for greenfield nuclear (PyPSA#1519) If nuclear is only part of extendable carriers, but not of convential carriers, then the typical "conventional: nuclear: p_max_pu" setting writes the file name all over the p_max_pu time-series, breaking the add_electricity rule. Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * solve_network: add customisable logging frequency (PyPSA#1521) Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Configurable cutout directory (PyPSA#1515) * make cutout dir configurable * feat: add global cutout_directory to config * add release notes * feat: add description to configtables --------- Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * remove `highspy` pin (PyPSA#1528) * remove `highspy` pin * Update environment.yaml * Update environment.yaml --------- Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * Update pinned environment files for all platforms (PyPSA#1533) Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> * Fixing snakemake shadow directory bug that occures with Windows/SMB drives (PyPSA#1535) * Fixing snakemake shadow directory bug that occures with Windows/SMB/network drives * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1526) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.3 → v0.9.4](astral-sh/ruff-pre-commit@v0.9.3...v0.9.4) - [github.com/codespell-project/codespell: v2.4.0 → v2.4.1](codespell-project/codespell@v2.4.0...v2.4.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * Update validate.yaml (PyPSA#1536) * Update validate.yaml * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update validate.yaml --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Consistent function scope (PyPSA#1537) * start removing snakemake from functions * move snakemake objects out of prepare_sector_functions * continue with add_existing_baseyear * continue with prepare_perfect_foresight * apply it to add_electricity * continue with solve network * add pylint check to reusable functions * iron out follow up issues * adjust pylint check * Update pinned environment files for all platforms (PyPSA#1538) Co-authored-by: FabianHofmann <19226431+FabianHofmann@users.noreply.github.com> * Increase default mem_mb for config.default scope (PyPSA#1541) * Add a note to avoid updating pins without testing (PyPSA#1545) * [pre-commit.ci] pre-commit autoupdate (PyPSA#1539) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.6](astral-sh/ruff-pre-commit@v0.9.4...v0.9.6) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Update pinned environment files for all platforms (PyPSA#1544) Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> * Fix geothermal heat potentials (PyPSA#1516) * feat: refactor build_heat_source_potentials into build_geothermal_heat_potential and and use ISI Excel output as data source * feat: remove unnecessary snapshot index * chore: refactor accordingly * feat: fix scaling factor * feat: remove obsolete retrieve * docs: update docs * docs: update release notes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: use retrieve separate retrieve rules * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: remove conda env from new retrieve rules * fix file names * fix config passing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * style: remove snakemake access in add_heat --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1552) * Change heat_vent config.default setting (PyPSA#1556) * change central_heat_vent in config.default to true, since water tank charger and discharger cant be used as heat vent anymore with branch ltes_stores * change release notes * corrected release notes * fix(benchmark): ignore keyboard interrupt in memory logging subprocess (PyPSA#1555) ensures a C-c is not consumed by the logging child process, but instead properly terminates the solver or main python process. * feat(ci): Remove defaults channel from conda setup and use strict channel prio (PyPSA#1554) * feat(ci): Remove defaults channel from conda setup and use strict channel prio With the default settings the generated pinned environments may still contain references to the defaults channel in certain cases (currently to zstandard), and the test workflow does not fail because it also falls back to the defaults channel. * Use miniforge version Avoids dependencies on proprietary anaconda pkgs channels * Update pinned environment files for all platforms (PyPSA#1553) Co-authored-by: coroa <2552981+coroa@users.noreply.github.com> * Updated eia hydro generation and capacity csv file to include data up to 2023 (PyPSA#1543) Updated the eia hydro generation and capacity up to 2023 Co-authored-by: Lukas Trippe <lkstrp@pm.me> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * extend pre-built cutout years available (PyPSA#1559) * fix(solve): Fix planning_horizons in solve_network (PyPSA#1560) * fix: the planning_horizons params in CCL need to be string * docs: Update the docstring for planning_horizons. * fix: update planning_horizons parameter to use config_provider in perfect foresight * fix(solve): Consolidate use of planning horizons In perfect foresight, the planning_horizons param remains the list of all planning horizons (to trigger re-runs). The current planning horizons is always taken from the planning_horizons wildcard and passed into extra_functionality as a partial kwarg. * Add a specific not implemented AssertionError for CCL in perfect foresight * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: yechenyan <ccyechenyan@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1564) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.7 → v0.9.9](astral-sh/ruff-pre-commit@v0.9.7...v0.9.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor: Rename "fixed" to "capital_cost" for annualised investment … (PyPSA#1566) * refactor: Rename "fixed" to "capital_cost" for annualised investment costs in sector-coupled networks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Follow-up to PyPSA#1532 (PyPSA#1565) * overwrite cost table attributes specified in config in prepare_costs and load_costs (#23) * overwrite cost table attributes specified in config in prepare_costs and load_costs * add custom fuel prices from EEE project --------- Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org> (cherry picked from commit 926f763) * Add logging to existing code * Document the changes * Fix issues with configuration * Add example in the default configuration file * Add release note * Remove fuel cost suggestion * feat(costs): Add support for overwriting investment, lifetime, FOM, VOM, efficiency, and fuel attributes in configuration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor(config): simplify overwrites structure in default configuration * refactor(costs): streamline overwriting of marginal and capital costs in configuration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(prepare_sector_network): iterate over overwrites dictionary items for clarity * fix(prepare_costs): ensure overwrites are applied only to existing costs --------- Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com> Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Allow heat-venting in all heating systems and impose small penalty for water tank chargers (PyPSA#1563) * feat: add marginal cost for water tank chargers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs: update add_retrofit_gas_boiler_constraint docs * docs: update return type annotation for get_heat_system_type method * feat: add heat vents for decentral and rural heating * feat: remove unnecessary central_heat_vent config * feat: pass heat vent marginal cost to prepare_sector_network * refactor: rename marginal_cost_storage to marginal_cost_home_battery_storage * docs: update configtables * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs: update release notes * feat: use :config:cost:marginal_cost where possible * Apply suggestions from code review --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1569) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.9 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.9...v0.9.10) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * enh: enable running mock_snakemake from a separate workdir (PyPSA#1571) * enh: enable running mock_snakemake from a separate workdir Working directories are a simple mechanism of snakemake to have multiple not competing runs of the same workflow. Data, config and logfiles are expected relative to the directory specified with `--directory`, while all the source files (mainly script snakemake and environment definitions are resolved relative from the current snakefile). A new working directory can be quickly set up: ```shell mkdir workdir cp -lr -t workdir --parents cutouts $(git ls-files data) cp -r config workdir ``` Running it then works in the repo root like ```shell snakemake --directory workdir ... ``` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Jonas Hoersch <j.hoersch.extern@transnetbw.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Cluster network by shapes (NUTS0, NUTS1, NUTS2, NUTS3, and ADM1) (PyPSA#1502) * Updated shapes to high-res 1M NUTS2024 shapes and geoboundaries for non-NUTS countries (UA, MD, BA). * Updated nuts to 2021 and non-nuts to gadm. * Updated non NUTS country adm 1 boundaries to OSM via Overpass (MD, BA, UA, XK * Minor order update in rule. * Updated eurostat population data * Updated dataset to JRC ARDECO 2021, including UK, RS, CH * Finished new implementation of build_shapes. * Small fix. * Running sector-coupling workflow * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated build_shapes * Added release nots and updated data-bundle docs. * Added sandbox databundle for temporary CI testing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix in build_shapes: Build only for selected countries in config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed jrc-data from repo, use REST API instead. removed params from build_osm_boundaries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-added shape/polygon simplification. * Made build_osm_boundaries more robust, if wrong country components are retrieved. Reduced nuts shapes to 03m resolution. * Updated build_shapes with simplifying europe function to remove remote islands. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated back to 01m resolution. takes longer. * Removed test_build_shapes from testing, as outdated and not compatible with shapes PR * Readded test_build_shapes.py, however reduced testing, as 'countries()' does not exist anymore, replaced by create_regions(). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for major Danish islands and Kopenhagen. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Initial working commit. * Initial working commit. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Finalised PR. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reversed default mock_snakemake params. * Reversed prepare_network.py mock_snakemake * Updated workflow: Voronoi cell generation within admin shapes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Code clean in cluster_network.py * Reverted mock snakemake to default. * Fixed deprecated typing. * Implemented robust merging algorith for clustering substation-less regions. * Finalised: Updating x,y coordinates to internal point; Always aggregating London area. * Bug fixes. * Reversed mock_snakemake. * Added in simplify_network: Stub-removal only within admin regions. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix merge issue * Apply suggestions * Apply pre-commit * Update default value in existing_capacities * fix: fix type for existing_capacities --------- Co-authored-by: Lukas Trippe <lkstrp@pm.me> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: Jonas Hörsch <jonas.hoersch@openenergytransition.org> Co-authored-by: Philipp Glaum <95913147+p-glaum@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> Co-authored-by: Bobby Xiong <36541459+bobbyxng@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: FabianHofmann <19226431+FabianHofmann@users.noreply.github.com> Co-authored-by: Amos Schledorn <60692940+amos-schledorn@users.noreply.github.com> Co-authored-by: TomKae00 <144129436+TomKae00@users.noreply.github.com> Co-authored-by: Jonas Hörsch <coroa@posteo.de> Co-authored-by: coroa <2552981+coroa@users.noreply.github.com> Co-authored-by: Lukas <97829088+LTBK23@users.noreply.github.com> Co-authored-by: yechenyan <ccyechenyan@gmail.com> Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com> Co-authored-by: Jonas Hoersch <j.hoersch.extern@transnetbw.de>

* Revert "Transition from stores to storage units for LTES, introducing energy-…" (PyPSA#1520) This reverts commit 0e6a8f8. * use pypi `pycipopt` (PyPSA#1524) * Update pinned environment files for all platforms (PyPSA#1523) Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> * fix: ensure correct dtype of "reverse" column (PyPSA#1525) * fix: ensure correct dtype of "reverse" column * apply boolean dtype to reverse column in perfect-foresight * reinsert objective check * fix: improve selection of bidirectionals * fix: relax condition on sanitizing "reversed" column * revert some unneeded changes * debug: print dataframes in summary (to be reverted) * filter empty dfs in make summary * use highs-simplex instead of highs-default for ci --------- Co-authored-by: Lukas Trippe <lkstrp@pm.me> * fix(add_electricity): Fix nuclear p_max_pu time-series for greenfield nuclear (PyPSA#1519) If nuclear is only part of extendable carriers, but not of convential carriers, then the typical "conventional: nuclear: p_max_pu" setting writes the file name all over the p_max_pu time-series, breaking the add_electricity rule. Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * solve_network: add customisable logging frequency (PyPSA#1521) Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Configurable cutout directory (PyPSA#1515) * make cutout dir configurable * feat: add global cutout_directory to config * add release notes * feat: add description to configtables --------- Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * remove `highspy` pin (PyPSA#1528) * remove `highspy` pin * Update environment.yaml * Update environment.yaml --------- Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * Update pinned environment files for all platforms (PyPSA#1533) Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> * Fixing snakemake shadow directory bug that occures with Windows/SMB drives (PyPSA#1535) * Fixing snakemake shadow directory bug that occures with Windows/SMB/network drives * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1526) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.3 → v0.9.4](astral-sh/ruff-pre-commit@v0.9.3...v0.9.4) - [github.com/codespell-project/codespell: v2.4.0 → v2.4.1](codespell-project/codespell@v2.4.0...v2.4.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * Update validate.yaml (PyPSA#1536) * Update validate.yaml * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update validate.yaml --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Consistent function scope (PyPSA#1537) * start removing snakemake from functions * move snakemake objects out of prepare_sector_functions * continue with add_existing_baseyear * continue with prepare_perfect_foresight * apply it to add_electricity * continue with solve network * add pylint check to reusable functions * iron out follow up issues * adjust pylint check * Update pinned environment files for all platforms (PyPSA#1538) Co-authored-by: FabianHofmann <19226431+FabianHofmann@users.noreply.github.com> * Increase default mem_mb for config.default scope (PyPSA#1541) * Add a note to avoid updating pins without testing (PyPSA#1545) * [pre-commit.ci] pre-commit autoupdate (PyPSA#1539) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.6](astral-sh/ruff-pre-commit@v0.9.4...v0.9.6) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Update pinned environment files for all platforms (PyPSA#1544) Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> * Fix geothermal heat potentials (PyPSA#1516) * feat: refactor build_heat_source_potentials into build_geothermal_heat_potential and and use ISI Excel output as data source * feat: remove unnecessary snapshot index * chore: refactor accordingly * feat: fix scaling factor * feat: remove obsolete retrieve * docs: update docs * docs: update release notes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: use retrieve separate retrieve rules * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: remove conda env from new retrieve rules * fix file names * fix config passing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * style: remove snakemake access in add_heat --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1552) * Change heat_vent config.default setting (PyPSA#1556) * change central_heat_vent in config.default to true, since water tank charger and discharger cant be used as heat vent anymore with branch ltes_stores * change release notes * corrected release notes * fix(benchmark): ignore keyboard interrupt in memory logging subprocess (PyPSA#1555) ensures a C-c is not consumed by the logging child process, but instead properly terminates the solver or main python process. * feat(ci): Remove defaults channel from conda setup and use strict channel prio (PyPSA#1554) * feat(ci): Remove defaults channel from conda setup and use strict channel prio With the default settings the generated pinned environments may still contain references to the defaults channel in certain cases (currently to zstandard), and the test workflow does not fail because it also falls back to the defaults channel. * Use miniforge version Avoids dependencies on proprietary anaconda pkgs channels * Update pinned environment files for all platforms (PyPSA#1553) Co-authored-by: coroa <2552981+coroa@users.noreply.github.com> * Updated eia hydro generation and capacity csv file to include data up to 2023 (PyPSA#1543) Updated the eia hydro generation and capacity up to 2023 Co-authored-by: Lukas Trippe <lkstrp@pm.me> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * extend pre-built cutout years available (PyPSA#1559) * fix(solve): Fix planning_horizons in solve_network (PyPSA#1560) * fix: the planning_horizons params in CCL need to be string * docs: Update the docstring for planning_horizons. * fix: update planning_horizons parameter to use config_provider in perfect foresight * fix(solve): Consolidate use of planning horizons In perfect foresight, the planning_horizons param remains the list of all planning horizons (to trigger re-runs). The current planning horizons is always taken from the planning_horizons wildcard and passed into extra_functionality as a partial kwarg. * Add a specific not implemented AssertionError for CCL in perfect foresight * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: yechenyan <ccyechenyan@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1564) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.7 → v0.9.9](astral-sh/ruff-pre-commit@v0.9.7...v0.9.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor: Rename "fixed" to "capital_cost" for annualised investment … (PyPSA#1566) * refactor: Rename "fixed" to "capital_cost" for annualised investment costs in sector-coupled networks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Follow-up to PyPSA#1532 (PyPSA#1565) * overwrite cost table attributes specified in config in prepare_costs and load_costs (#23) * overwrite cost table attributes specified in config in prepare_costs and load_costs * add custom fuel prices from EEE project --------- Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org> (cherry picked from commit 926f763) * Add logging to existing code * Document the changes * Fix issues with configuration * Add example in the default configuration file * Add release note * Remove fuel cost suggestion * feat(costs): Add support for overwriting investment, lifetime, FOM, VOM, efficiency, and fuel attributes in configuration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor(config): simplify overwrites structure in default configuration * refactor(costs): streamline overwriting of marginal and capital costs in configuration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(prepare_sector_network): iterate over overwrites dictionary items for clarity * fix(prepare_costs): ensure overwrites are applied only to existing costs --------- Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com> Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Allow heat-venting in all heating systems and impose small penalty for water tank chargers (PyPSA#1563) * feat: add marginal cost for water tank chargers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs: update add_retrofit_gas_boiler_constraint docs * docs: update return type annotation for get_heat_system_type method * feat: add heat vents for decentral and rural heating * feat: remove unnecessary central_heat_vent config * feat: pass heat vent marginal cost to prepare_sector_network * refactor: rename marginal_cost_storage to marginal_cost_home_battery_storage * docs: update configtables * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs: update release notes * feat: use :config:cost:marginal_cost where possible * Apply suggestions from code review --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1569) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.9 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.9...v0.9.10) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * enh: enable running mock_snakemake from a separate workdir (PyPSA#1571) * enh: enable running mock_snakemake from a separate workdir Working directories are a simple mechanism of snakemake to have multiple not competing runs of the same workflow. Data, config and logfiles are expected relative to the directory specified with `--directory`, while all the source files (mainly script snakemake and environment definitions are resolved relative from the current snakefile). A new working directory can be quickly set up: ```shell mkdir workdir cp -lr -t workdir --parents cutouts $(git ls-files data) cp -r config workdir ``` Running it then works in the repo root like ```shell snakemake --directory workdir ... ``` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Jonas Hoersch <j.hoersch.extern@transnetbw.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Cluster network by shapes (NUTS0, NUTS1, NUTS2, NUTS3, and ADM1) (PyPSA#1502) * Updated shapes to high-res 1M NUTS2024 shapes and geoboundaries for non-NUTS countries (UA, MD, BA). * Updated nuts to 2021 and non-nuts to gadm. * Updated non NUTS country adm 1 boundaries to OSM via Overpass (MD, BA, UA, XK * Minor order update in rule. * Updated eurostat population data * Updated dataset to JRC ARDECO 2021, including UK, RS, CH * Finished new implementation of build_shapes. * Small fix. * Running sector-coupling workflow * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated build_shapes * Added release nots and updated data-bundle docs. * Added sandbox databundle for temporary CI testing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix in build_shapes: Build only for selected countries in config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed jrc-data from repo, use REST API instead. removed params from build_osm_boundaries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-added shape/polygon simplification. * Made build_osm_boundaries more robust, if wrong country components are retrieved. Reduced nuts shapes to 03m resolution. * Updated build_shapes with simplifying europe function to remove remote islands. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated back to 01m resolution. takes longer. * Removed test_build_shapes from testing, as outdated and not compatible with shapes PR * Readded test_build_shapes.py, however reduced testing, as 'countries()' does not exist anymore, replaced by create_regions(). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for major Danish islands and Kopenhagen. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Initial working commit. * Initial working commit. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Finalised PR. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reversed default mock_snakemake params. * Reversed prepare_network.py mock_snakemake * Updated workflow: Voronoi cell generation within admin shapes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Code clean in cluster_network.py * Reverted mock snakemake to default. * Fixed deprecated typing. * Implemented robust merging algorith for clustering substation-less regions. * Finalised: Updating x,y coordinates to internal point; Always aggregating London area. * Bug fixes. * Reversed mock_snakemake. * Added in simplify_network: Stub-removal only within admin regions. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update common.smk (PyPSA#1577) Wrong variable description in has_internet_access (mentioned Google and should be Zenodo) * Windows snakefile (PyPSA#1575) * Bugfix wdpa downloads on windows Bugfix: - add special handling to the rules download_wdpa and download_wdpa_marine in retrieve.smk, since Linux shell commands don't work on Windows. * Remove unused lines of code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Switch wdpa downloads to shutil Minor changes: - change the download wdpa rules in rules/retrieve.smk to shutil to properly work on Windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * clustering with NUTS3 in Spain breaks (PyPSA#1580) * Update base_network.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor bugfix in contains columns: Only unique IDs * Dropped redundant else: ... --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: bobbyxng <bobbyxng@gmail.com> Co-authored-by: Bobby Xiong <36541459+bobbyxng@users.noreply.github.com> * match CH and any number of digits (PyPSA#1585) * Add PTES and introduce PTES/TES energy-to-power ratios (PyPSA#1546) * introduce etpr for TES using stores * add constraints * copleted implementation of constraints for TES stores * updated release notes * adjusted mock snakemake default * library change * update release notes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Delete .DS_Store * change retrieval of etpr values in etpr_constraint * changed TES constraints check based on the configs * constraint checks * enhance error handling for the tes constraints * small changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * changed mock snakemake back to default settings * feat: add marginal cost for water tank chargers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs: update add_retrofit_gas_boiler_constraint docs * docs: update return type annotation for get_heat_system_type method * feat: add heat vents for decentral and rural heating * feat: remove unnecessary central_heat_vent config * feat: pass heat vent marginal cost to prepare_sector_network * refactor: rename marginal_cost_storage to marginal_cost_home_battery_storage * docs: update configtables * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs: update release notes * add suggestions to ensure that charger and discharger align for charger ratio constraints and energy power constraints * add lifetime to PTES and TTES charger/discharger, so that they are built with correct baseyear * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: add small penalty on ptes charging * feat: update fixed cost to capital cost str * feat: remove unnecessary passing of obsolete config * feat: remove unneeded config passing * feat: remove unneeded config passing * feat: remove unneeded config passing * Update scripts/solve_network.py * Update solve_network.py * feat: re-add TES necessary parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor string contains checks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Amos Schledorn <60692940+amos-schledorn@users.noreply.github.com> Co-authored-by: Amos Schledorn <a.schledorn@tu-berlin.de> * Update pinned environment files for all platforms (PyPSA#1588) Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (PyPSA#1592) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.10 → v0.11.0](astral-sh/ruff-pre-commit@v0.9.10...v0.11.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Failing on 'warning' solving status (PyPSA#1591) * Reduce pipeline entrypoints to country coverage / exclude RU+BY (PyPSA#1582) * handle pipeline entrypoints for reduced country coverage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --------- Co-authored-by: Lukas Trippe <lkstrp@pm.me> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: Jonas Hörsch <jonas.hoersch@openenergytransition.org> Co-authored-by: Philipp Glaum <95913147+p-glaum@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> Co-authored-by: Bobby Xiong <36541459+bobbyxng@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: FabianHofmann <19226431+FabianHofmann@users.noreply.github.com> Co-authored-by: Amos Schledorn <60692940+amos-schledorn@users.noreply.github.com> Co-authored-by: TomKae00 <144129436+TomKae00@users.noreply.github.com> Co-authored-by: Jonas Hörsch <coroa@posteo.de> Co-authored-by: coroa <2552981+coroa@users.noreply.github.com> Co-authored-by: Lukas <97829088+LTBK23@users.noreply.github.com> Co-authored-by: yechenyan <ccyechenyan@gmail.com> Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com> Co-authored-by: Jonas Hoersch <j.hoersch.extern@transnetbw.de> Co-authored-by: Markus Groissböck <markus.groissbock@openenergytransition.org> Co-authored-by: Eric Nitschke <eric.nitschke@tuhh.de> Co-authored-by: Cristóbal José Gallego Castillo <83344529+cristobal-GC@users.noreply.github.com> Co-authored-by: bobbyxng <bobbyxng@gmail.com> Co-authored-by: pworschischek-aggmag <163411786+pworschischek-aggmag@users.noreply.github.com> Co-authored-by: Amos Schledorn <a.schledorn@tu-berlin.de>

cristobal-GC · 2026-01-12T15:00:11Z

Hi,
Sorry if this is obvious, or if I missed something, but what happened to the "atlite:cutout_directory:" field in the config file? I can't find it, and I don't know when it was deleted.
Thanks

Snakefile

coroa · 2026-01-13T12:43:19Z

Hi, Sorry if this is obvious, or if I missed something, but what happened to the "atlite:cutout_directory:" field in the config file? I can't find it, and I don't know when it was deleted. Thanks

Please open an issue and add your use case there. This PR has been merged about a year ago, not a good place for any discussion about it.

lkstrp · 2026-01-13T12:47:31Z

@cristobal-GC It was deleted in here #1675

cristobal-GC · 2026-01-15T10:31:06Z

Thanks, guys. I see that removing the cutout directory was part of a larger process. I don't think there's a need to open a new issue. I'll get familiar with the new approach to handling data versions. I just found it very useful to have all the cutouts in an external folder used by different PyPSA-EUR and PyPSA-Spain clones (that's why I recently missed this folder). Thanks again for the answers.

lkstrp · 2026-01-15T12:36:12Z

You're right, but you can still do this by using symlinks on the cutouts dir!

cristobal-GC · 2026-01-16T13:04:28Z

True! thank you @lkstrp !

FabianHofmann requested a review from fneum January 26, 2025 14:57

FabianHofmann added 2 commits January 26, 2025 15:57

make cutout dir configurable

31a8074

feat: add global cutout_directory to config

c9cc079

add release notes

c99d828

FabianHofmann force-pushed the cutout-dir branch from 03c5f3c to c99d828 Compare January 26, 2025 15:01

feat: add description to configtables

c92487d

fneum approved these changes Feb 6, 2025

View reviewed changes

Merge branch 'master' into cutout-dir

e776e6e

FabianHofmann merged commit be7b9c6 into master Feb 6, 2025
14 checks passed

FabianHofmann deleted the cutout-dir branch February 6, 2025 09:34

Eric-Nitschke mentioned this pull request Mar 20, 2025

Snakemake can't find retrieve_cutouts #1598

Closed

2 tasks

coroa reviewed Jan 13, 2026

View reviewed changes

Snakefile Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configurable cutout directory#1515

Configurable cutout directory#1515
FabianHofmann merged 5 commits intomasterfrom
cutout-dir

FabianHofmann commented Jan 26, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jan 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

cristobal-GC commented Jan 12, 2026

Uh oh!

Uh oh!

coroa commented Jan 13, 2026

Uh oh!

lkstrp commented Jan 13, 2026

Uh oh!

cristobal-GC commented Jan 15, 2026

Uh oh!

lkstrp commented Jan 15, 2026

Uh oh!

cristobal-GC commented Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

FabianHofmann commented Jan 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

github-actions bot commented Jan 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Validator Report

Uh oh!

Uh oh!

cristobal-GC commented Jan 12, 2026

Uh oh!

Uh oh!

coroa commented Jan 13, 2026

Uh oh!

lkstrp commented Jan 13, 2026

Uh oh!

cristobal-GC commented Jan 15, 2026

Uh oh!

lkstrp commented Jan 15, 2026

Uh oh!

cristobal-GC commented Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

FabianHofmann commented Jan 26, 2025 •

edited

Loading

github-actions bot commented Jan 26, 2025 •

edited

Loading