PyPSA-Earth-OSM: enabling synergies between OSM enhansment and energy modelling.

by

Development Status: In progress

That is an experimental modeling implementation intended to be used for bridging OSM mapping work with enhansement of the energy modelling workflow.

To run the stable modelling workflow for any country of the workd, please refer to PyPSA-Earth repository.

Repository Structure

PyPSA-Earth-OSM uses a multi-branch development strategy to enable parallel work on general features and country-specific modeling:

Branch Overview

• development (Main development branch): Contains general features, enhancements, and infrastructure improvements for the Earth-OSM workflow. All new features and bug fixes should be developed here first.

• columbia-modeling: Dedicated branch for Colombia-specific energy system modeling. Contains configurations, data, and analysis specific to Colombia's power grid.

• bolivia-modeling: Dedicated branch for Bolivia-specific energy system modeling. Contains configurations, data, and analysis specific to Bolivia's power grid.

Contributing Guidelines

• For general features: Create feature branches from development and submit pull requests to development.
• For country-specific modeling: Work on the respective country branch (columbia-modeling or bolivia-modeling).
• Keeping branches updated: Country-specific branches should periodically merge updates from development to incorporate new features and fixes.

This structure allows researchers to focus on specific country analyses while maintaining a common codebase for shared functionality.

How Earth-OSM Works

PyPSA-Earth-OSM leverages OpenStreetMap (OSM) data to build power grid topology models for energy system analysis. The workflow extracts, processes, and validates power infrastructure data from OSM, enabling researchers to analyze electricity networks using crowd-sourced geographic information.

Data Sources and Workflow

The Earth-OSM workflow supports three different approaches for obtaining OpenStreetMap data:

1. Latest OSM Data (Default)

Downloads the most current version of OSM data directly from OpenStreetMap servers. This ensures access to the latest updates from the OSM community, including recent infrastructure additions and corrections.

2. Historical OSM Data

Enables analysis of past grid configurations by downloading OSM data from a specific date. This is valuable for:

• Temporal analysis and grid evolution studies
• Validation against historical records
• Reproducibility of previous research
• Comparison of infrastructure development over time

Historical data is available from approximately 2012 onwards, depending on OSM data availability.

3. Custom OSM Data

Allows users to provide their own OpenStreetMap data files (in .pbf format). This option is useful for:

• Working with locally cached OSM data
• Using pre-processed or validated OSM datasets
• Offline workflows or environments with limited internet access
• Testing with specific OSM data snapshots

Configuration Options

The osm_data section in the configuration file (config.default.yaml or config.yaml) controls how Earth-OSM retrieves and processes OpenStreetMap data:

osm_data:
  source: "latest"  # Options: "latest", "historical", or "custom"

  # For historical data
  target_date: "2020-01-01"  # Format: YYYY-MM-DD

  # For custom data
  custom_path:
    pbf: "data/custom/osm/pbf"      # Path to custom .pbf files (required)
    power: "data/custom/osm/power"  # Path to processed power files (optional)

Key Configuration Parameters:

• source: Determines the OSM data source

  • "latest": Downloads current OSM data (default)
  • "historical": Downloads OSM data from a specific date
  • "custom": Uses user-provided OSM files

• target_date: Specifies the date for historical data (format: YYYY-MM-DD)

  • Only used when source: "historical"
  • Example dates: "2018-01-01", "2020-06-15", "2022-12-31"

• custom_path: Defines paths for custom OSM data files

  • pbf: Directory containing raw OSM data files (.osm.pbf format)
    • Files must follow naming pattern: <country>-latest.osm.pbf
    • Example: bolivia-latest.osm.pbf, nigeria-latest.osm.pbf
  • power: Directory for pre-processed power infrastructure JSON files (optional)

Data Processing Pipeline

Once OSM data is obtained, Earth-OSM processes it through several stages:

1. Download/Load: Retrieves OSM data based on the configured source
2. Filter: Extracts power infrastructure elements (substations, lines, cables, generators)
3. Clean: Validates and standardizes the data using configurable thresholds and rules
4. Build Network: Constructs the power network topology, connecting buses and lines
5. Validate: Performs quality checks and consistency validation

Collaborators

FabianHofmann	fneum	ekatef	euronion	Justus-coded	mnm-matin
GbotemiB	martacki	LukasFrankenQ	pz-max	davide-f	koen-vg
Eddy-JV	hazemakhalek	energyLS	AnasAlgarei	yerbol-akhmetov	doneachh
danielelerede-oet	DeniseGiub	finozzifa	virio-andreyana	Tomkourou	Eric-Nitschke
GridGrapher	drifter089	glenkiely-ieg	ljansen-iee	Emre-Yorat89	giacfalk
Ekaterina-Vo	lkstrp	TosinGeorge	Ly0n	Tooblippe	cpschau
arizeosalac	SermishaNarayana	AlexanderMeisinger	asolavi	rajesh-ieg	LucieRC
choiHenry	gianvicolux	carlosfv92	saikumarvasa100-hash	rsparks3	ollie-bell
juli-a-ko	squoilin	siddharth-krishna	pitmonticone	PierreKara1	Netotse
milyas009	kma33	jome1	jessLryan	jarry7	huyhoang-mike
HanaElattar	Vamsipriya22	flacombe	EmreYorat	AndreCNF	AlessandroPampado99