Solar Consumer

This code can be used to download solar forecasts and save them to a PostgreSQL database. It fetches solar generation estimates for embedded solar farms and processes the data for analysis. We currently collect

• UK: Forecast can be retreived from NESO. Generation Data can be retrevied from PVLive.
• NL: Generation values from Ned NL, both national and region. National Forecast values from Ned NL too.
• DE: Generation values from ENTSOE for several TSOs.
• BE: Solar PV forecast data (national and regional) from Elia Open Data API.
• India (Rajasthan): Real-time solar and wind generation data from RUVNL (Rajasthan Urja Vikas Nigam Limited).

Here are the different sources of data, and which methods can be used to save the results

Source	Country	CSV	Data Platform	DB (Legacy)	Site DB (Legacy)
PVLive	🇬🇧	✅	✅
NESO forecast	🇬🇧	✅		✅
Ned-nl	🇳🇱	✅			✅
Ned-nl forecast	🇳🇱	✅			✅
Germany (ENTSOE)	🇩🇪	✅			✅
Elia Open Data	🇧🇪	✅
RUVNL (Rajasthan SLDC)	🇮🇳	✅

Requirements

• Docker
• Docker Compose

Installation & Running

1. Clone the repository:

git clone https://github.com/openclimatefix/neso-solar-consumer.git
cd neso-solar-consumer

2. Copy the example environment file:

cp .example.env .env

3. Start the application:

docker compose up -d

The above command will:

• Start a PostgreSQL database container
• Build and start the NESO Solar Consumer application
• Configure all necessary networking between containers

To stop the application:

docker compose down

To view logs:

docker compose logs -f

Note: The PostgreSQL data is persisted in a Docker volume. To completely reset the database, use:

docker compose down -v

Documentation

The package provides three main functionalities:

1. Data Fetching: Retrieves solar forecast data from the NESO API
2. Data Formatting: Processes the data into standardized forecast objects
3. Data Storage: Saves the formatted forecasts to a PostgreSQL database

Key Components:

• fetch_data.py: Handles API data retrieval
• format_forecast.py: Converts raw data into forecast objects
• save_forecast.py: Manages database operations
• app.py: Orchestrates the entire pipeline

Environment Variables: (Can be found in the .example.env / .env file)

• DB_URL=postgresql://postgres:postgres@localhost:5432/neso_solar : Database Configuration
• COUNTRY="gb" : Country code for fetching data. Currently, other options are ["be", "ind_rajasthan", "nl"]
• SAVE_METHOD="db": Ways to store the data. Currently other options are ["csv", "site-db"]
• CSV_DIR=None : Directory to save CSV files if SAVE_METHOD="csv".
• UK_PVLIVE_REGIME=in-day: For UK PVLive, the regime. Can be "in-day" or "day-after"
• UK_PVLIVE_N_GSPS=342: For UK PVLive, the amount of gsps we pull data for.
• UK_PVLIVE_BACKFILL_HOURS=2: For UK PVLive, the amount of backfill hours we pull, when regime="in-day"

Adding a New Country

This guide explains how to add a new country data source to Solar Consumer.

Overview

Adding a country typically involves:

• Identifying a reliable data source or API
• Implementing a country-specific fetch function
• Adding tests
• Saving data locally (CSV) and/or to the data platform

---

Step 1: Find a Data Source / API

Identify a reliable data source for the country:

• Prefer official grid operators or government-backed APIs
• Ensure timestamps, units, and generation values are clearly defined

If the API requires credentials:

• Add the variable to .example.env
• Document the required environment variable name

---

Step 2: Create a Fetch Function

Add a new country-specific fetch module inside the solar_consumer package.

Example naming convention:

solar_consumer/data/fetch_<country>.py

Next Steps

After adding the fetch function:

• Register the country in the main fetch dispatcher
• Add unit and integration tests under tests/
• Verify the data runs locally and can be saved to CSV
• If supported, ensure data can be saved to the data platform
• Open a pull request for review

Saving Data to the Data Platform

If the country supports saving data to the data platform:

1. Clone the data platform repository:

   git clone https://github.com/openclimatefix/data-platform.git

2. Follow the data-platform README to start it locally (Docker-based setup).
3. Configure Solar Consumer to point to the local data-platform instance.
4. Run the consumer and verify data is ingested successfully.

Development

1. Set up the development environment:

pip install ".[dev]"

2. Run tests:

pytest

3. Format code:

black .

4. Run linter:

ruff check .

Running the Test Suite

The test suite includes unit tests and integration tests:

# Run all tests
pytest

# Run specific test file
pytest tests/test_fetch_data.py

# Run with coverage
pytest --cov=neso_solar_consumer

Continuous Integration (CI)

This reposistory has 2 main CI workflows - branch-ci and merged-ci.

• branch-ci is triggered on all pushes to any branch except main, and on any pull request that is opened, reopened or updated. It runs the tests suite, lints the project, and builds and pushes a dev image.
• merged-ci is triggered on any pull request merged into main. It bumps the git tag, and builds and pushes a container with that tag.

FAQ

Q: What format is the data stored in? A: The data is stored in PostgreSQL using SQLAlchemy models, with timestamps in UTC and power values in megawatts.

Q: How often should I run the consumer? A: This depends on your use case and the NESO API update frequency. The consumer can be scheduled using cron jobs or other scheduling tools.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing and community

• PR's are welcome! See the Organisation Profile for details on contributing
• Find out about our other projects in the OCF Meta Repo
• Check out the OCF blog for updates
• Follow OCF on LinkedIn

Contributors

Peter Dudfield 🤔	Seao7 💻	Siddharth 👀 🚇 💻	Conor O Callaghan 📖	Ali Rashid ⚠️	Manzoor Ahmed Shaikh 💻	Anas Khan 📖
Peter Ireland 📖	vashisthrahul13 💻	rahul-ahuja 💻	Rahul Joon 💻	michael-gendy 🚇	Shohail Ismail 💻	Prafful Vyas 💻
Ramkumar R 💻	Mohit 💻	Vishal Goyal 💻

Part of the Open Climate Fix community.