Merge pull request #702 from OpenEnergyPlatform/release-v0.16.1

Release v0.16.1

Author: FlorianK13

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.16.0
+current_version = 0.16.1
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)((?P<release>(a|na))+(?P<build>\d+))?
 serialize = 
 	{major}.{minor}.{patch}{release}{build}

.github/workflows/ci-production.yml

@@ -33,7 +33,7 @@ jobs:
     - name: create package
       run: python -m build --sdist
     - name: import open-mastr
-      run: python -m pip install ./dist/open_mastr-0.16.0.tar.gz
+      run: python -m pip install ./dist/open_mastr-0.16.1.tar.gz
     - name: Create credentials file
       env:
         MASTR_TOKEN: ${{ secrets.MASTR_TOKEN }}

CHANGELOG.md

@@ -6,6 +6,16 @@ For each version important additions, changes and removals are listed here.
 The format is inspired from [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [v0.16.1] Hotfix - 2025-12-17
+
+### Changed
+- Adapt introduction text in README and docs
+  [#647](https://github.com/OpenEnergyPlatform/open-MaStR/pull/647)
+- Hotfix of date selection and incorporate new url structure
+  [#699](https://github.com/OpenEnergyPlatform/open-MaStR/pull/699)
+
+
+
 ## [v0.16.0] Partial downloads with open-MaStR PartialPumpkinPull - 2025-11-26
 ### Added
 - Add partial bulk download

CITATION.cff

@@ -41,7 +41,7 @@ authors:
 title: "open-MaStR"
 type: software
 license: AGPL-3.0
-version: 0.16.0
+version: 0.16.1
 doi: 
-date-released: 2025-11-26
+date-released: 2025-12-17
 url: "https://github.com/OpenEnergyPlatform/open-MaStR/"

README.rst

@@ -37,19 +37,40 @@ open-mastr
 Introduction
 ============
 
-The `Marktstammdatenregister (MaStR) <https://www.marktstammdatenregister.de/MaStR>`_ is a German register 
-provided by the German Federal Network Agency (Bundesnetzagentur / BNetza) that keeps track of all power and gas units located in Germany.
+The python package ``open-mastr`` provides an interface for accessing the `Marktstammdatenregister (MaStR) <https://www.marktstammdatenregister.de/MaStR>`_ data. The MaStR is a German register provided by the German Federal Network Agency (Bundesnetzagentur / BNetza) that keeps track of all power and gas units located in Germany. It is a daily growing dataset with millions of data points covering electricity and gas production units, electricity and gas consumers, storages, grids, and energy market participants.
 
-The MaStR data can be
- 
-#. browsed and filtered `online <https://www.marktstammdatenregister.de/MaStR>`_
-#. taken from `daily provided dumps <https://www.marktstammdatenregister.de/MaStR/Datendownload>`_
-#. be accessed via the `web service <https://www.marktstammdatenregister.de/MaStRHilfe/subpages/webdienst.html>`_
+Generally, and besides the offerings of ``open-mastr``, the MaStR data can be accessed via three main options:
 
-| The python package ``open-mastr`` provides an interface for accessing the data. 
-| It contains methods to download and parse the xml files (bulk) and the SOAP web service (API).
-| In this repository we are developing methods to analyze, validate and enrich the data.
-| We want to collect and compile post processing scripts to improve data quality.
+#. browse, filter and download `in the browser <https://www.marktstammdatenregister.de/MaStR>`_
+#. download `daily provided dumps <https://www.marktstammdatenregister.de/MaStR/Datendownload>`_
+#. access via the `web service <https://www.marktstammdatenregister.de/MaStRHilfe/subpages/webdienst.html>`_
+
+These access options, however, are not exactly frictionless. ``open-mastr`` thus provides an interface for and improved developer experience of accessing the data. This project is intended for individuals who wish to "just work" with the MaStR data and who do not want to deal with the idiosyncrasies of the three access options above.
+
+In particular, ``open-mastr`` facilitates access to the daily provided MaStR dumps with download methods (bulk) and by parsing the XML files to a relational database. Furthermore, the software provides a Python wrapper to access the MaStR SOAP web service (API).
+
+
+**Does open-mastr edit or change the MaStR data?**
+No. ``open-mastr`` is a wrapper around the MaStR data and does not edit or change the data. It is intended to be used as a tool for working with the MaStR data.
+
+Benefits provided by ``open-mastr``
+==================================
+
+.. list-table::
+   :widths: 30, 70
+   :header-rows: 1
+
+   * - Benefit
+     - Description
+   * - Data download and parsing
+     - Download, decode, and write data to a local database
+   * - Translation to English
+     - Translate table names and columns from German to English as well as an English documentation page of the dataset
+   * - Data processing
+     - Merge relevant information about different technologies to single csv files
+
+**Just here for the data?**
+We regularly run the whole download and cleansing pipeline and upload the dataset as csv files at `zenodo <https://doi.org/10.5281/zenodo.6807425>`_!
 
 
 Documentation
@@ -140,7 +161,7 @@ Software
 
 Data
 ----
-| The data has the license **Datenlizenz Deutschland – Namensnennung – Version 2.0** (DL-DE-BY-2.0)
+| The original dataset is licensed under the **Datenlizenz Deutschland – Namensnennung – Version 2.0** (DL-DE-BY-2.0)
 | Copyright: `Marktstammdatenregister <https://www.marktstammdatenregister.de/MaStR>`_ - © Bundesnetzagentur für Elektrizität, Gas, Telekommunikation, Post und Eisenbahnen | `DL-DE-BY-2.0 <https://www.govdata.de/dl-de/by-2-0>`_
 
 

docs/index.md

@@ -1,29 +1,27 @@
 # Introduction
 
-The [Marktstammdatenregister (MaStR)](https://www.marktstammdatenregister.de/MaStR) is a German register 
-provided by the German Federal Network Agency (Bundesnetzagentur / BNetza) that keeps track of all power and gas units located in Germany.
-It is a daily growing dataset with more than 8.2 million data points covering electricity and gas production units, electricity and gas consumers, storages, grids, and energy market participants (as of spring 2024).
+The python package `open-mastr` provides an interface for accessing the [Marktstammdatenregister (MaStR)](https://www.marktstammdatenregister.de/MaStR) data. The MaStR is a German register provided by the German Federal Network Agency (Bundesnetzagentur / BNetza) that keeps track of all power and gas units located in Germany. It is a daily growing dataset with millions of data points covering electricity and gas production units, electricity and gas consumers, storages, grids, and energy market participants.
 
-Generally, the MaStR data can be accessed via various options:
+Generally, and besides the offerings of `open-mastr`, the MaStR data can be accessed via three main options:
 
-  1. browse, filter and download [online](https://www.marktstammdatenregister.de/MaStR)
-  1. download [daily provided dumps](https://www.marktstammdatenregister.de/MaStR/Datendownload)
-  1. access via the [web service](https://www.marktstammdatenregister.de/MaStRHilfe/subpages/webdienst.html)
+  1. browse, filter and download [in the browser](https://www.marktstammdatenregister.de/MaStR)
+  2. download [daily provided dumps](https://www.marktstammdatenregister.de/MaStR/Datendownload)
+  3. access via the [web service](https://www.marktstammdatenregister.de/MaStRHilfe/subpages/webdienst.html)
 
-The python package `open-mastr` provides an interface for accessing the data and contributes to improving the 
-usability of the access options above. This repository is intended for people who wish to simply work with the 
-MaStR data and do not want to deal with the individual obstacles to data access of the three options above.  <br> <br>
-It facilitates access to the daily provided MaStR dumps with download methods (bulk) and by 
-parsing the XML files to a relational database. Furthermore, the software provides a Python wrapper to access the MaStR 
-SOAP web service (API).
+These access options, however, are not exactly frictionless. `open-mastr` thus provides an interface for and improved developer experience of accessing the data. This project is intended for individuals who wish to "just work" with the MaStR data and who do not want to deal with the idiosyncrasies of the three access options above.
+
+In particular, `open-mastr` facilitates access to the daily provided MaStR dumps with download methods (bulk) and by parsing the XML files to a relational database. Furthermore, the software provides a Python wrapper to access the MaStR SOAP web service (API).
+
+!!! info "Does `open-mastr` edit or change the MaStR data?"
+    No. `open-mastr` is a wrapper around the MaStR data and does not edit or change the data. It is intended to be used as a tool for working with the MaStR data.
 
 ## Benefits provided by `open-mastr`
 
-Benefit | Description 
-------- | ------ 
-Data download and parsing | Download, decode, and write data to a local database 
-Translation to English | Translate table names and columns from German to English as well as an English documentation page of the dataset 
-Data processing | Merge relevant information about different technologies to single csv files
+| Benefit                   | Description                                                                                                      |
+| ------------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| Data download and parsing | Download, decode, and write data to a local database                                                             |
+| Translation to English    | Translate table names and columns from German to English as well as an English documentation page of the dataset |
+| Data processing           | Merge relevant information about different technologies to single csv files                                      |
 
 !!! question "Just here for the data?"
     :sparkles: We regularly run the whole download and cleansing pipeline and upload the dataset as csv files at [zenodo](https://doi.org/10.5281/zenodo.6807425)! 

open_mastr/mastr.py

@@ -247,7 +247,9 @@ def download(
                     xml_folder_path,
                 )
 
-            download_xml_Mastr(zipped_xml_file_path, date, data, xml_folder_path)
+            download_xml_Mastr(
+                zipped_xml_file_path, bulk_download_date, data, xml_folder_path
+            )
 
             log.info(
                 "\nWould you like to speed up the creation of your MaStR database?\n"

open_mastr/xml_download/utils_cleansing_bulk.py

@@ -5,6 +5,7 @@
     columns_replace_list,
 )
 from zipfile import ZipFile
+import io
 
 
 def cleanse_bulk_data(df: pd.DataFrame, zipped_xml_file_path: str) -> pd.DataFrame:
@@ -59,8 +60,11 @@ def create_katalogwerte_from_bulk_download(zipped_xml_file_path) -> dict:
     """Creates a dictionary from the id -> value mapping defined in the table
     katalogwerte from MaStR."""
     with ZipFile(zipped_xml_file_path, "r") as f:
-        data = f.read("Katalogwerte.xml")
-        df_katalogwerte = pd.read_xml(data, encoding="UTF-16", compression="zip")
+        with f.open("Katalogwerte.xml") as xml_bytes_io:
+            xml_bytes = xml_bytes_io.read()
+        xml_text = xml_bytes.decode("utf-16")
+        xml_io = io.StringIO(xml_text)
+        df_katalogwerte = pd.read_xml(xml_io)
     katalogwerte_array = np.array(df_katalogwerte[["Id", "Wert"]])
     katalogwerte = dict(
         (katalogwerte_array[n][0], katalogwerte_array[n][1])

open_mastr/xml_download/utils_download_bulk.py

@@ -1,6 +1,7 @@
 import os
 import shutil
 import time
+from datetime import datetime as dt
 from importlib.metadata import PackageNotFoundError, version
 from zipfile import ZipFile
 from pathlib import Path
@@ -84,7 +85,9 @@ def gen_version(
     return f"{year}.{release}"
 
 
-def gen_url(when: time.struct_time = time.localtime(), use_version="current") -> str:
+def gen_url(
+    when: time.struct_time = time.localtime(), use_version="current", use_stichtag=False
+) -> str:
     """Generates the download URL for the specified date.
 
     Note that not all dates are archived on the website.
@@ -112,7 +115,11 @@ def gen_url(when: time.struct_time = time.localtime(), use_version="current") ->
     version = gen_version(when, use_version)
     date = time.strftime("%Y%m%d", when)
 
-    return f"https://download.marktstammdatenregister.de/Gesamtdatenexport_{date}_{version}.zip"
+    if use_stichtag:
+        url_str = f"https://download.marktstammdatenregister.de/Stichtag/Gesamtdatenexport_{date}_{version}.zip"
+    else:
+        url_str = f"https://download.marktstammdatenregister.de/Gesamtdatenexport_{date}_{version}.zip"
+    return url_str
 
 
 def download_xml_Mastr(
@@ -134,9 +141,8 @@ def download_xml_Mastr(
 
     log.info("Starting the Download from marktstammdatenregister.de.")
 
-    # TODO this should take bulk_date_string
-    now = time.localtime()
-    url = gen_url(now)
+    url_time = dt.strptime(bulk_date_string, "%Y%m%d").date().timetuple()
+    url = gen_url(url_time)
 
     time_a = time.perf_counter()
     r = requests.get(url, stream=True, headers={"User-Agent": USER_AGENT})
@@ -145,23 +151,31 @@ def download_xml_Mastr(
             "Download file was not found. Assuming that the new file was not published yet and retrying with yesterday."
         )
         now = time.localtime(
-            time.mktime(now) - (24 * 60 * 60)
+            time.mktime(url_time) - (24 * 60 * 60)
         )  # subtract 1 day from the date
         url = gen_url(now)
         r = requests.get(url, stream=True, headers={"User-Agent": USER_AGENT})
     if r.status_code == 404:
-        url = gen_url(now, use_version="before")  # Use lower MaStR Version
+        url = gen_url(url_time, use_version="before")  # Use lower MaStR Version
         log.warning(
             f"Download file was not found. Assuming that the version of MaStR has changed and retrying with download link: {url}"
         )
         r = requests.get(url, stream=True, headers={"User-Agent": USER_AGENT})
     if r.status_code == 404:
-        url = gen_url(now, use_version="after")  # Use higher MaStR Version
+        url = gen_url(url_time, use_version="after")  # Use higher MaStR Version
         log.warning(
             f"Download file was not found. Assuming that the version of MaStR has changed and retrying with download link: {url}"
         )
         r = requests.get(url, stream=True, headers={"User-Agent": USER_AGENT})
 
+    if r.status_code == 404:
+        url = gen_url(
+            url_time, use_stichtag=True
+        )  # Use different url-structure for older downloads
+        log.warning(
+            f"Download file was not found. Assuming that the link structure of MaStR has changed and retrying with download link: {url}"
+        )
+        r = requests.get(url, stream=True, headers={"User-Agent": USER_AGENT})
     if r.status_code == 404:
         log.error("Could not download file: download URL not found")
         return

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "open_mastr"
-version = "0.16.0"
+version = "0.16.1"
 dependencies = [
   "pandas>=2.2.2",
   "numpy",
@@ -79,4 +79,4 @@ open_mastr = [
 include = ["open_mastr", "open_mastr.soap_api", "open_mastr.soap_api.metadata", "open_mastr.utils", "open_mastr.utils.config", "open_mastr.xml_download"] # package names should match these glob patterns (["*"] by default)
 
 # from setup.py - not yet included in here
-# download_url="https://github.com/OpenEnergyPlatform/open-MaStR/archive""/refs/tags/v0.16.0.tar.gz",
+# download_url="https://github.com/OpenEnergyPlatform/open-MaStR/archive""/refs/tags/v0.16.1.tar.gz",