Optionally use ohsomeDB to compute Currentness indicator

Jenkinsfile

@@ -57,7 +57,7 @@ pipeline {
             steps {
                 script {
                     // run pytest
-                    sh 'VCR_RECORD_MODE=none pytest --cov=ohsome_quality_api --cov-report=xml --numprocesses=4 tests'
+                    sh 'VCR_RECORD_MODE=none pytest --cov=ohsome_quality_api --cov-report=xml tests'
                     // run static analysis with sonar-scanner
                     def scannerHome = tool 'SonarScanner 4'
                     withSonarQubeEnv('sonarcloud GIScience/ohsome') {

config/sample.config.yaml

@@ -1,5 +1,10 @@
 ---
 # Database connection parameters;
+ohsomedb_host: localhost
+ohsomedb_port: 5432
+ohsomedb_db: postgres
+ohsomedb_user: postgres
+ohsomedb_password: mylocalpassword
 postgres_host: localhost
 postgres_port: 5445
 postgres_db: oqapi

docs/configuration.md

@@ -8,7 +8,12 @@ Below is a table listing all possible configuration variables.
 
 | Configuration Variable Name | Environment Variable Name       | Configuration File Name   | Default Value                  | Description                                                                 |
 | --------------------------- | ------------------------------- | ------------------------- | ------------------------------ | --------------------------------------------------------------------------- |
-| Postgres Host               | `POSTGRES_HOST`                 | `postgres_host`           | `localhost`                    | Database connection parameter                                               |
+| ohsomeDB Host               | `OHSOMEDB_HOST`                 | `ohsomedb_host`           | `localhost`                    | ohsomeDB database connection parameter                                      |
+| ohsomeDB Port               | `OHSOMEDB_PORT`                 | `ohsomedb_port`           | `5432`                         | "                                                                           |
+| ohsomeDB Database           | `OHSOMEDB_DB`                   | `ohsomedb_db`             | `postgres`                     | "                                                                           |
+| ohsomeDB User               | `OHSOMEDB_USER`                 | `ohsomedb_user`           | `postgres`                     | "                                                                           |
+| ohsomeDB Password           | `OHSOMEDB_PASSWORD`             | `ohsomedb_password`       | `mylocalpassword`              | "                                                                           |
+| Postgres Host               | `POSTGRES_HOST`                 | `postgres_host`           | `localhost`                    | Postgres database connection parameter                                      |
 | Postgres Port               | `POSTGRES_PORT`                 | `postgres_port`           | `5445`                         | "                                                                           |
 | Postgres Database           | `POSTGRES_DB`                   | `postgres_db`             | `oqapi`                        | "                                                                           |
 | Postgres User               | `POSTGRES_USER`                 | `postgres_user`           | `oqapi`                        | "                                                                           |

docs/development_setup.md

@@ -38,6 +38,14 @@ Resolved 85 packages in 33ms
 ```
 
 
+## ohsomeDB
+
+For some experimental development work a local setup of the ohsomeDB is needed.
+Currently, ohsomeDB is under active development.
+You can find up-to-date setup instruction on [HeiGIT's GitLab](https://gitlab.heigit.org/giscience/big-data/ohsome/ohsomedb/ohsomedb/-/tree/main/local_setup).
+If you run a local ohsomeDB make sure to set the appropriate configuration variables (see next section).
+
+
 ## Configuration
 
 For all possible configuration parameter please refer to the [configuration documentation](/docs/configuration.md).

ohsome_quality_api/api/request_models.py

@@ -77,12 +77,24 @@ class IndicatorRequest(BaseBpolys, BaseRequestContext):
         alias="topic",
     )
     include_figure: bool = True
+    ohsomedb: bool | str = False
 
     @field_validator("topic")
     @classmethod
     def transform_topic(cls, value) -> TopicDefinition:
         return get_topic_preset(value.value)
 
+    @field_validator("ohsomedb")
+    @classmethod
+    def transform_ohsomedb(cls, value) -> bool:
+        if isinstance(value, str):
+            if value == "true":
+                return True
+            else:
+                return False
+        else:
+            return value
+
     @model_validator(mode="after")
     def validate_indicator_topic_combination(self):
         indicator = self.request_context.path_parameters["key"]

ohsome_quality_api/config.py

@@ -24,6 +24,11 @@ def get_config_path() -> str:
 
 def load_config_default() -> dict:
     return {
+        "ohsomedb_host": "localhost",
+        "ohsomedb_port": 5432,
+        "ohsomedb_db": "postgres",
+        "ohsomedb_user": "postgres",
+        "ohsomedb_password": "mylocalpassword",
         "postgres_host": "localhost",
         "postgres_port": 5445,
         "postgres_db": "oqapi",
@@ -56,6 +61,11 @@ def load_config_from_file(path: str) -> dict:
 def load_config_from_env() -> dict:
     """Load configuration from environment variables."""
     cfg = {
+        "ohsomedb_host": os.getenv("OHSOMEDB_HOST"),
+        "ohsomedb_port": os.getenv("OHSOMEDB_PORT"),
+        "ohsomedb_db": os.getenv("OHSOMEDB_DB"),
+        "ohsomedb_user": os.getenv("OHSOMEDB_USER"),
+        "ohsomedb_password": os.getenv("OHSOMEDB_PASSWORD"),
         "postgres_host": os.getenv("POSTGRES_HOST"),
         "postgres_port": os.getenv("POSTGRES_PORT"),
         "postgres_db": os.getenv("POSTGRES_DB"),

ohsome_quality_api/geodatabase/client.py

@@ -16,6 +16,7 @@
 
 import os
 from contextlib import asynccontextmanager
+from typing import Literal
 
 import asyncpg
 import geojson
@@ -28,24 +29,40 @@
 
 
 @asynccontextmanager
-async def get_connection():
+async def get_connection(database: Literal["oqapidb", "ohsomedb"] = "oqapidb"):
     # DNS in libpq connection URI format
-    dns = "postgres://{user}:{password}@{host}:{port}/{database}".format(
-        host=get_config_value("postgres_host"),
-        port=get_config_value("postgres_port"),
-        database=get_config_value("postgres_db"),
-        user=get_config_value("postgres_user"),
-        password=get_config_value("postgres_password"),
-    )
+    match database:
+        case "oqapidb":
+            dns = "postgres://{user}:{password}@{host}:{port}/{database}".format(
+                host=get_config_value("postgres_host"),
+                port=get_config_value("postgres_port"),
+                database=get_config_value("postgres_db"),
+                user=get_config_value("postgres_user"),
+                password=get_config_value("postgres_password"),
+            )
+        case "ohsomedb":
+            dns = "postgres://{user}:{password}@{host}:{port}/{database}".format(
+                host=get_config_value("ohsomedb_host"),
+                port=get_config_value("ohsomedb_port"),
+                database=get_config_value("ohsomedb_db"),
+                user=get_config_value("ohsomedb_user"),
+                password=get_config_value("ohsomedb_password"),
+            )
+        case _:
+            raise ValueError()
     conn = await asyncpg.connect(dns)
     try:
         yield conn
     finally:
         await conn.close()
 
 
-async def fetch(query: str, *args) -> list:
-    async with get_connection() as conn:
+async def fetch(
+    query: str,
+    *args,
+    database: Literal["oqapidb", "ohsomedb"] = "oqapidb",
+) -> list:
+    async with get_connection(database) as conn:
         return await conn.fetch(query, *args)
 
 

ohsome_quality_api/indicators/currentness/indicator.py

@@ -8,19 +8,23 @@
     th: Threshold
 """
 
+import json
 import locale
 import logging
 import os
 from dataclasses import dataclass
+from pathlib import Path
 from string import Template
 
 import plotly.graph_objects as pgo
 import yaml
 from dateutil.parser import isoparse
 from geojson import Feature
+from ohsome_filter_to_sql.main import ohsome_filter_to_sql
 from plotly.subplots import make_subplots
 
 from ohsome_quality_api.definitions import Color
+from ohsome_quality_api.geodatabase import client
 from ohsome_quality_api.indicators.base import BaseIndicator
 from ohsome_quality_api.ohsome import client as ohsome_client
 from ohsome_quality_api.topics.models import BaseTopic as Topic
@@ -29,7 +33,7 @@
 try:
     locale.setlocale(locale.LC_ALL, ["en_US", locale.getencoding()])
 except locale.Error:
-    logging.warn(
+    logging.warning(
         "Could not set locale to en_US. Output may be different than expected."
     )
 
@@ -69,7 +73,13 @@ def __init__(
         self.bin_in_between: Bin
         self.bin_out_of_date: Bin
 
-    async def preprocess(self):
+    async def preprocess(self, ohsomedb: bool = False):
+        if ohsomedb:
+            await self.preprocess_ohsomedb()
+        else:
+            await self.preprocess_ohsomeapi()
+
+    async def preprocess_ohsomeapi(self):
         """Fetch all latest contributions in monthly buckets since 2008
 
         Beside the creation, latest contribution includes also the change to the
@@ -114,6 +124,46 @@ async def preprocess(self):
         self.contrib_sum = contrib_sum
         self.result.timestamp_osm = self.bin_total.to_timestamps[0]
 
+    async def preprocess_ohsomedb(self):
+        where = ohsome_filter_to_sql(self.topic.filter)
+        with open(Path(__file__).parent / "query.sql", "r") as file:
+            template = file.read()
+        query = Template(template).substitute(
+            {
+                "aoi": json.dumps(self.feature["geometry"]),
+                "filter": where,
+            }
+        )
+        results = await client.fetch(query, database="ohsomedb")
+        if len(results) == 0:
+            # no data
+            self.contrib_sum = 0
+            return
+        to_timestamps = []
+        from_timestamps = []
+        timestamps = []
+        contrib_abs = []
+        contrib_sum = 0
+        for r in reversed(results):  # latest contributions first
+            to_timestamps.append(r[0])
+            from_timestamps.append(r[0])
+            timestamps.append(r[0])
+            contrib_abs.append(r[1])
+            contrib_sum += r[1]
+        if contrib_sum == 0:
+            contrib_rel = [0 for _ in contrib_abs]
+        else:
+            contrib_rel = [c / contrib_sum for c in contrib_abs]
+        self.bin_total = Bin(
+            contrib_abs,
+            contrib_rel,
+            to_timestamps,
+            from_timestamps,
+            timestamps,
+        )
+        self.contrib_sum = contrib_sum
+        self.result.timestamp_osm = self.bin_total.to_timestamps[0]
+
     def calculate(self):
         """Determine up-to-date, in-between and out-of-date contributions.
 

ohsome_quality_api/indicators/currentness/query.sql

@@ -0,0 +1,18 @@
+WITH bpoly AS (
+    SELECT
+        ST_Setsrid (ST_GeomFromGeoJSON ('$aoi'), 4326) AS geom
+)
+SELECT
+    Date_trunc('month', valid_from) AS month,
+    Count(*)
+FROM
+    contributions c,
+    bpoly b
+WHERE
+    ST_Intersects (c.geom, b.geom)
+    AND (status_geom_type).status = 'latest'  -- excludes deleted
+    AND ($filter)
+GROUP BY
+    month
+ORDER BY
+    month;

ohsome_quality_api/oqt.py

@@ -6,6 +6,7 @@
 from geojson import Feature, FeatureCollection
 
 from ohsome_quality_api.indicators.base import BaseIndicator as Indicator
+from ohsome_quality_api.indicators.currentness.indicator import Currentness
 from ohsome_quality_api.topics.models import BaseTopic as Topic
 from ohsome_quality_api.topics.models import TopicData, TopicDefinition
 from ohsome_quality_api.utils.helper import get_class_from_key
@@ -58,6 +59,7 @@ async def _create_indicator(
     feature: Feature,
     topic: Topic,
     include_figure: bool = True,
+    ohsomedb: bool = False,
     **kwargs,
 ) -> Indicator:
     """Create an indicator from scratch."""
@@ -74,7 +76,10 @@ async def _create_indicator(
     )
 
     logging.info("Run preprocessing")
-    await indicator.preprocess()
+    if isinstance(indicator, Currentness):
+        await indicator.preprocess(ohsomedb=ohsomedb)
+    else:
+        await indicator.preprocess()
 
     logging.info("Run calculation")
     indicator.calculate()

pyproject.toml

@@ -20,6 +20,7 @@ dependencies = [
     "geojson-pydantic>=2.0.0",
     "httpx>=0.28.1",
     "kaleido==0.2.1",
+    "ohsome-filter-to-sql",
     "plotly>=6.0.1",
     "pydantic>=2.11.4",
     "pyproj>=3.7.1",
@@ -37,17 +38,18 @@ Repository = "https://github.com/GIScience/ohsome-quality-api"
 [dependency-groups]
 dev = [
     "approvaltests>=14.5.0",
+    "asyncpg-recorder",
     "click>=8.2.0",
     "fastapi[standard]>=0.115.12",
     "pre-commit>=4.2.0",
-    "pygments>=2.19.1",  # add syntax highlighting to pytest
+    "pygments>=2.19.1", # add syntax highlighting to pytest
     "pytest>=8.3.5",
     "pytest-asyncio>=0.26.0",
     "pytest-cov>=6.1.1",
     "pytest-mock>=3.14.0",
     "pytest-sugar>=1.0.0",
     "pytest-xdist>=3.6.1",
-    "ruff==0.11.9",  # fixed because of pre-commit
+    "ruff==0.11.9", # fixed because of pre-commit
     "vcrpy>=7.0.0",
 ]
 
@@ -58,6 +60,10 @@ build-backend = "uv_build"
 [tool.uv.build-backend]
 module-root = ""
 
+[tool.uv.sources]
+ohsome-filter-to-sql = { git = "https://github.com/GIScience/ohsome-filter-to-sql.git" }
+asyncpg-recorder = { git = "https://github.com/GIScience/asyncpg-recorder.git" }
+
 [tool.ruff.lint]
 select = [
     "E",    # pycodestyle Error

tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_calculate[False].approved.txt

@@ -0,0 +1,2 @@
+In the area of interest 52% of the 30471 features were edited (created or modified) for the last time in the period between 20 Aug 2022 and 20 Aug 2025.
+Most features are up-to-date.

tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_calculate[True].approved.txt

@@ -0,0 +1,2 @@
+In the area of interest 71% of the 6879 features were edited (created or modified) for the last time in the period between 01 Aug 2022 and 01 Aug 2025.
+Most features are up-to-date.

tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_low_contributions.approved.txt → tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_low_contributions[False].approved.txt

@@ -1,2 +1,2 @@
-Please note that in the area of interest less than 25 features of the selected topic are present today. In the area of interest 55% of the 20 features were edited (created or modified) for the last time in the period between 02 Feb 2022 and 02 Feb 2025.
+Please note that in the area of interest less than 25 features of the selected topic are present today. In the area of interest 52% of the 20 features were edited (created or modified) for the last time in the period between 20 Aug 2022 and 20 Aug 2025.
 Most features are up-to-date.

tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_low_contributions[True].approved.txt

@@ -0,0 +1,2 @@
+Please note that in the area of interest less than 25 features of the selected topic are present today. In the area of interest 71% of the 20 features were edited (created or modified) for the last time in the period between 01 Aug 2022 and 01 Aug 2025.
+Most features are up-to-date.

tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_months_without_edit.approved.txt → tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_months_without_edit[False].approved.txt

@@ -1,2 +1,2 @@
-Please note that there was no mapping activity for 13 months in this region. In the area of interest 55% of the 30 features were edited (created or modified) for the last time in the period between 02 Feb 2022 and 02 Feb 2025.
+Please note that there was no mapping activity for 13 months in this region. In the area of interest 52% of the 30 features were edited (created or modified) for the last time in the period between 20 Aug 2022 and 20 Aug 2025.
 Most features are up-to-date.

tests/approvals/integrationtests/indicators/test_currentness.py-TestCalculation-test_months_without_edit[True].approved.txt

@@ -0,0 +1,2 @@
+Please note that there was no mapping activity for 13 months in this region. In the area of interest 71% of the 30 features were edited (created or modified) for the last time in the period between 01 Aug 2022 and 01 Aug 2025.
+Most features are up-to-date.