review preparation, many changes see the changelog

Author: jzvolensky

.github/workflows/docker-build-components.yaml

@@ -0,0 +1,60 @@
+name: Build and Publish Docker Images
+
+on:
+  push:
+    branches: [main, update]
+    paths:
+      - 'docker/**'
+  release:
+    types: [published]
+  workflow_dispatch:
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+
+    strategy:
+      matrix:
+        component: [hydromt, surrogate, wflow]
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Extract metadata
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/${{ matrix.component }}
+          tags: |
+            type=raw,value=latest,enable={{is_default_branch}}
+            type=sha
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v5
+        with:
+          context: ./docker/${{ matrix.component }}
+          file: ./docker/${{ matrix.component }}/Dockerfile
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          platforms: linux/amd64,linux/arm64
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

.gitignore

@@ -4,6 +4,9 @@
 .env*
 tests/*.z*
 
+scripts/
+
+.secrets.baseline
 /openeo/hydromt/hydromt-output/
 /openeo/wflow/wflow-output/
 /openeo/surrogate/surrogate-output/

CHANGELOG.md

@@ -5,6 +5,25 @@
 Since we are working on 1 branch now and my commit messages are getting \
 ridiculously long, I thought it would be a good idea to start a changelog.
 
+## 01/11/2025 preparation for the review
+
+### Changes
+
+- Move old stuff to the archive
+- Rework the README to reflect the current state of the project
+- Update the environment.yaml to include all necessary packages for local development
+- add `example/usecase.ipynb` with a simple OpenEO workflow to run the use case
+- Add github action to build and push the docker images
+- Add documentation for the new features and changes
+- Updated labels of docker images in the Dockerfiles
+
+### In progress
+
+- Final fixes for the use case containers
+- Final testing and preparation of the use case example
+- Final review of the documentation
+- SQAaaS integration after this update
+
 ## 10/07/2024 Reworking the project structure in preparation for OpenEO demo
 
 ### Fixes

README.md

@@ -4,6 +4,21 @@
 
 ## Table of Contents
 
+- [Introduction](#introduction)
+- [Repository structure](#repository-structure)
+- [Environment setup](#environment-setup)
+- [Use case components](#use-case-components)
+    - [HydroMT](#hydromt)
+    - [Wflow](#wflow)
+    - [Surrogate model based on ItwinAI](#surrogate-model-based-on-it
+winai)
+- [OSCAR](#oscar)
+- [Running the use case using openEO and OSCAR](#running-the-use-case-using-openeo-and-oscar)
+- [openEO OSCAR integration](#openeo-oscar-integration)
+- [Tests](#tests)
+- [License](#license)
+- [Project framework](#project-framework)
+
 ## Introduction
 
 HyDroForM stands for "Hydrological Drought Forecasting Model with HydroMT and Wflow". It is a Digital Twin for Drought Early Warning in the Alps developed as a use case for the [InterTwin project](https://www.intertwin.eu/). The details of the use case are also available online [here](https://www.intertwin.eu/intertwin-use-case-a-digital-twin-for-drought-early-warning-in-the-alps).
@@ -19,7 +34,17 @@ InterTwin components used in this use case are:
 - [ItwinAI](https://github.com/interTwin-eu/itwinai)
 - [OSCAR](https://github.com/grycap/oscar)
 - [Hython](https://github.com/interTwin-eu/hython)
-- [InterLink](https://github.com/interTwin-eu/interLink)
+
+## Repository structure
+
+- `Archive`: contains older versions and CWL descriptions of the use case
+- `docker`: contains the Dockerfiles and scripts to build and run the components of the use case
+- `docs`: documentation and images
+- `example`: example files for running the use case
+- `OSCAR`: contains the OSCAR deployment files for the use case
+- `scripts`: helper scripts used during the development of the use case
+- `tests`: scripts to test the components of the use case
+- `environment.yaml`: conda environment file to set up the development environment
 
 ## Environment setup
 
@@ -30,10 +55,6 @@ conda env create -f environment.yaml
 conda activate hydroform
 ```
 
-## TODO: Use case diagram
-
-## TODO: System design diagram
-
 ## Use case components
 
 There are **three main components** in the HyDroForM use case:
@@ -42,27 +63,41 @@ There are **three main components** in the HyDroForM use case:
 
 HydroMT (Hydro Model Tools) is an open-source Python package that facilitates the process of building and analyzing spatial geoscientific models with a focus on water system models. It does so by automating the workflow to go from raw data to a complete model instance which is ready to run and to analyse model results once the simulation has finished. HydroMT builds on the latest packages in the scientific and geospatial python eco-system including xarray, rasterio, rioxarray, geopandas, scipy and pyflwdir. Source: [Deltares HydroMT](https://deltares.github.io/hydromt/latest/)
 
-#### Running HydroMT
-
-To run HydroMT from start to finish you can use the `validation` script which is located in `/docker/hydromt/validation.sh`. This script will run the HydroMT validation test which includes the following steps:
+### Wflow
 
-1. Update the configuration file of HydroMT
-2. Run HydroMT using the configuration file
-3. Convert the output Wflow configuration file to lowercase letters
-4. Wrap the outputs into STAC collections
+Wflow is Deltares’ solution for modelling hydrological processes, allowing users to account for precipitation, interception, snow accumulation and melt, evapotranspiration, soil water, surface water and groundwater recharge in a fully distributed environment. Successfully applied worldwide for analyzing flood hazards, drought, climate change impacts and land use changes, wflow is growing to be a leader in hydrology solutions. Wflow is conceived as a framework, within which multiple distributed model concepts are available, which maximizes the use of open earth observation data, making it the hydrological model of choice for data scarce environments. Based on gridded topography, soil, land use and climate data, wflow calculates all hydrological fluxes at any given grid cell in the model at a given time step.
 
-### Wflow
+Source: [Deltares Wflow](https://deltares.github.io/Wflow.jl/stable/)
 
-Wflow is Deltares’ solution for modelling hydrological processes, allowing users to account for precipitation, interception, snow accumulation and melt, evapotranspiration, soil water, surface water and groundwater recharge in a fully distributed environment. Successfully applied worldwide for analyzing flood hazards, drought, climate change impacts and land use changes, wflow is growing to be a leader in hydrology solutions. Wflow is conceived as a framework, within which multiple distributed model concepts are available, which maximizes the use of open earth observation data, making it the hydrological model of choice for data scarce environments. Based on gridded topography, soil, land use and climate data, wflow calculates all hydrological fluxes at any given grid cell in the model at a given time step. Source: [Deltares Wflow](https://deltares.github.io/Wflow.jl/stable/)
+### Surrogate model based on ItwinAI
 
-#### Running Wflow
+`itwinai` is a Python toolkit designed to help scientists and researchers streamline AI and machine learning workflows, specifically for digital twin applications. It provides easy-to-use tools for distributed training, hyper-parameter optimization on HPC systems, and integrated ML logging, reducing engineering overhead and accelerating research. Developed primarily by CERN, in collaboration with Forschungszentrum Jülich (FZJ), itwinai supports modular and reusable ML workflows, with the flexibility to be extended through third-party plugins, empowering AI-driven scientific research in digital twins.
 
-### TODO: Surrogate model
+Source: [ItwinAI](https://github.com/interTwin-eu/itwinai)
 
 ## OSCAR
 
 OSCAR is an open-source platform to support the event-driven serverless computing model for data-processing applications. It can be automatically deployed on multi-Clouds, and even on low-powered devices, to create highly-parallel event-driven data-processing serverless applications along the computing continuum. These applications execute on customized runtime environments provided by Docker containers that run on elastic Kubernetes clusters. It is also integrated with the SCAR framework, which supports a High Throughput Computing Programming Model to create highly-parallel event-driven data-processing serverless applications that execute on customized runtime environments provided by Docker containers run on AWS Lambda and AWS Batch. [OSCAR](https://github.com/grycap/oscar)
 
+## Running the use case using openEO and OSCAR
+
+The `OSCAR` directory contains the files necessary to deploy the use case on the OSCAR platform. There are 2 main components to do so, a bash script and a yaml service definition file.
+
+These can be found in the respective subdirectories:
+`OSCAR/oscar_hydromt`, `OSCAR/oscar_wflow`, and `OSCAR/oscar_surrogate`
+
+To run the use case we have created a sample Jupyter notebook `example/usecase.ipynb` which can be used to run the use case using the openEO API.
+
+The example shows how the three components are linked together to create a drought forecasting workflow.
+
+## openEO OSCAR integration
+
+The integration of openEO with OSCAR is done in the dask/xarray implementation of openEO called `openeo-processes-dask`. The openEO backend is the main orchestration component of the use case. It is responsible for managing the execution of the different components of the use case on OSCAR.
+
+The backend now implements the `oscar_python` library to submit tasks to OSCAR from the process graph.
+
+When the process graph is executed, the `run_oscar` process authenticates with the OSCAR platform, validates the service definition file and submits the job to OSCAR. The process then monitors the job status and retrieves the results once the job is completed. If the service definition contains a process not yet registered in OSCAR it will be created on the fly. The process parameters are passed as environment variables to the container where the scripts are executed. The results are stored as STAC collections and returned to openEO as a string URL to the collection.
+
 ## Tests
 
 The components of the use case are set up in `Docker containers`. We have a set of scripts available to build and run the base images. These can be found in the `/tests` directory and can be run from `root` directory of the repository.
@@ -73,8 +108,12 @@ For example:
 ./tests/test_hydromt.sh
 ```
 
-## TODO: Use case demonstration
-
 ## License
 
 This project is licensed under the Apache 2.0 - see the [LICENSE](LICENSE) file for details.
+
+## Project framework
+
+interTwin is an EU-funded project with the goal to co-design and implement the prototype of an interdisciplinary Digital Twin Engine – an open source platform based on open standards that offers the capability to integrate with application-specific Digital Twins.
+
+interTwin is funded by the European Union Grant Agreement Number 101058386

docker/dummy/Dockerfile

@@ -0,0 +1,5 @@
+FROM python:3.13-slim
+
+WORKDIR /app
+
+COPY script.py .

docker/dummy/script.py

@@ -0,0 +1,30 @@
+import os
+import time
+import logging
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s - %(levelname)s - %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+
+logger = logging.getLogger(__name__)
+
+
+def main():
+    # Read environment variables (simulate required inputs)
+    input1 = os.getenv("DUMMY_INPUT1", "default1")
+    input2 = os.getenv("DUMMY_INPUT2", "default2")
+    input_stac = os.getenv("INPUT_STAC", "default_stac")
+    logger.info(f"Received inputs: DUMMY_INPUT1={input1},\n DUMMY_INPUT2={input2},\n INPUT_STAC={input_stac}")
+
+    # Simulate processing
+    logger.info("Simulating processing...")
+    time.sleep(2)
+
+    # Return a fixed URL as output
+    output_url = "https://stac.intertwin.fedcloud.eu/collections/8db57c23-4013-45d3-a2f5-a73abf64adc4_WFLOW_FORCINGS_STATICMAPS"
+    logger.info(f"STAC OUTPUT URL {output_url}")
+
+if __name__ == "__main__":
+    main()

docker/hydromt/Dockerfile

@@ -1,8 +1,8 @@
 FROM python:3.10-bullseye AS build
 
-LABEL version="EC Demo Review"
+LABEL version="1.0"
 LABEL description="Hydromt Docker image for building and updating hydromt models"
-LABEL maintainer="Juraj Zvolensky"
+LABEL maintainer="Juraj Zvolensky, Iacopo Ferrario"
 LABEL organization="Eurac Research"
 
 WORKDIR /hydromt
@@ -28,6 +28,8 @@ RUN cd .
 
 RUN pip uninstall -y pydantic && pip install pydantic==2.8.2 openeo_pg_parser_networkx==2024.10.0
 
+RUN pip install pystac==1.14.0
+
 ##################### HydroMT Components setup #####################
 
 RUN mkdir -p /hydromt/output /hydromt/data
@@ -39,13 +41,15 @@ COPY data_catalog.yaml /hydromt/data_catalog.yaml
 COPY stac.py /hydromt/stac.py
 COPY config_gen.py /hydromt/config_gen.py
 COPY convert_lowercase.py /hydromt/convert_lowercase.py
+COPY decode_keys.sh /hydromt/decode_keys.sh
 COPY build.sh /hydromt/build.sh
 ##################### Set executables #####################
 
 RUN chmod +x  /hydromt/stac.py \
               /hydromt/config_gen.py \
               /hydromt/build.sh \
-              /hydromt/convert_lowercase.py
+              /hydromt/convert_lowercase.py \
+              /hydromt/decode_keys.sh
 
 FROM python:3.10-bullseye