Merge pull request #2 from interTwin-eu/update

merge update to main

Author: jzvolensky

.github/workflows/docker-build-components.yaml

@@ -0,0 +1,101 @@
+name: Build and Publish Docker Images
+
+on:
+  push:
+    branches: [main, update]
+    paths:
+      - 'docker/**'
+      - '.github/workflows/docker-*.yaml'
+      - '.github/workflows/docker-*.yml'
+  release:
+    types: [published]
+  workflow_dispatch:
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+
+    strategy:
+      matrix:
+        component: [hydromt, surrogate, wflow]
+
+    steps:
+      - name: Show initial disk usage
+        run: df -h
+
+      - name: Free up disk space and move Docker storage
+        run: |
+          echo "=== Cleaning up system to free space ==="
+          sudo apt-get clean
+          sudo apt-get autoremove -y
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
+          
+          echo "=== Removing preinstalled Docker images and cache ==="
+          docker image prune -af || true
+          docker builder prune -af || true
+          sudo rm -rf /var/lib/docker || true
+          
+          echo "=== Moving Docker data to /mnt/docker for more space ==="
+          sudo mkdir -p /mnt/docker
+          sudo systemctl stop docker
+          sudo rsync -aP /var/lib/docker/ /mnt/docker/ || true
+          sudo rm -rf /var/lib/docker
+          sudo ln -s /mnt/docker /var/lib/docker
+          sudo systemctl start docker
+          
+          echo "=== Running final cleanup ==="
+          docker system prune -af --volumes || true
+          echo "=== Disk usage after cleanup ==="
+          df -h
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Extract metadata
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/${{ matrix.component }}
+          tags: |
+            type=raw,value=latest,enable={{is_default_branch}}
+            type=sha
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v5
+        with:
+          context: ./docker/${{ matrix.component }}
+          file: ./docker/${{ matrix.component }}/Dockerfile
+          push: true
+          load: false 
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          platforms: linux/amd64
+          cache-from: type=gha,scope=${{ matrix.component }}
+          cache-to: type=gha,mode=max,scope=${{ matrix.component }}
+          build-args: |
+            BUILDKIT_INLINE_CACHE=1
+
+      - name: Clean up after build
+        if: always()
+        run: |
+          echo "=== Cleaning up after build ==="
+          docker system prune -af --volumes || true
+          echo "=== Final disk usage ==="
+          df -h

.gitignore

@@ -1,17 +1,24 @@
 !hydromt/cwl/81iegmjn/
 
 .oscar-venv/
+.env*
+tests/*.z*
 
+scripts/
+
+.secrets.baseline
 /openeo/hydromt/hydromt-output/
 /openeo/wflow/wflow-output/
 /openeo/surrogate/surrogate-output/
 
-tests/tmp/
 
+docker/surrogate/src/outputs/
+tests/tmp/
+tests/tmp*
 .venv
 object_urls.txt
 
-*.geojson
+
 *.nc
 *.log
 .vscode/

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "example/openeo-processes-dask"]
+	path = example/openeo-processes-dask
+	url = https://github.com/jzvolensky/openeo-processes-dask.git
+	branch = run_oscar

.pre-commit-config.yaml

@@ -0,0 +1,33 @@
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
+    hooks:
+    -   id: check-yaml
+    -   id: end-of-file-fixer
+    -   id: trailing-whitespace
+    -   id: check-added-large-files
+    -   id: check-merge-conflict
+    -   id: debug-statements
+    -   id: check-docstring-first
+-   repo: https://github.com/pre-commit/mirrors-autopep8
+    rev: v2.0.4
+    hooks:
+    -   id: autopep8
+        args: [--in-place, --aggressive, --aggressive, --max-line-length=88]
+-   repo: https://github.com/psf/black
+    rev: 23.12.1
+    hooks:
+    -   id: black
+        language_version: python3
+        args: [--line-length=88]
+-   repo: https://github.com/pycqa/isort
+    rev: 5.13.2
+    hooks:
+    -   id: isort
+        args: [--profile=black, --line-length=88]
+-   repo: https://github.com/pycqa/flake8
+    rev: 7.0.0
+    hooks:
+    -   id: flake8
+        args: [--max-line-length=88, --ignore=E203,W503]
+        exclude: ^(archive/|docker/dummy/)

CHANGELOG.md

@@ -5,6 +5,25 @@
 Since we are working on 1 branch now and my commit messages are getting \
 ridiculously long, I thought it would be a good idea to start a changelog.
 
+## 01/11/2025 preparation for the review
+
+### Changes
+
+- Move old stuff to the archive
+- Rework the README to reflect the current state of the project
+- Update the environment.yaml to include all necessary packages for local development
+- add `example/usecase.ipynb` with a simple OpenEO workflow to run the use case
+- Add github action to build and push the docker images
+- Add documentation for the new features and changes
+- Updated labels of docker images in the Dockerfiles
+
+### In progress
+
+- Final fixes for the use case containers
+- Final testing and preparation of the use case example
+- Final review of the documentation
+- SQAaaS integration after this update
+
 ## 10/07/2024 Reworking the project structure in preparation for OpenEO demo
 
 ### Fixes

CONTRIBUTING.md

@@ -1,4 +1,4 @@
-# Contributing to HyDroForM (WIP)
+# Contributing to HyDroForM
 
 First off, thanks for taking the time to contribute! ❤️
 
@@ -39,7 +39,7 @@ We will then take care of the issue as soon as possible.
 
 ## I Want To Contribute
 
-> ### Legal Notice 
+> ### Legal Notice
 > When contributing to this project, you must agree that you have authored 100% of the content, that you have the necessary rights to the content and that the content you contribute may be provided under the project license.
 
 ### Reporting Bugs
@@ -101,7 +101,7 @@ Enhancement suggestions are tracked as [GitHub issues](/issues).
 - Use a **clear and descriptive title** for the issue to identify the suggestion.
 - Provide a **step-by-step description of the suggested enhancement** in as many details as possible.
 - **Describe the current behavior** and **explain which behavior you expected to see instead** and why. At this point you can also tell which alternatives do not work for you.
-- You may want to **include screenshots and animated GIFs** which help you demonstrate the steps or point out the part which the suggestion is related to. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux. 
+- You may want to **include screenshots and animated GIFs** which help you demonstrate the steps or point out the part which the suggestion is related to. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux.
 - **Explain why this enhancement would be useful** to most CONTRIBUTING.md users. You may also want to point out the other projects that solved it better and which could serve as inspiration.