Skip to content

Commit c69e90e

Browse files
jzvolenskyjzvolensky
committed
use case notebook setup
1 parent ecfc2a7 commit c69e90e

13 files changed

+606
-5
lines changed

‎.gitmodules‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
[submodule "example/openeo-processes-dask"]
2+
path = example/openeo-processes-dask
3+
url = https://github.com/jzvolensky/openeo-processes-dask.git
4+
branch = run_oscar

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ There are **three main components** in the HyDroForM use case:
6161

6262
### HydroMT
6363

64-
HydroMT (Hydro Model Tools) is an open-source Python package that facilitates the process of building and analyzing spatial geoscientific models with a focus on water system models. It does so by automating the workflow to go from raw data to a complete model instance which is ready to run and to analyse model results once the simulation has finished. HydroMT builds on the latest packages in the scientific and geospatial python eco-system including xarray, rasterio, rioxarray, geopandas, scipy and pyflwdir. Source: [Deltares HydroMT](https://deltares.github.io/hydromt/latest/)
64+
HydroMT is an open-source Python package that facilitates the process of building and analyzing spatial geoscientific models with a focus on water system models. It does so by automating the workflow to go from raw data to a complete model instance which is ready to run and to analyse model results once the simulation has finished. HydroMT builds on the latest packages in the scientific and geospatial python eco-system including xarray, rasterio, rioxarray, geopandas, scipy and pyflwdir. Source: [Deltares HydroMT](https://deltares.github.io/hydromt/latest/)
6565

6666
### Wflow
6767

‎example/configs/dummy.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
python script.py

‎example/configs/dummy.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
functions:
2+
oscar:
3+
- cluster_id:
4+
name: openeo-dummy
5+
cpu: '1.0'
6+
memory: 2Gi
7+
image: potato55/dummy-container:latest #change to -test if broken
8+
script: configs/dummy.sh # Path to the script from where the script is run
9+
log_level: CRITICAL # To avoid supervisor logs in response
10+
vo: dev.intertwin.eu
11+
environment:
12+
variables:
13+
DUMMY_INPUT1: "dummy_value"
14+
DUMMY_INPUT2: "another_value"
15+
INPUT_STAC: "https://example.com/dummy-stac.json"

‎example/configs/oscar_hydromt_script.sh‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
#!/usr/bin/env bash
2+
3+
echo "HydroMT OSCAR Script Started"
4+
5+
# Decode keys
6+
# if [ -f "decode_keys.sh" ]; then
7+
# . decode_keys.sh
8+
# echo "Finished decoding keys"
9+
# else
10+
# echo "Warning: decode_keys.sh not found, skipping key decoding."
11+
# fi
12+
13+
echo "Generating config file for HydroMT"
14+
# Set defaults or use environment variables if provided
15+
model_resolution="${MODEL_RESOLUTION:-0.008999999999}"
16+
precip_fn="${PRECIP_FN:-emo1_stac}"
17+
starttime="${STARTTIME:-2001-01-01T00:00:00}"
18+
endtime="${ENDTIME:-2001-03-31T00:00:00}"
19+
20+
if ! python config_gen.py \
21+
--res "$model_resolution" \
22+
--precip_fn "$precip_fn" \
23+
--starttime "$starttime" \
24+
--endtime "$endtime"
25+
then
26+
echo "Error: Failed to generate HydroMT config."
27+
exit 1
28+
fi
29+
30+
echo "Finished generating config file for HydroMT (wflow.ini)"
31+
32+
# Check config file
33+
if [ ! -f "wflow.ini" ]; then
34+
echo "Error: wflow.ini not found."
35+
exit 2
36+
fi
37+
38+
echo "Running hydromt build wflow model"
39+
region="{'subbasin': [11.4750, 46.8720]}"
40+
setupconfig="wflow.ini"
41+
catalog="data_catalog.yaml"
42+
echo "region: $region"
43+
echo "setupconfig: $setupconfig"
44+
echo "catalog: $catalog"
45+
if ! hydromt build wflow model -r "$region" -d "$catalog" -i "$setupconfig" -vvv; then
46+
echo "Error: HydroMT build wflow model failed."
47+
exit 3
48+
fi
49+
echo "Finished running hydromt build wflow model"
50+
51+
# Convert wflow_sbm.toml to lowercase
52+
if [ ! -f "/hydromt/model/wflow_sbm.toml" ]; then
53+
echo "Error: /hydromt/model/wflow_sbm.toml not found."
54+
exit 4
55+
fi
56+
echo "Converting wflow_sbm.toml to lowercase"
57+
if ! python /hydromt/convert_lowercase.py "/hydromt/model/wflow_sbm.toml"; then
58+
echo "Error: Conversion to lowercase failed."
59+
exit 5
60+
fi
61+
echo "Finished converting wflow_sbm.toml to lowercase"
62+
63+
# Generate STAC files from HydroMT model output
64+
STATICMAPS="/hydromt/model/staticmaps.nc"
65+
FORCINGS="/hydromt/model/forcings.nc"
66+
WFLOW_SBM="/hydromt/model/wflow_sbm.toml"
67+
STAC_OUT="/hydromt/model/stac"
68+
69+
for f in "$STATICMAPS" "$FORCINGS" "$WFLOW_SBM"; do
70+
if [ ! -f "$f" ]; then
71+
echo "Error: Required file $f not found."
72+
exit 6
73+
fi
74+
done
75+
76+
echo "Generating STAC files from HydroMT model output"
77+
if ! python /hydromt/stac.py --staticmaps_path "$STATICMAPS" --forcings_path "$FORCINGS" --wflow_sbm_path "$WFLOW_SBM" --output_dir "$STAC_OUT"; then
78+
echo "Error: STAC file generation failed."
79+
exit 7
80+
fi
81+
echo "Finished generating STAC files from HydroMT model output"
82+
83+
echo "End of script"
84+
# # Generate config file for HydroMT
85+
# echo "decoding keys"
86+
# source decode_keys.sh
87+
# echo "Finished decoding keys"
88+
# echo "Generating config file for HydroMT"
89+
# model_resolution=0.008999999999
90+
# dataset="emo1_stac"
91+
# python config_gen.py "$model_resolution" "$dataset"
92+
# echo "Finished generating config file for HydroMT (wflow.ini)"
93+
94+
# # Run HydroMT build wflow model
95+
# echo "Running hydromt build wflow model"
96+
# region="{'subbasin': [11.4750, 46.8720]}"
97+
# setupconfig="wflow.ini"
98+
# catalog="data_catalog.yaml"
99+
# echo "region: $region"
100+
# echo "setupconfig: $setupconfig"
101+
# echo "catalog: $catalog"
102+
# hydromt build wflow model \
103+
# -r "$region" \
104+
# -d "$catalog" \
105+
# -i "$setupconfig" -vvv
106+
# echo "Finished running hydromt build wflow model"
107+
108+
# # Convert wflow_sbm.toml to lowercase
109+
# echo "Converting wflow_sbm.toml to lowercase (yes... this is necessary)"
110+
# python /hydromt/convert_lowercase.py "/hydromt/model/wflow_sbm.toml"
111+
# echo "Finished converting wflow_sbm.toml to lowercase"
112+
113+
# # Generate STAC files from HydroMT model output
114+
# echo "Generating STAC files from HydroMT model output"
115+
# python /hydromt/stac.py --staticmaps_path "/hydromt/model/staticmaps.nc" --forcings_path "/hydromt/model/forcings.nc" --wflow_sbm_path "/hydromt/model/wflow_sbm.toml" --output_dir "/hydromt/model/stac"
116+
# echo "Finished generating STAC files from HydroMT model output"
117+
118+
# # Party hard
119+
# echo "End of script"

‎example/configs/oscar_hydromt_svc.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
functions:
2+
oscar:
3+
- cluster_id:
4+
name: hydroform-hydromt
5+
cpu: '1.0'
6+
memory: 3Gi
7+
image: ghcr.io/intertwin-eu/hydroform/hydromt:sha-ecfc2a7 #ghcr.io/intertwin-eu/hydroform/hydromt:sha-17359f3
8+
script: ./configs/oscar_hydromt_script.sh
9+
log_level: CRITICAL
10+
vo: dev.intertwin.eu
11+
environment:
12+
secrets:
13+
AWS_ACCESS_KEY_ID: ""
14+
AWS_SECRET_ACCESS_KEY: ""
15+
RES: "0.0089999999"
16+
PRECIP_FN: "emo1_stac"
17+
TEMP_PET_FN: "emo1_stac"
18+
STARTTIME: "2001-01-01T00:00:00"
19+
ENDTIME: "2001-03-31T00:00:00"

‎example/configs/oscar_surrogate_script.sh‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/usr/bin/env bash
2+
3+
echo "OSCAR Surrogate Script Started"
4+
5+
echo "Reading environment variables for surrogate training"
6+
7+
echo $PWD
8+
9+
# Read environment variables
10+
mlflow_tracking_uri="${MLFLOW_TRACKING_URI}"
11+
mlflow_tracking_username="${MLFLOW_TRACKING_USERNAME}"
12+
mlflow_tracking_insecure_tls="${MLFLOW_TRACKING_INSECURE_TLS}"
13+
mlflow_tracking_password="${MLFLOW_TRACKING_PASSWORD}"
14+
aws_access_key="${AWS_ACCESS_KEY_ID}"
15+
aws_secret_key="${AWS_SECRET_ACCESS_KEY}"
16+
17+
SURR_DIR=$PWD/docker/surrogate
18+
19+
if [ ! -d "$SURR_DIR/tmp" ]; then
20+
mkdir -p $SURR_DIR/tmp
21+
fi
22+
23+
if [ -z "$1" ]; then
24+
# CPU
25+
docker run -it --rm \
26+
-v "$PWD/tests/tmp:/data" \
27+
--ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
28+
-e MLFLOW_TRACKING_URI=$mlflow_tracking_uri \
29+
-e MLFLOW_TRACKING_USERNAME=$mlflow_tracking_username \
30+
-e MLFLOW_TRACKING_INSECURE_TLS=$mlflow_tracking_insecure_tls \
31+
-e MLFLOW_TRACKING_PASSWORD=$mlflow_tracking_password \
32+
-e AWS_ACCESS_KEY_ID=$aws_access_key \
33+
-e AWS_SECRET_ACCESS_KEY=$aws_secret_key \
34+
surrogate-test:latest /bin/bash -c "python ./use-case/gen_surr_config.py \
35+
--config './use-case/training_local.yaml' \
36+
--output './use-case/training_local_updated.yaml' \
37+
--cp 'train_temporal_range=[\"2001-01-01\", \"2001-03-31\"]' \
38+
--cp 'valid_temporal_range=[\"2001-01-01\", \"2001-03-31\"]' \
39+
--cp 'test_temporal_range=[\"2001-01-01\", \"2001-03-31\"]' \
40+
&& cat ./use-case/training_local_updated.yaml \
41+
&& itwinai exec-pipeline --config-dir ./use-case --config-name training_local_updated"
42+
43+
44+
elif [ "$1" == "gpu" ]; then
45+
echo "Running with GPU"
46+
docker run -it --rm \
47+
-v $PWD/tests/tmp:/model \
48+
-v /mnt/CEPH_PROJECTS/InterTwin/hydrologic_data/surrogate_input:/data \
49+
--gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
50+
-e MLFLOW_TRACKING_URI=$mlflow_tracking_uri -e MLFLOW_TRACKING_USERNAME=$mlflow_tracking_username \
51+
-e MLFLOW_TRACKING_INSECURE_TLS=$mlflow_tracking_insecure_tls -e MLFLOW_TRACKING_PASSWORD=$mlflow_tracking_password \
52+
-e AWS_ACCESS_KEY_ID=$aws_access_key -e AWS_SECRET_ACCESS_KEY=$aws_secret_key \
53+
surrogate-test:latest /bin/bash -c '
54+
echo "Inside container:"
55+
echo "Working dir: $(pwd)"
56+
echo "Contents of /app/use-case:"
57+
ls -lah /app/use-case
58+
echo "Contents of /data:"
59+
ls -lah /data
60+
echo "Environment:"
61+
env
62+
echo "Starting pipeline..."
63+
itwinai exec-pipeline --config-dir ./use-case --config-name training_local
64+
'
65+
fi

‎example/configs/oscar_surrogate_svc.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
functions:
2+
oscar:
3+
- cluster_id:
4+
name: hydroform-surrogate
5+
cpu: '1.0'
6+
memory: 8Gi
7+
image: ghcr.io/intertwin-eu/hydroform/surrogate:sha-b598b58
8+
script: configs/oscar_surrogate_script.sh
9+
log_level: CRITICAL
10+
vo: dev.intertwin.eu
11+
environment:
12+
secrets:
13+
AWS_ACCESS_KEY_ID: ""
14+
AWS_SECRET_ACCESS_KEY: ""
15+
MLFLOW_TRACKING_URI: ""
16+
MLFLOW_TRACKING_USERNAME: ""
17+
MLFLOW_TRACKING_PASSWORD: ""
18+
MLFLOW_TRACKING_INSECURE_TLS: ""
19+

‎example/configs/oscar_wflow_script.sh‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
#!/usr/bin/env bash
2+
# filepath: /home/jzvolensky/eurac/projects/openeo-processes-dask/OSCAR/configs/oscar_wflow_script.sh
3+
4+
set -euo pipefail
5+
6+
echo "HyDroForM Wflow Script"
7+
8+
# # Set AWS credentials if provided
9+
# if [[ -n "${AWS_ACCESS_KEY_ID:-}" && -n "${AWS_SECRET_ACCESS_KEY:-}" ]]; then
10+
# export AWS_ACCESS_KEY_ID
11+
# export AWS_SECRET_ACCESS_KEY
12+
# echo "AWS credentials set."
13+
# else
14+
# echo "Warning: AWS credentials not set."
15+
# fi
16+
17+
INPUT_STAC="${INPUT_STAC:-}"
18+
DATA_DIR="${DATA_DIR:-/data}"
19+
20+
if [ -z "$INPUT_STAC" ]; then
21+
echo "Error: INPUT_STAC variable is not set."
22+
exit 1
23+
fi
24+
25+
echo "Preparing environment"
26+
27+
if ! python3 /app/src/read_stac.py "$INPUT_STAC" "$DATA_DIR"; then
28+
echo "Error: Failed to read STAC files."
29+
exit 2
30+
fi
31+
32+
echo "STAC files read successfully"
33+
34+
WFLOW_TOML="$DATA_DIR/wflow_sbm.toml"
35+
if [ ! -f "$WFLOW_TOML" ]; then
36+
echo "Error: Wflow TOML file not found at $WFLOW_TOML"
37+
exit 3
38+
fi
39+
40+
echo "Running Wflow model"
41+
if ! run_wflow "$WFLOW_TOML"; then
42+
echo "Error: Wflow model run failed."
43+
exit 4
44+
fi
45+
46+
echo "Wflow model run completed"
47+
48+
FORCINGS_NC="$DATA_DIR/forcings.nc"
49+
STATICMAPS_NC="$DATA_DIR/staticmaps.nc"
50+
OUTPUT_NC="$DATA_DIR/run_default/output.nc"
51+
52+
for f in "$FORCINGS_NC" "$STATICMAPS_NC" "$OUTPUT_NC"; do
53+
if [ ! -f "$f" ]; then
54+
echo "Error: Required file $f not found."
55+
exit 5
56+
fi
57+
done
58+
59+
if ! python3 /app/src/to_zarr.py "$FORCINGS_NC" "$STATICMAPS_NC" "$OUTPUT_NC" "$DATA_DIR"; then
60+
echo "Error: Conversion to Zarr format failed."
61+
exit 6
62+
fi
63+
64+
echo "Conversion to Zarr format completed & STAC files generated"

‎example/configs/oscar_wflow_svc.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
functions:
2+
oscar:
3+
- cluster_id:
4+
name: hydroform-wflow
5+
cpu: '1.0'
6+
memory: 2Gi
7+
image: ghcr.io/intertwin-eu/hydroform/wflow:sha-27b00ad
8+
script: configs/oscar_wflow_script.sh
9+
log_level: CRITICAL
10+
vo: dev.intertwin.eu
11+
environment:
12+
secrets:
13+
AWS_ACCESS_KEY_ID: ""
14+
AWS_SECRET_ACCESS_KEY: ""

0 commit comments

Comments
 (0)