Adapt ERA5 atmospheric forcing generation script

README.md

@@ -164,23 +164,48 @@ The created surface and domain file have negative longitudes that CLM5 does not
 
 ## Creation of forcing data from ERA5
 
-A possible source of atmospheric forcing for CLM5 is ERA5.
-The folder `mkforcing/` contains two scripts that assist the ERA5 retrieval.
-- `download_ERA5.py` contains a prepared retrieval for the cdsapi python module.
-By modifying the two loops inside the script it is possible to download ERA5 for any timerange.
-However, the script requires that cdsapi is installed with an user specific key.
-More information about the installation can be found [here](https://cds.climate.copernicus.eu/api-how-to).
-- `prepare_ERA5.sh` prepares ERA5 as an input by changing names and modifying units.
-ERA5 has to be regridded to your resolution before the script can be used.
+A possible source of atmospheric forcing for CLM (eCLM, CLM5, CLM3.5) is ERA5. It is safer to extract the lowermost level of temperature, humidity and wind of ERA5 instead of taking mixed 2m-values and 10m values. [This internal issue](https://gitlab.jsc.fz-juelich.de/HPSCTerrSys/tsmp-internal-development-tracking/-/issues/36) provides some details. The `download_ERA5_input.py` can be adapted to download another set of quantities.
 
-`download_ERA5_v2.py`, `prepare_ERA5_v2.sh` and `extract_ERA5_meteocloud.sh` provide an alternative pathway. [This issue](https://gitlab.jsc.fz-juelich.de/HPSCTerrSys/tsmp-internal-development-tracking/-/issues/36) provides some details. Basically it is safer to extract the lowermost level of temperature, humidity and wind of ERA5 instead of taking 2m-values. The workflow goes like this:
+The folder `mkforcing/` contains three scripts that assist the ERA5 retrieval.
 
+Note: This worfklow is not fully tested.
+
+### Download of ERA5 data
+
+`download_ERA5_input.py` contains a prepared retrieval for the cdsapi python module.
+The script requires that cdsapi is installed with a user specific key (API access token).
+
+More information about the installation and access can be found [here](https://cds.climate.copernicus.eu/how-to-api) or alternatively [here](https://github.com/ecmwf/cdsapi?tab=readme-ov-file#install).
+
+Usage:
+Either directly:
+`python download_ERA5_input.py <year> <month> <output_directory>`
+Or using the wrapper script:
+`./download_ERA5_input_wrapper.sh`
+after changing dates and output directory in the `Settings` section inside this wrapper script.
+
+Non-JSC users should adapt the download script to include temperature, specific humidity and horizontal wind speed.
+
+### Preparation of ERA5 data I: Names and units
+`extract_ERA5_meteocloud.sh` prepares ERA5 as an input by changing names and modifying units (JSC users only).
+
+Usage:
+Running the wrapper job
+`sbatch extract_ERA5_meteocloud_wrapper.job`
+after adapting `year` and `month` loops according to needed dates.
+
+### Preparation of ERA5 data II: Remapping
+`prepare_ERA5_input.sh` prepares ERA5 as an input by remapping the ERA5 data, changing names and modifying units. The script is divided into three parts, which could be handled separately. Remapping, merging the data, and special treatment in case CLM3.5 forces data preparation. 
+
+If remapping is to be used, the remapping weights for the ERA data as well as the grid definition file of the target domain should be created beforehand. The following commands can be used to create the necessary files:
 ```
-bash extract_ERA5_meteocloud.sh
-python download_ERA5_v2.py
-regridding
-bash prepare_ERA5_v2.sh
+cdo gendis,<eclm_domainfile.nc> <era5caf_yyyy_mm.nc> <wgtdis_era5caf_to_domain.nc>
+cdo gendis,<eclm_domainfile.nc> <era5meteo_yyyy_mm.nc> <wgtdis_era5meteo_to_domain.nc>
+cdo griddes <eclm_domainfile.nc> > <domain_griddef.txt>
 ```
 
-Note: This worfklow is not fully tested.
+Usage:
+`sh prepare_ERA5_input.sh iyear=<year> imonth=<month> wgtcaf=<wgtcaf> wgtmeteo=<wgtmeteo> griddesfile=<griddesfile>`
+More options are available, see script for details.
+
 

mkforcing/download_ERA5_input.py

@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+import calendar
+import cdsapi
+import sys
+import os
+
+def generate_days(year, month):
+    # Get the number of days in the given month
+    num_days = calendar.monthrange(year, month)[1]
+
+    # Generate the list of days as integers
+    days = [day for day in range(1, num_days + 1)]
+
+    return days
+
+def generate_datarequest(year, monthstr, days):
+
+    # active download client for climate data service (cds)
+    client = cdsapi.Client()
+
+    # dataset to download rom cds
+    dataset = "reanalysis-era5-single-levels"
+    # request for cds
+    request = {
+        "product_type": ["reanalysis"],
+        "variable": [
+            "surface_pressure",
+            "mean_surface_downward_long_wave_radiation_flux",
+            "mean_surface_downward_short_wave_radiation_flux",
+            "mean_total_precipitation_rate"
+        ],
+        "year": [str(year)],
+        "month": [monthstr],
+        "day": days,
+        "time": [
+            "00:00", "01:00", "02:00",
+            "03:00", "04:00", "05:00",
+            "06:00", "07:00", "08:00",
+            "09:00", "10:00", "11:00",
+            "12:00", "13:00", "14:00",
+            "15:00", "16:00", "17:00",
+            "18:00", "19:00", "20:00",
+            "21:00", "22:00", "23:00"
+        ],
+        "data_format": "netcdf",
+        "download_format": "unarchived",
+        "area": [74, -42, 20, 69]
+    }
+    # filename of downloaded file
+    target = 'download_era5_'+str(year)+'_'+monthstr+'.zip'
+
+    # Get the data from cds
+    client.retrieve(dataset, request, target)
+
+if __name__ == "__main__":
+    # Check if the correct number of arguments are provided
+    if len(sys.argv) != 4:
+        print("Usage: python download_ERA5_input.py <year> <month> <output_directory>")
+        sys.exit(1)
+
+    # Get the year and month from command-line arguments
+    year = int(sys.argv[1])
+    month = int(sys.argv[2])
+    dirout = sys.argv[3]
+
+    # Ensure the output directory exists, if not, create it
+    if not os.path.exists(dirout):
+        os.makedirs(dirout)
+
+    # change to output directory
+    os.chdir(dirout)
+
+    # Format the month with a leading zero if needed
+    monthstr = f"{month:02d}"
+
+    # Get the list of days for the request
+    days = generate_days(year, month)
+
+    # do download request
+    generate_datarequest(year, monthstr, days)

mkforcing/download_ERA5_input_wrapper.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Before using this script CDSAPI has to be configured (see README)
+# Needs to be executed at LOGIN node as connection to "outside" is required
+set -eo pipefail
+
+# load environment
+module load Python
+
+# Settings
+start_date="2017-07" # yyyy-mm
+end_date="2018-08"   # yyyy-mm + 1
+out_dir="cdsapidwn"
+
+# Function to parse input
+parse_arguments() {
+    for arg in "$@"; do
+        key="${arg%%=*}"
+        value="${arg#*=}"
+
+        case "$key" in
+            start_date) start_date="$value" ;;
+            end_date) end_date="$value" ;;
+            out_dir) out_dir="$value" ;;
+            *) echo "Warning: Unknown parameter: $key" ;;
+        esac
+    done
+}
+
+# Call the function to parse the input arguments
+# Users needs to make sure for consistent input
+parse_arguments "$@"
+
+
+# create output directory
+mkdir -p $out_dir
+
+# loop over months
+current_date=$start_date
+while [[ "$current_date" < "$end_date" ]]; do
+    echo "Processing month: $current_date"
+
+    year="${current_date%%-*}"
+    month="${current_date#*-}"
+
+    # start download script with data request
+    ./download_ERA5_input.py $year $month $out_dir
+
+    # Increment the month
+    current_date=$(date -I -d "$current_date-01 + 1 month" | cut -d'-' -f1,2)
+done