Merge branch 'master' into horayzon

Author: stelliom

.github/PULL_REQUEST_TEMPLATE/default.md

@@ -0,0 +1,46 @@
+## Workflow for merging PRs
+
+Please read these instructions carefully and follow the steps below before 
+requesting a review by the maintainers. This way we can ensure a smoother 
+review process and your changes will be merged sooner.
+
+Additionally, if this is the first PR you open in EXTPAR make sure to read the 
+[*"Information for EXTPAR Developers"*](https://c2sm.github.io/extpar/development/) 
+section in the documentation.
+
+### Checklist
+
+- [ ] Provide a detailed description of your changes.
+- [ ] If you implemented a new feature:
+  - [ ] Document it in the correct Markdown file(s) under the 
+[`docs/`](https://github.com/C2SM/extpar/tree/master/docs) directory.
+  - [ ] [Add a new test](https://c2sm.github.io/extpar/testing/#add-a-new-test) 
+or make sure your changes are already tested.
+- [ ] Your code follows the [style guidelines](https://c2sm.github.io/extpar/development/#coding-rules-and-best-practices).
+- [ ] Your changes only touch the files/lines relevant for you.
+- [ ] All four required checks pass (see 
+[*"Testing and debugging"*](#testing-and-debugging) for more details).
+- [ ] No conflicts with the base branch.
+
+If all the points above are satisfied you can ask for a review by selecting 
+`stelliom` as a reviewer.
+
+For any questions please ping `@stelliom` on this PR.
+
+### Testing and debugging
+
+The most important test for PRs is the one labeled 
+*"EXTPAR Testsuite on Jenkins"*. This checks that the results of all testcases 
+(described by the namelists in 
+[`test/testsuite/data`](https://github.com/C2SM/extpar/tree/master/test/testsuite/data)) 
+did not change compared to the references.
+
+You can launch the testsuite by writing `launch jenkins` as a comment in the 
+PR that you want to test. Once completed, the result of the testsuite will be 
+shown on the PR (failure or success).
+
+If you need more details on the testsuite results (e.g., if you are trying to 
+debug an error or you are simply unsure why the tests fail) you can launch the 
+testsuite with `launch jenkins(debug)`. This will run the tests as usual, but, 
+once completed, you will be given a URL (via a comment on the PR) to access the 
+logfiles and namelists of all tests that were run.

docs/user_manual/user_manual_04_python_modules.md

@@ -153,7 +153,8 @@ The data processing with CDO for it_cl_type = 2 involves 5 steps:
     *distance-weighted average* interpolation.
 
 5.  Correct data processed in step 4 with the surface height extracted
-    in step 3.
+    in step 3. The lapse rate and offset can be selected via the 
+    `tcorr_lapse_rate` and `tcorr_offset` entries in the namelist.
 
 All subsequent processing on the data follows the general workflow of
 the Python scripts.

docs/user_manual/user_manual_06_namelist_input.md

@@ -179,7 +179,9 @@ The COSMO grid is defined by a rotated latlon-grid.
 | `raw_data_t_clim_path`         | character |            |         | path to T2m climatology data |
 | `raw_data_t_clim_coarse`       | character |            |         | filename of coarse T2m climatology data |
 | `raw_data_t_clim_fine`         | character |            |         | filename of fine T2m climatology data |
-| `it_cl_type`                   | integer   |            |         | switch to choose between the <br> 1: new and fine <br> 2: the old and coarse over sea and the fine over land raw data set. <br> Note that the fine data set (1) is topographically corrected. |
+| `it_cl_type`                   | integer   |            |         | switch to choose between the <br> 1: Aggregates the fine data over land, sea points are not considered. No lapse rate correction of soil temperature based on difference between CRU and ICON model topography <br> 2: Aggregates the coarse data over sea and the fine data over land. Lapse rate correction takes place (lapse rate and additional temperature offset can be specified with `tcorr_lapse_rate` and `tcorr_offset`). |
+| `tcorr_lapse_rate`             | real      | 0.0065     |         | Lapse rate used for correction of the climatological 2m temperature. **Only available for `it_cl_type=2`** |
+| `tcorr_offset`                 | real      | 0.0        |         | Offset used for correction of the climatological 2m temperature. **Only available for `it_cl_type=2`** |
 | `t_clim_buffer_file`           | character |            |         | name for T_clim buffer file |
 
 ## NDVI Data

python/WrapExtpar.py

@@ -85,14 +85,16 @@ def main():
     lradtopo = config.get('lradtopo', False)
     nhori = config.get('nhori', 24)
     radtopo_radius = config.get('radtopo_radius', 40000.0)
+    tcorr_lapse_rate = config.get('tcorr_lapse_rate', 0.0065)
+    tcorr_offset = config.get('tcorr_offset', 0.0)
 
     generate_external_parameters(
         igrid_type, args.input_grid, iaot_type, ilu_type, ialb_type,
         isoil_type, itopo_type, it_cl_type, iera_type, iemiss_type,
         ilookup_table_lu, enable_cdnc, enable_edgar, enable_art,
-        use_array_cache, nhori, radtopo_radius, args.raw_data_path,
-        args.run_dir, args.account, args.host, args.no_batch_job, lurban,
-        lsgsl, lfilter_oro, l_use_corine, lradtopo)
+        use_array_cache, nhori, radtopo_radius, tcorr_lapse_rate, tcorr_offset,
+        args.raw_data_path, args.run_dir, args.account, args.host,
+        args.no_batch_job, lurban, lsgsl, lfilter_oro, l_use_corine, lradtopo)
 
 
 def generate_external_parameters(igrid_type,
@@ -112,6 +114,8 @@ def generate_external_parameters(igrid_type,
                                  use_array_cache,
                                  nhori,
                                  radtopo_radius,
+                                 tcorr_lapse_rate,
+                                 tcorr_offset,
                                  raw_data_path,
                                  run_dir,
                                  account,
@@ -151,6 +155,8 @@ def generate_external_parameters(igrid_type,
         'lradtopo': lradtopo,
         'nhori': nhori,
         'radtopo_radius': radtopo_radius,
+        'tcorr_lapse_rate': tcorr_lapse_rate,
+        'tcorr_offset': tcorr_offset,
         'lsgsl': lsgsl,
         'lfilter_oro': lfilter_oro,
         'lurban': lurban,
@@ -426,7 +432,6 @@ def setup_oro_namelist_icon(args, lonmax, lonmin, latmax, latmin):
         namelist['ntiles_row'] = 4
 
         namelist['lscale_separation'] = ".FALSE."
-        namelist['lsso_param'] = ".FALSE."
         namelist['scale_sep_files'] = "'placeholder_file'"
         namelist['lsso_param'] = ".TRUE."
 
@@ -587,11 +592,15 @@ def setup_tclim_namelist(args):
     namelist['t_clim_buffer_file'] = 'tclim_buffer.nc'
 
     namelist['it_cl_type'] = args['it_cl_type']
-    namelist['raw_data_tclim_coarse'] = 'absolute_hadcrut3.nc'
-    namelist['raw_data_tclim_fine'] = 'CRU_T_SOIL_clim.nc'
     if args['it_cl_type'] > 2:
         raise ValueError(f'Unknown it_cl_type {args["it_cl_type"]}')
 
+    namelist['raw_data_tclim_coarse'] = 'absolute_hadcrut3.nc'
+    namelist['raw_data_tclim_fine'] = 'CRU_T_SOIL_clim.nc'
+
+    namelist['tcorr_lapse_rate'] = args['tcorr_lapse_rate']
+    namelist['tcorr_offset'] = args['tcorr_offset']
+
     return namelist
 
 

python/extpar_cru_to_buffer.py

@@ -180,16 +180,23 @@
                        'smooth,maxpoints=16', '-setmisstonn',
                        f'-remapdis,{grid}', step2_cdo, step4_cdo)
 
+    tcorr_lapse_rate = utils.check_tcorr_lapse_rate(
+        itcl.get('tcorr_lapse_rate', 0.0065))
+    tcorr_offset = utils.check_tcorr_offset(itcl.get('tcorr_offset', 0.0))
+
     logging.info(f'STEP 5: '
                  f'correct T_CL from {step4_cdo} '
                  f'with HH_TOPO from {buffer_topo} '
+                 f'with temperature lapse rate of {tcorr_lapse_rate} K/m '
+                 f'and offset of {tcorr_offset} K for land points '
                  f'--> {step5_cdo}')
     logging.info('')
 
     utils.launch_shell(
         'cdo', lock, 'expr, T_CL = ((FR_LAND != 0.0)) ? '
-        'T_CL+0.0065*(HSURF-HH_TOPO) : T_CL; HSURF;', '-merge', step4_cdo,
-        step3_cdo, step5_cdo)
+        f'T_CL+{tcorr_lapse_rate}*(HSURF-HH_TOPO){tcorr_offset:+} : '
+        f'T_CL{tcorr_offset:+}; HSURF;', '-merge', step4_cdo, step3_cdo,
+        step5_cdo)
 else:
 
     logging.info('STEP 1: '

python/lib/utilities.py

@@ -280,7 +280,43 @@ def check_itype_cru(itype_cru):
         logging.info('Process fine resolution for land, '
                      'coarse resolution for sea')
 
-        return itype_cru
+    return itype_cru
+
+
+def check_tcorr_lapse_rate(tcorr_lapse_rate):
+    '''
+    check tcorr_lapse_rate for correctness and return value,
+    if not exit programme
+    '''
+
+    if (tcorr_lapse_rate > 0.0098 or tcorr_lapse_rate < 0.005):
+        logging.error(f'Invalid value entered for temperature lapse rate. '
+                      f'Use a value between 0.005 and 0.0098 K/m instead!')
+
+        raise ValueError(f'Invalid value entered for temperature lapse rate. '
+                         f'Use a value between 0.005 and 0.0098 K/m instead!')
+
+    logging.info(f'Temperate lapse rate is set to {tcorr_lapse_rate} K/m')
+
+    return tcorr_lapse_rate
+
+
+def check_tcorr_offset(tcorr_offset):
+    '''
+    check tcorr_offset for correctness and return value,
+    if not exit programme
+    '''
+
+    if (tcorr_offset > 10.0 or tcorr_offset < -10.0):
+        logging.error(f'Invalid value entered for temperature offset. '
+                      f'Use a value between -10.0 and +10.0 K instead!')
+
+        raise ValueError(f'Invalid value entered for temperature offset. '
+                         f'Use a value between -10.0 and +10.0 K instead!')
+
+    logging.info(f'Chosen temperature offset: {tcorr_offset} K')
+
+    return tcorr_offset
 
 
 def determine_emiss_varnames(iemiss_type):

src/extpar_consistency_check.f90

@@ -497,6 +497,8 @@ PROGRAM extpar_consistency_check
 
 
   REAL (KIND=wp)                                :: t2mclim_hc, &
+       &                                           tcorr_lapse_rate, &
+       &                                           tcorr_offset, &
        &                                           step, &
        &                                           thr_cr, & !< control threshold
        &                                           fill_value_real, & !< value to indicate undefined grid elements
@@ -805,8 +807,10 @@ PROGRAM extpar_consistency_check
        it_cl_type,               &
        raw_data_t_clim_path,     &
        raw_data_t_clim_filename, &
-       t_clim_buffer_file      , &
-       t_clim_output_file        )
+       t_clim_buffer_file,       &
+       t_clim_output_file,       &
+       tcorr_lapse_rate,         &
+       tcorr_offset              )
 
   ! read namelist for input EMISS data
   namelist_file = 'INPUT_EMISS'

src/mo_python_routines.f90

@@ -137,21 +137,26 @@ SUBROUTINE read_namelists_extpar_t_clim(namelist_file,            &
        &                                  raw_data_t_clim_path,     &
        &                                  raw_data_t_clim_filename, &
        &                                  t_clim_buffer_file,       &
-       &                                  t_clim_output_file)
+       &                                  t_clim_output_file,       &
+       &                                  tcorr_lapse_rate,         &
+       &                                  tcorr_offset)
 
     CHARACTER (len=*), INTENT(IN)             :: namelist_file !< filename with namelists for for EXTPAR settings
     INTEGER (KIND=i4), INTENT(OUT)            :: it_cl_type    !< integer switch to choose a land use raw data set
 
-    CHARACTER (len=filename_max), INTENT(OUT) :: raw_data_t_clim_path, &        !< path to raw data
-         &                                       raw_data_t_clim_filename, &    !< filename temperature climatology raw data
-         &                                       t_clim_buffer_file, & !< name for temperature climatology buffer
-         &                                       t_clim_output_file !< name for temperature climatology output file
+    CHARACTER (len=filename_max), INTENT(OUT) :: raw_data_t_clim_path, &     !< path to raw data
+         &                                       raw_data_t_clim_filename, & !< filename temperature climatology raw data
+         &                                       t_clim_buffer_file, &       !< name for temperature climatology buffer
+         &                                       t_clim_output_file          !< name for temperature climatology output file
+    
+    REAL(KIND=wp), INTENT(OUT) :: tcorr_lapse_rate, & !< lapse rate for temperature correction
+         &                        tcorr_offset        !< offset for temperature correction
 
 
     INTEGER (KIND=i4)                         :: nuin, ierr
 
     !> namelist with filename for temperature climatlogy data output
-    NAMELIST /t_clim_raw_data/ raw_data_t_clim_path, raw_data_t_clim_filename, it_cl_type
+    NAMELIST /t_clim_raw_data/ raw_data_t_clim_path, raw_data_t_clim_filename, it_cl_type, tcorr_lapse_rate, tcorr_offset
 
     !> namelist with filename for temperature climatlogy data output
     NAMELIST /t_clim_io_extpar/ t_clim_buffer_file, t_clim_output_file

templates/namelist

@@ -3,7 +3,9 @@ input_tclim = {
         'raw_data_tclim_coarse': '@RAW_DATA_TCLIM_COARSE@',
         'raw_data_tclim_fine': '@RAW_DATA_TCLIM_FINE@',
         't_clim_buffer_file': '@T_CLIM_BUFFER_FILE@',
-        'it_cl_type': @IT_CL_TYPE@
+        'it_cl_type': @IT_CL_TYPE@,
+        'tcorr_lapse_rate': @TCORR_LAPSE_RATE@,
+        'tcorr_offset': @TCORR_OFFSET@
         }
 
 input_alb = {

test/jenkins/test.sh

@@ -2,8 +2,38 @@
 case "$(hostname)" in
     *co2* | *iacdipl-7*)
         set -e
-        podman run -e OMP_NUM_THREADS=16 -v /net/co2/c2sm-data/extpar-input-data:/data extpar:$ghprbPullId bash -c /workspace/test/jenkins/test_docker.sh || (podman image rm -f extpar:$ghprbPullId && exit 1)
-        podman image rm -f extpar:$ghprbPullId
-        exit 0
+
+        trap "podman image rm -f extpar:$ghprbPullId || true" EXIT
+
+        MODE=${1:-'NORMAL'}
+        if [[ ${MODE} == "DEBUG" ]]; then
+            CURRENT_DATETIME=$(date --iso-8601=seconds)
+            HASH=$(echo -n "${CURRENT_DATETIME}" | sha256sum | awk '{print $1}')
+            mkdir -p /net/co2/c2sm-services/extpar/${HASH}
+
+            if podman run \
+               -e OMP_NUM_THREADS=16 \
+               -v /net/co2/c2sm-data/extpar-input-data:/data \
+               -v /net/co2/c2sm-services/extpar/${HASH}:/artifacts \
+               extpar:$ghprbPullId \
+               bash -c "/workspace/test/jenkins/test_docker.sh ${MODE}"; then
+                REPORT_STATUS="completed successfully"
+                STATUS=0
+            else
+                REPORT_STATUS="failed"
+                STATUS=1
+            fi
+
+            curl -H "Authorization: token ${GITHUB_TOKEN}" \
+                 -H "Accept: application/vnd.github+json" \
+                 -H "Content-Type: application/json" \
+                 https://api.github.com/repos/C2SM/extpar/issues/${ghprbPullId}/comments \
+                 -d "{\"body\": \"The testsuite you launched for this PR has ${REPORT_STATUS}. You can inspect the results at this [link](https://data.iac.ethz.ch/extpar/${HASH}).\"}"
+        else
+            podman run -e OMP_NUM_THREADS=16 -v /net/co2/c2sm-data/extpar-input-data:/data extpar:$ghprbPullId bash -c "/workspace/test/jenkins/test_docker.sh ${MODE}"
+            STATUS="$?"
+        fi
+
+        exit "${STATUS}"
         ;;
 esac