Skip to content

Commit 725b855

Browse files
ehoganalistairsellar
ehogan
and
alistairsellar
authored
Add DKRZ as a site to RTW (#3837)
Co-authored-by: Alistair Sellar <[email protected]>
1 parent ba9cfe4 commit 725b855

File tree

10 files changed

+242
-11
lines changed

10 files changed

+242
-11
lines changed

‎doc/sphinx/source/utils/RTW/user_guide/quick_start.rst‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,5 +38,17 @@ Quick Start Guide
3838
export CYLC_VERSION=8
3939
cylc vip -O jasmin
4040

41-
* Browse the logs using `Cylc Review`_, a web service for browsing logs via an
42-
HTTP interface.
41+
* on DKRZ:
42+
43+
* add the following line to your ``~/.bashrc`` file to ensure the Cylc and
44+
Rose executables can be found::
45+
46+
export PATH=/work/bd0854/metomi/bin:$PATH
47+
48+
* run the RTW on DKRZ::
49+
50+
cd ESMValTool/esmvaltool/utils/recipe_test_workflow
51+
cylc vip -O dkrz
52+
53+
* Optionally browse the logs using `Cylc Review`_,
54+
a web service for browsing logs via an HTTP interface.

‎doc/sphinx/source/utils/RTW/user_guide/workflow.rst‎

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,7 @@ The |RTW| performs the following steps:
2525
GitHub, or gets the latest container image from DockerHub and converts to
2626
a singularity image, depending on ``SITE``
2727
:Runs on:
28-
Localhost (if cloning), or ``COMPUTE`` (if getting container), which
29-
depends on the ``SITE``; on JASMIN, the ``get_esmval`` jobs will run on
30-
LOTUS
28+
Localhost, or ``COMPUTE`` on JASMIN
3129
:Executes:
3230
The ``clone_latest_esmval.sh`` script (if cloning), or a
3331
``singularity build`` command (if getting container) from the |Rose| app
@@ -48,8 +46,7 @@ The |RTW| performs the following steps:
4846
:Description:
4947
Runs the requested recipes using |ESMValTool|
5048
:Runs on:
51-
``COMPUTE``, which depends on the ``SITE``; at the Met Office, the
52-
``process`` jobs will run on SPICE
49+
``COMPUTE``, which depends on the ``SITE``
5350
:Executes:
5451
The |ESMValTool| command line script from the |Rose| app
5552
:Details:
@@ -60,8 +57,7 @@ The |RTW| performs the following steps:
6057
:Description:
6158
Compares the output from the ``process`` job with |KGOs|
6259
:Runs on:
63-
``COMPUTE``, which depends on the ``SITE``; at the Met Office, the
64-
``compare`` jobs will run on SPICE
60+
``COMPUTE``, which depends on the ``SITE``
6561
:Executes:
6662
The :ref:`compare.py <compare_recipe_runs>` script from |ESMValTool|
6763
from the |Rose| app

‎esmvaltool/utils/recipe_test_workflow/app/get_esmval/opt/rose-app-dkrz.conf‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[command]
2+
default=singularity-env singularity build ${CONTAINER_PATH} ${DOCKER_SOURCE}
3+
4+
[env]
5+
DOCKER_SOURCE=docker://esmvalgroup/esmvaltool:${ENV_NAME}
6+
7+
[file:${CONTAINER_DIR}]
8+
mode=mkdir

‎esmvaltool/utils/recipe_test_workflow/app/install_env_file/opt/rose-app-dkrz.conf‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Copy the site specific environment files to the 'share/bin' directory in the
2+
# installed Cylc workflow (this directory is automatically added to the
3+
# ${PATH} by Cylc).
4+
5+
[command]
6+
default=BASH_XTRACEFD=1 # Send the output from 'set -x' to 'stdout' rather than 'stderr'.
7+
=set -euxo pipefail
8+
=cp ${ENV_FILE_SITE_PATH} ${SHARE_BIN_DIR}/${ENV_FILE}
9+
=cp ${SINGULARITY_ENV_FILE_SITE_PATH} ${SHARE_BIN_DIR}/${SINGULARITY_ENV_FILE}
10+
11+
[file:${SHARE_BIN_DIR}]
12+
mode=mkdir

‎esmvaltool/utils/recipe_test_workflow/flow.cylc‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,6 @@
5353
ROSE_APP_OPT_CONF_KEYS = {{ SITE }}
5454

5555
[[configure]]
56-
pre-script = "mkdir -p ${USER_CONFIG_DIR}"
5756
[[[environment]]]
5857
ROSE_TASK_APP = configure
5958
DRS_CMIP3 = {{ DRS_CMIP3 }}

‎esmvaltool/utils/recipe_test_workflow/opt/rose-suite-dkrz.conf‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[template variables]
2+
DRS_CMIP5="DKRZ"
3+
DRS_CMIP6="DKRZ"
4+
ENV_NAME="experimental"
5+
KGO_ROOT_PATH="/work/bd0854/KGO/esmvaltool_output/"
6+
ROOTPATH_CMIP5="/work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ"
7+
ROOTPATH_CMIP6="/work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ"
8+
ROOTPATH_OBS="/work/bd0854/DATA/ESMValTool2/OBS"
9+
ROOTPATH_OBS4MIPS="/work/bd0854/DATA/ESMValTool2/OBS"
10+
SITE="dkrz"

‎esmvaltool/utils/recipe_test_workflow/site/dkrz-env‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#!/bin/bash
2+
#
3+
# USAGE dkrz-env COMMAND
4+
#
5+
# ENVIRONMENT
6+
# ENV_NAME The name of the container being used
7+
# QUIET_MODE Don't print confirmation messages
8+
# PYTHONPATH_PREPEND The path to prepend to PYTHONPATH
9+
#
10+
# OPTIONS
11+
# COMMAND The command to execute with options
12+
set -eu
13+
14+
# Must be run before importing numpy, see
15+
# https://docs.dask.org/en/stable/array-best-practices.html#avoid-oversubscribing-threads
16+
export OMP_NUM_THREADS=1
17+
export OPENBLAS_NUM_THREADS=1
18+
export MKL_NUM_THREADS=1
19+
export VECLIB_MAXIMUM_THREADS=1
20+
export NUMEXPR_NUM_THREADS=1
21+
22+
# Ensure '~/.local' isn't added to 'sys.path'.
23+
export PYTHONNOUSERSITE=True
24+
25+
WORKFLOW_RUN_BIN_DIR="${CYLC_WORKFLOW_RUN_DIR}/bin"
26+
WORKFLOW_SHARE_BIN_DIR="${CYLC_WORKFLOW_SHARE_DIR}/cycle/bin"
27+
ROSE_APP_BIN_DIR="${CYLC_WORKFLOW_RUN_DIR}/app/${ROSE_TASK_APP:-$CYLC_TASK_NAME}/bin"
28+
29+
# Bind paths for container.
30+
export SINGULARITY_BIND="/home/b,/work,/scratch/b"
31+
32+
# Suppress an ESMValTool "file not found" warning.
33+
export SINGULARITYENV_PROJ_DATA="/opt/conda/envs/esmvaltool/share/proj"
34+
35+
# Provide mkfile needed to build esmfpy package.
36+
export SINGULARITYENV_ESMFMKFILE="/opt/conda/envs/esmvaltool/lib/esmf.mk"
37+
38+
# Ensure that `singularity exec` finds the right version of python.
39+
export SINGULARITYENV_PREPEND_PATH="/opt/conda/envs/esmvaltool/bin"
40+
41+
# Include Rose/Cylc workflow directories in container PATH.
42+
export SINGULARITYENV_APPEND_PATH="${WORKFLOW_RUN_BIN_DIR}:${WORKFLOW_SHARE_BIN_DIR}:${ROSE_APP_BIN_DIR}"
43+
44+
# If PYTHONPATH_PREPEND has been set, prepend it to PYTHONPATH to extend the
45+
# Python environment.
46+
if [[ ! -z ${PYTHONPATH_PREPEND:-} ]]; then
47+
echo "[INFO] Prepending the following to PYTHONPATH: ${PYTHONPATH_PREPEND}"
48+
export PYTHONPATH=${PYTHONPATH_PREPEND}:${PYTHONPATH:-}
49+
fi
50+
51+
if [[ -z ${QUIET_MODE:-} ]]; then
52+
echo "[INFO] Using the ${ENV_NAME} container"
53+
fi
54+
55+
singularity_command="singularity-env singularity -q exec ${CONTAINER_PATH} $@"
56+
command="/usr/bin/time -v -o ${CYLC_TASK_LOG_ROOT}.time ${singularity_command}"
57+
exec ${command}

‎esmvaltool/utils/recipe_test_workflow/site/dkrz-singularity-env‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/bin/bash
2+
#
3+
# USAGE dkrz-singularity-env COMMAND
4+
#
5+
# OPTIONS
6+
# COMMAND The command to execute with options
7+
#
8+
# Since DKRZ uses the module environment to dynamically modify a user's
9+
# environment via modulefiles (more details are available at
10+
# https://docs.dkrz.de/doc/levante/access-and-environment.html#module-environment)
11+
# this additional environment file is required to enable access to singularity
12+
# on DKRZ. This file is used directly in the 'get_esmval' app for DKRZ, as well
13+
# as in the 'dkrz-env' environment file.
14+
set -eu
15+
16+
module_count(){
17+
module list -t 2>&1 | wc -l
18+
}
19+
20+
safe_load(){
21+
PRE_LOAD_COUNT=$(module_count)
22+
23+
module load "${1}"
24+
# Check module count to determine whether module load was successful.
25+
26+
if (( PRE_LOAD_COUNT == $(module_count) )); then
27+
echo "[ERROR] Failed to load: ${1}"
28+
exit 1
29+
fi
30+
}
31+
safe_load "singularity"
32+
33+
command="/usr/bin/time -v -o ${CYLC_TASK_LOG_ROOT}.time $@"
34+
exec ${command}

‎esmvaltool/utils/recipe_test_workflow/site/dkrz.cylc‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
#!jinja2
2+
[runtime]
3+
[[root]]
4+
[[[environment]]]
5+
CONTAINER_DIR = ${ROSE_DATAC}/container
6+
CONTAINER_FILE = esmvaltool.sif
7+
CONTAINER_PATH = ${CONTAINER_DIR}/${CONTAINER_FILE}
8+
# Warning: fragile assumption of location of esmvaltool code,
9+
# issue #3437 contains more details.
10+
ESMVALTOOL_DIR = /opt/conda/envs/esmvaltool/lib/python3.12/site-packages
11+
12+
# COMPUTE provides defaults for computation-heavy tasks.
13+
# Specific tasks below override some defaults, e.g. time & memory.
14+
[[COMPUTE]]
15+
platform = levante
16+
execution time limit = PT3M
17+
[[[directives]]]
18+
--wckey = RTW
19+
--account = bk1088
20+
--partition = interactive
21+
--mem = 15G
22+
23+
[[install_env_file]]
24+
[[[environment]]]
25+
ROSE_APP_OPT_CONF_KEYS = {{ SITE }}
26+
SINGULARITY_ENV_FILE=singularity-env
27+
SINGULARITY_ENV_FILE_SITE_PATH=${CYLC_WORKFLOW_RUN_DIR}/site/${SITE}-singularity-env
28+
29+
[[get_esmval]]
30+
platform = localhost
31+
execution time limit = PT10M # Actual: 3m06s on 2024-12-18.
32+
[[[environment]]]
33+
# Move the location of the singularity cache from the user's
34+
# home area to the directory where the container is saved.
35+
# This variable must be specified only for the 'get_esmval'
36+
# task, otherwise, for some reason, subsequent steps take much
37+
# longer to run than the resources specified.
38+
SINGULARITY_CACHEDIR = ${CONTAINER_DIR}
39+
40+
[[configure]]
41+
platform = localhost
42+
execution time limit = PT2M # Actual: 0m14s on 2024-12-18.
43+
[[[directives]]]
44+
--mem = 2G
45+
46+
# Resources for recipes that need more than the default. Both time and
47+
# memory should be specified, in case the default changes.
48+
# Variable (fast, medium) must be consistent with flow.cylc.
49+
# Comment indicates example recorded usage on DKRZ.
50+
[[process_examples--recipe_python]]
51+
# Actual: 0m07s 2.1 GB on 2024-12-18.
52+
execution time limit = PT2M
53+
[[[directives]]]
54+
--mem = 15G
55+
56+
[[process_recipe_albedolandcover]]
57+
# Actual: 0m10s 3.2 GB on 2024-12-18.
58+
execution time limit = PT2M
59+
[[[directives]]]
60+
--mem = 15G
61+
62+
[[process_recipe_autoassess_landsurface_soilmoisture]]
63+
# Actual: 0m13s 3.8 GB on 2024-12-18.
64+
execution time limit = PT2M
65+
[[[directives]]]
66+
--mem = 15G
67+
68+
[[process_recipe_consecdrydays]]
69+
# Actual: 0m08s 2.3 GB on 2024-12-18.
70+
execution time limit = PT2M
71+
[[[directives]]]
72+
--mem = 15G
73+
74+
[[process_recipe_ensclus]]
75+
# Actual: 0m35s 2.0 GB on 2024-12-18.
76+
execution time limit = PT2M
77+
[[[directives]]]
78+
--mem = 15G
79+
80+
[[process_recipe_heatwaves_coldwaves]]
81+
# Actual: 0m30s 1.6 GB on 2024-12-18.
82+
execution time limit = PT2M
83+
[[[directives]]]
84+
--mem = 15G
85+
86+
[[process_recipe_ocean_amoc]]
87+
# Actual: Currently broken: No input files found for Dataset
88+
execution time limit = PT2M
89+
[[[directives]]]
90+
--mem = 15G
91+
92+
[[process_recipe_ocean_multimap]]
93+
# Actual: Currently broken: No input files found for Dataset
94+
execution time limit = PT2M
95+
[[[directives]]]
96+
--mem = 15G
97+
98+
[[process_recipe_radiation_budget]]
99+
# Actual: Currently broken: No input files found for Dataset
100+
execution time limit = PT2M
101+
[[[directives]]]
102+
--mem =15G

‎esmvaltool/utils/recipe_test_workflow/site/jasmin.cylc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
[runtime]
33
[[root]]
44
[[[environment]]]
5-
# Warning: fragile assumption of location of esmvaltool code, see #3437
5+
# Warning: fragile assumption of location of esmvaltool code,
6+
# issue #3437 contains more details.
67
ESMVALTOOL_DIR = /opt/conda/envs/esmvaltool/lib/python3.11/site-packages
78

89
# COMPUTE provides defaults for computation-heavy tasks.

0 commit comments

Comments
 (0)