Skip to content

Commit 4be8dc7

Browse files
Peter9192Peter9192
authored
Merge pull request #34 from ESMValGroup/origin/first_example_recipe
"episode 4" -- first example recipe
2 parents 362b23f + edb424e commit 4be8dc7

10 files changed

+911
-53
lines changed

‎_episodes/03-configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@ resource_usage.txt and temporary files created by the diagnostic scripts.
179179
are not plots, e.g. files in NetCDF format (depends on the diagnostic script).
180180
>
181181
> We explain more about output in the next
182-
[lesson]({{ page.root }}{% link _episodes/04-toy-example.md %})
182+
[lesson]({{ page.root }}{% link _episodes/04-recipe.md %})
183183
{: .callout}
184184
185185
## Auxiliary data directory

‎_episodes/04-recipe.md‎

Lines changed: 697 additions & 0 deletions
Large diffs are not rendered by default.

‎_episodes/04-toy-example.md‎

Lines changed: 0 additions & 52 deletions
This file was deleted.

‎data/recipe_example.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# ESMValTool
2+
# recipe_example.yml
3+
---
4+
documentation:
5+
description: Demonstrate basic ESMValTool example
6+
7+
authors:
8+
- demora_lee
9+
- mueller_benjamin
10+
- swaminathan_ranjini
11+
12+
maintainer:
13+
- demora_lee
14+
15+
references:
16+
- demora2018gmd
17+
# Some plots also appear in ESMValTool paper 2.
18+
19+
projects:
20+
- ukesm
21+
22+
datasets:
23+
- {dataset: HadGEM2-ES, project: CMIP5, exp: historical, mip: Omon, ensemble: r1i1p1, start_year: 1859, end_year: 2005}
24+
25+
preprocessors:
26+
prep_timeseries: # For 0D fields
27+
annual_statistics:
28+
operator: mean
29+
30+
diagnostics:
31+
# --------------------------------------------------
32+
# Time series diagnostics
33+
# --------------------------------------------------
34+
diag_timeseries_temperature:
35+
description: simple_time_series
36+
variables:
37+
timeseries_variable:
38+
short_name: thetaoga
39+
preprocessor: prep_timeseries
40+
scripts:
41+
timeseries_diag:
42+
script: ocean/diagnostic_timeseries.py

‎fig/MultipleModels__thetaoga_prep_timeseries_diag_timeseries_temperature_1859_2005_timeseries_.png‎

32.7 KB
Loading

‎fig/diag_CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_prep_timeseries_diag_timeseries_temperature_1859_2005_timeseries_0.png‎

32.2 KB
Loading

‎files/recipe_example_output/log.txt‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
Starting diagnostic script timeseries_diag with configuration:
2+
auxiliary_data_dir: /mnt/lustre01/pf/b/b380506/work/auxiliary_data
3+
input_data:
4+
? /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_1859-2005.nc
5+
: alias: HadGEM2-ES
6+
dataset: HadGEM2-ES
7+
diagnostic: diag_timeseries_temperature
8+
end_year: 2005
9+
ensemble: r1i1p1
10+
exp: historical
11+
filename: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_1859-2005.nc
12+
frequency: mon
13+
institute:
14+
- INPE
15+
- MOHC
16+
long_name: Global Average Sea Water Potential Temperature
17+
mip: Omon
18+
modeling_realm:
19+
- ocean
20+
preprocessor: prep_timeseries
21+
project: CMIP5
22+
recipe_dataset_index: 0
23+
short_name: thetaoga
24+
standard_name: sea_water_potential_temperature
25+
start_year: 1859
26+
units: K
27+
variable_group: timeseries_variable
28+
input_files:
29+
- /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/metadata.yml
30+
log_level: info
31+
output_file_type: png
32+
plot_dir: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag
33+
profile_diagnostic: false
34+
recipe: recipe_example.yml
35+
run_dir: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/diag_timeseries_temperature/timeseries_diag
36+
script: timeseries_diag
37+
version: 2.0.0b9
38+
work_dir: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/work/diag_timeseries_temperature/timeseries_diag
39+
write_netcdf: true
40+
write_plots: true
41+
42+
Creating /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/work/diag_timeseries_temperature/timeseries_diag
43+
Creating /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag
44+
metadata filename: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/metadata.yml
45+
No handles with labels found to put in legend.
46+
Image path will be: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag/MultipleModels__thetaoga_prep_timeseries_diag_timeseries_temperature_1859_2005_timeseries_.png
47+
Saving plots to /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag/MultipleModels__thetaoga_prep_timeseries_diag_timeseries_temperature_1859_2005_timeseries_.png
48+
-----------------
49+
model filenames: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_1859-2005.nc
50+
Image path will be: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag/diag_CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_prep_timeseries_diag_timeseries_temperature_1859_2005_timeseries_0.png
51+
Saving plots to /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag/diag_CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_prep_timeseries_diag_timeseries_temperature_1859_2005_timeseries_0.png
52+
Success
53+
End of diagnostic script run.

‎files/recipe_example_output/main_log.txt‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
INFO [33433]
2+
______________________________________________________________________
3+
_____ ____ __ ____ __ _ _____ _
4+
| ____/ ___|| \/ \ \ / /_ _| |_ _|__ ___ | |
5+
| _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| |
6+
| |___ ___) | | | | \ V / (_| | | | | (_) | (_) | |
7+
|_____|____/|_| |_| \_/ \__,_|_| |_|\___/ \___/|_|
8+
______________________________________________________________________
9+
10+
ESMValTool - Earth System Model Evaluation Tool.
11+
12+
http://www.esmvaltool.org
13+
14+
CORE DEVELOPMENT TEAM AND CONTACTS:
15+
Veronika Eyring (PI; DLR, Germany - [email protected])
16+
Bouwe Andela (NLESC, Netherlands - [email protected])
17+
Bjoern Broetz (DLR, Germany - [email protected])
18+
Lee de Mora (PML, UK - [email protected])
19+
Niels Drost (NLESC, Netherlands - [email protected])
20+
Nikolay Koldunov (AWI, Germany - [email protected])
21+
Axel Lauer (DLR, Germany - [email protected])
22+
Benjamin Mueller (LMU, Germany - [email protected])
23+
Valeriu Predoi (URead, UK - [email protected])
24+
Mattia Righi (DLR, Germany - [email protected])
25+
Manuel Schlund (DLR, Germany - [email protected])
26+
Javier Vegas-Regidor (BSC, Spain - [email protected])
27+
Klaus Zimmermann (SMHI, Sweden - [email protected])
28+
29+
For further help, please read the documentation at
30+
http://esmvaltool.readthedocs.io. Have fun!
31+
32+
INFO [33433] Using config file /pf/b/b380506/work/config-BM_DKRZ.yml
33+
INFO [33433] Writing program log files to:
34+
/scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/main_log.txt
35+
/scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/main_log_debug.txt
36+
INFO [33433] Starting the Earth System Model Evaluation Tool v2.0.0b9 at time: 2020-07-01 08:22:58 UTC
37+
INFO [33433] ----------------------------------------------------------------------
38+
INFO [33433] RECIPE = /pf/b/b380506/work/recipes/recipe_example.yml
39+
INFO [33433] RUNDIR = /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run
40+
INFO [33433] WORKDIR = /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/work
41+
INFO [33433] PREPROCDIR = /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc
42+
INFO [33433] PLOTDIR = /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots
43+
INFO [33433] ----------------------------------------------------------------------
44+
INFO [33433] Running tasks using at most 8 processes
45+
INFO [33433] If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory.
46+
INFO [33433] If you experience memory problems, try reducing 'max_parallel_tasks' in your user configuration file.
47+
INFO [33433] Creating tasks from recipe
48+
INFO [33433] Creating tasks for diagnostic diag_timeseries_temperature
49+
INFO [33433] Creating preprocessor task diag_timeseries_temperature/timeseries_variable
50+
INFO [33433] Creating preprocessor 'prep_timeseries' task for variable 'thetaoga'
51+
INFO [33433] Using input files for variable thetaoga of dataset HadGEM2-ES:
52+
/mnt/lustre01/work/kd0956/CMIP5/data/cmip5/output1/MOHC/HadGEM2-ES/historical/mon/ocean/Omon/r1i1p1/v20110916/thetaoga/thetaoga_Omon_HadGEM2-ES_historical_r1i1p1_185912-200512.nc
53+
INFO [33433] PreprocessingTask diag_timeseries_temperature/timeseries_variable created. It will create the files:
54+
/scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_1859-2005.nc
55+
INFO [33433] Creating diagnostic task diag_timeseries_temperature/timeseries_diag
56+
INFO [33433] These tasks will be executed: diag_timeseries_temperature/timeseries_variable, diag_timeseries_temperature/timeseries_diag
57+
INFO [33433] Running 2 tasks using 2 processes
58+
INFO [39196] Starting task diag_timeseries_temperature/timeseries_variable in process [39196]
59+
INFO [33433] Progress: 1 tasks running, 1 tasks waiting for ancestors, 0/2 done
60+
INFO [39196] Successfully completed task diag_timeseries_temperature/timeseries_variable (priority 0) in 0:00:07.241490
61+
INFO [33433] Progress: 0 tasks running, 1 tasks waiting for ancestors, 1/2 done
62+
INFO [39197] Starting task diag_timeseries_temperature/timeseries_diag in process [39197]
63+
INFO [39197] Running command ['/pf/b/b380506/miniconda3/envs/esmvaltool_pub/bin/python', '/mnt/lustre01/pf/b/b380506/work/GIT/ESMValTool/esmvaltool/diag_scripts/ocean/diagnostic_timeseries.py', '/scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/diag_timeseries_temperature/timeseries_diag/settings.yml']
64+
INFO [39197] Writing output to /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/work/diag_timeseries_temperature/timeseries_diag
65+
INFO [39197] Writing plots to /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag
66+
INFO [39197] Writing log to /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/diag_timeseries_temperature/timeseries_diag/log.txt
67+
INFO [39197] To re-run this diagnostic script, run:
68+
cd /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/diag_timeseries_temperature/timeseries_diag; MPLBACKEND="Agg" /pf/b/b380506/miniconda3/envs/esmvaltool_pub/bin/python /mnt/lustre01/pf/b/b380506/work/GIT/ESMValTool/esmvaltool/diag_scripts/ocean/diagnostic_timeseries.py /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/diag_timeseries_temperature/timeseries_diag/settings.yml
69+
INFO [33433] Progress: 1 tasks running, 0 tasks waiting for ancestors, 1/2 done
70+
INFO [39197] Maximum memory used (estimate): 0.2 GB
71+
INFO [39197] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
72+
WARNING [39197] No provenance information was written to /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/diag_timeseries_temperature/timeseries_diag/diagnostic_provenance.yml
73+
INFO [39197] Successfully completed task diag_timeseries_temperature/timeseries_diag (priority 1) in 0:00:07.347165
74+
INFO [33433] Progress: 0 tasks running, 0 tasks waiting for ancestors, 2/2 done
75+
INFO [33433] Successfully completed all tasks.
76+
INFO [33433] Ending the Earth System Model Evaluation Tool v2.0.0b9 at time: 2020-07-01 08:23:13 UTC
77+
INFO [33433] Time for running the recipe was: 0:00:14.820802
78+
INFO [33433] Maximum memory used (estimate): 0.7 GB
79+
INFO [33433] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
80+
INFO [33433] Run was successful

‎files/recipe_example_output/metadata.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
? /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_1859-2005.nc
2+
: alias: HadGEM2-ES
3+
dataset: HadGEM2-ES
4+
diagnostic: diag_timeseries_temperature
5+
end_year: 2005
6+
ensemble: r1i1p1
7+
exp: historical
8+
filename: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_1859-2005.nc
9+
frequency: mon
10+
institute:
11+
- INPE
12+
- MOHC
13+
long_name: Global Average Sea Water Potential Temperature
14+
mip: Omon
15+
modeling_realm:
16+
- ocean
17+
preprocessor: prep_timeseries
18+
project: CMIP5
19+
recipe_dataset_index: 0
20+
short_name: thetaoga
21+
standard_name: sea_water_potential_temperature
22+
start_year: 1859
23+
units: K
24+
variable_group: timeseries_variable

‎files/recipe_example_output/settings.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
auxiliary_data_dir: /mnt/lustre01/pf/b/b380506/work/auxiliary_data
2+
input_files:
3+
- /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/metadata.yml
4+
log_level: info
5+
output_file_type: png
6+
plot_dir: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/plots/diag_timeseries_temperature/timeseries_diag
7+
profile_diagnostic: false
8+
recipe: recipe_example.yml
9+
run_dir: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/run/diag_timeseries_temperature/timeseries_diag
10+
script: timeseries_diag
11+
version: 2.0.0b9
12+
work_dir: /scratch/b/b380506/ESACCISST_test/recipe_example_20200701_082257/work/diag_timeseries_temperature/timeseries_diag
13+
write_netcdf: true
14+
write_plots: true

0 commit comments

Comments
 (0)