Merge pull request #130 from ESMValGroup/fix_bugs

SarahAlidoost · web-flow · commit 108ed3136ddf · 2020-09-07T17:22:13.000+02:00
fixing missing data and some bugs
diff --git a/_episodes/02-installation.md b/_episodes/02-installation.md
@@ -1,7 +1,7 @@
 ---
 title: "Installation"
 teaching: 10
-exercises: 20
+exercises: 10
 questions:
 - "What are the prerequisites for installing ESMValTool?"
 - "How do I confirm that the installation was successful?"
diff --git a/_episodes/03-configuration.md b/_episodes/03-configuration.md
@@ -188,13 +188,15 @@ Site-specific entries for Jasmin, DKRZ and ETHZ are listed at the end of the
 example configuration file.
 
 In this lesson, we will work with data from
-[CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
+[CMIP5](https://esgf-node.llnl.gov/projects/cmip5/)
+and [obs4mips](https://esgf-node.llnl.gov/projects/obs4mips/).
 We add the root path of the folder where  our/your data is available.
 
 ```yaml
   rootpath:
   ...
     CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2, ~/esmvaltool_tutorial/data]
+    obs4mips: ~/esmvaltool_tutorial/data
 ```
 
 > ## Setting the correct rootpath
diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md
@@ -181,14 +181,24 @@ Please note the following sections:
     The documentation consists of the following information:
 
     - description: a short description of the recipe
-    - authors: a list of authors (linked to `esmvaltool/config-references.yml`)
-    - maintainer: a list of maintainers (linked to
-      `esmvaltool/config-references.yml`)
-    - references: a list of references (linked to a bibtexfile in
-      `esmvaltool/references` with the same name)
-    - projects: a list of projects (linked to
-      `esmvaltool/config-references.yml`)
+    - authors: a list of authors
+    - maintainer: a list of maintainers
+    - references: a list of references
+    - projects: a list of projects
 
+> ## config-references.yml and references
+>
+> The values for the keys ``author``, ``maintainer``, ``projects`` and
+> ``references`` in the recipe should be known by ESMValTool:
+>
+> - A list of ESMValTool
+> author, maintainer, and projects are mentioned in the file ``config-references.yml``
+> that can be found in
+> [ESMValTool/esmvaltool](https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/config-references.yml).
+>
+> - ESMValTool references in `BibTeX` format can be found in
+> [ESMValTool/esmvaltool/references](https://github.com/ESMValGroup/ESMValTool/tree/master/esmvaltool/references) directory.
+{: .callout}
 
   - datasets: lines 22-23
 
@@ -399,9 +409,9 @@ available) can be found below. Note that the timestamps differ.
 
 > ## Your output plot(s).
 > Plots for the dataset(s) are located in `./recipe_example_#_#/plots/timeseries_diag/`
-> 
+>
 > For a single dataset:
-> 
+>
 > ![single dataset](../fig/diag_CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_prep_timeseries_diag_timeseries_temperature_1859_2005_timeseries_0.png "single dataset")
 >
 > Or an overlay plot, if multiple datasets are defined:
@@ -410,8 +420,8 @@ available) can be found below. Note that the timestamps differ.
 {: .solution}
 
 > ## Your main output log file.
-> The main output file is located at `./recipe_example_#_#/run/main_log.txt/` 
-> 
+> The main output file is located at `./recipe_example_#_#/run/main_log.txt/`
+>
 > ~~~bash
 > INFO    [33433]
 > ______________________________________________________________________
@@ -499,7 +509,7 @@ available) can be found below. Note that the timestamps differ.
 
 > ##  Your settings.yml file.
 > This file is located at `./recipe_example_#_#/run/diag_timeseries_temperature/timeseries_diag/settings.yaml`.
-> 
+>
 > ```YAML
 > auxiliary_data_dir: /mnt/lustre01/pf/b/b380506/work/auxiliary_data
 > input_files:
@@ -519,9 +529,9 @@ available) can be found below. Note that the timestamps differ.
 {: .solution}
 
 > ## A metadata.yml file.
-> 
+>
 > This file is located at `./recipe_example_#_#/preproc/diag_timeseries_temperature/timeseries_variable/metadata.yml`.
-> 
+>
 > ```YAML
 > ? /scratch/b/b380506/recipe_example_20200701_082257/preproc/diag_timeseries_temperature/timeseries_variable/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_thetaoga_1859-2005.nc
 > : alias: HadGEM2-ES
@@ -552,7 +562,7 @@ available) can be found below. Note that the timestamps differ.
 
 > ## The diagnostic log file.
 > This file is located at `./recipe_example_#_#/run/diag_timeseries_temperature/timeseries_diag/log.txt`
-> 
+>
 > ~~~bash
 > Starting diagnostic script timeseries_diag with configuration:
 > auxiliary_data_dir: /mnt/lustre01/pf/b/b380506/work/auxiliary_data
diff --git a/_episodes/05-preprocessor.md b/_episodes/05-preprocessor.md
@@ -88,22 +88,46 @@ file afterwards. These do not need to be explicitly included in recipes.
 
 > ## Exercise: Adding more preprocessor steps
 >
-> Edit the [example recipe](https://github.com/ESMValGroup/ESMValTool_Tutorial/blob/master/data/recipe_example.yml) to first change the variable
-> thetao, then add preprocessors to average over the latitude and longitude
+> Edit the [example recipe](https://github.com/ESMValGroup/ESMValTool_Tutorial/blob/master/data/recipe_example.yml) to first change the variable to
+> `thetao`, then add preprocessors to average over the latitude and longitude
 > dimensions and finally average over the depth. Now run the recipe.
 >
 >> ## Solution
 >>
->>```yaml
->> preprocessors:
->>   prep_timeseries:
->>     annual_statistics:
->>       operator: mean
->>     area_statistics:
->>       operator: mean
->>     depth_integration:
->>```
+>> In the `diff` file below you will see the changes we have made to the file. The top 2 lines are the filenames and the lines like @@ -20,12 +20,15 @@ indicate the line numbers in the original and modified file, respectively. For more info on this format, see [here](https://en.wikipedia.org/wiki/Diff#Unified_format).
+>>
+>>```diff
+>> --- data/recipe_example.yml
+>> --- data/recipe_example_thetao.yml
+>> @@ -20,12 +20,15 @@
+>>      - ukesm
+>>
+>>  datasets:
+>> -  - {dataset: HadGEM2-ES, project: CMIP5, exp: historical, mip: Omon, ensemble: r1i1p1, start_year: 1859, end_year: 2005}
+>> +  - {dataset: HadGEM2-ES, project: CMIP5, exp: historical, mip: Omon, ensemble: r1i1p1, start_year: 2000, end_year: 2005}
+>>
+>>  preprocessors:
+>>    prep_timeseries: # For 0D fields
+>>      annual_statistics:
+>>        operator: mean
+>> +    area_statistics:
+>> +      operator: mean
+>> +    depth_integration:
+>>
+>>  diagnostics:
+>>    # --------------------------------------------------
+>> @@ -35,7 +38,7 @@
+>>      description: simple_time_series
+>>      variables:
+>>        timeseries_variable:
+>> -        short_name: thetaoga
+>> +        short_name: thetao
+>>          preprocessor: prep_timeseries
+>>      scripts:
+>>        timeseries_diag:
+>> ```
 >>
+>> Complete recipe can be downloaded as [recipe_example_thetao.yml](https://github.com/ESMValGroup/ESMValTool_Tutorial/blob/master/data/recipe_example_thetao.yml)
 >{: .solution}
 {: .challenge}
 
@@ -116,6 +140,9 @@ specific preprocessor which should be applied.
 > ## Example
 >
 > ```yaml
+>datasets:
+>  - {dataset: HadGEM2-ES, project: CMIP5, exp: historical, mip: Omon, ensemble: r1i1p1, start_year: 2000, end_year: 2005}
+>
 > preprocessors:
 >   prep_timeseries_1:
 >     annual_statistics:
@@ -152,6 +179,7 @@ specific preprocessor which should be applied.
 >         script: ocean/diagnostic_timeseries.py
 > ```
 >
+> Complete recipe can be downloaded as [recipe_example_thetao_thetaoga.yml](https://github.com/ESMValGroup/ESMValTool_Tutorial/blob/master/data/recipe_example_thetao_thetaoga.yml)
 {: .solution}
 
 >## Challenge : How to write a recipe with multiple preprocessors
@@ -271,7 +299,7 @@ specific preprocessor which should be applied.
 ## Adding different datasets for different variables
 
 Sometimes, we may want to include specific datasets only for certain variables.
-An example is when we use observations for two different variables in a
+An example is when we use observations for variables in a
 diagnostic. While the CMIP dataset details for the two variables may be common,
 the observations will likely not be so. It would be useful to know how to
 include different datasets for different variables. Here is an example of a
@@ -303,9 +331,8 @@ simple preprocessor and diagnostic setup for that:
 >         preprocessor: prep_regrid
 >         mip: Amon
 >         grid: gn  # can change for variables from the same model
->         start_year: 1970
->         end_year: 2000 # start and end years for a 30 year period,
->                        # we assume this is common and exists for all
+>         start_year: 1979
+>         end_year: 2000 # we assume this exists for all
 >                        # model and obs data
 >         additional_datasets:
 >           - {dataset: GPCP-SG, project: obs4mips, level: L3,
@@ -314,11 +341,8 @@ simple preprocessor and diagnostic setup for that:
 >         preprocessor: prep_regrid
 >         mip: Amon
 >         grid: gn  # can change for variables from the same model
->         start_year: 1970  # some 30 year period
+>         start_year: 1979
 >         end_year: 2000
->         additional_datasets:
->           - {dataset: HadCRUT4, project: OBS, type: ground,
->              version: 1, tier: 2}  # dataset specific to the temperature variable
 >     scripts: null
 > ```
 >
@@ -329,7 +353,6 @@ simple preprocessor and diagnostic setup for that:
 Variable grouping can be used to preprocess different clusters of data for the
 same variable. For instance, the example below illustrates how we can compute
 separate multimodel means for CMIP5 and CMIP6 data given the same variable.
-Additionally we can also preprocess observed data for evaluation.
 
 > ## Example
 >```yaml
@@ -375,12 +398,6 @@ Additionally we can also preprocess observed data for evaluation.
 >        end_year: 2005
 >        tag: TAS_CMIP5  # tag is optional if you are using these settings just once
 >        additional_datasets: *cmip5_datasets
->      tas_obs:
->        <<: *variable_settings
->        preprocessor: prep_obs
->        tag: TAS_OBS
->        additional_datasets:
->          - {dataset: HadCRUT4, project: OBS, type: ground, version: 1, tier: 2}
 >      tas_cmip6:
 >        <<: *variable_settings
 >        tag: TAS_CMIP6
@@ -392,24 +409,9 @@ Additionally we can also preprocess observed data for evaluation.
 
 You should be able to see the variables grouped under different subdirectories
 under your output preproc directory. The different groupings can be accessed in
-your diagnostic by selecting the key name of the field `variable_group`  such as `tas_cmip5`, `tas_cmip6` or `tas_obs`.
+your diagnostic by selecting the key name of the field `variable_group`  such as `tas_cmip5`, or `tas_cmip6`.
 
 > ## How to find what CMIP data is available?
 >
-> [CMIP5](https://pcmdi.llnl.gov/mips/cmip5/index.html) and
-> [CMIP6](https://pcmdi.llnl.gov/CMIP6/Guide/dataUsers.html) data obey the
-> [CF-conventions](http://cfconventions.org/). Available variables can be found
-> under the [CMIP5 data
-> request](https://pcmdi.llnl.gov/mips/cmip5/docs/standard_output.pdf?id=28) and
-> the [CMIP6 Data Request](http://clipc-services.ceda.ac.uk/dreq/index.html).
->
-> CMIP data is widely available via the Earth System Grid Federation
-> ([ESGF](https://esgf.llnl.gov/)) and is accessible to users either via
-> download from the ESGF portal or through the ESGF data nodes hosted by large
-> computing facilities (like [CEDA-Jasmin](https://esgf-index1.ceda.ac.uk/),
-> [DKRZ](https://esgf-data.dkrz.de/), etc). The ESGF also hosts observations for
-> Model Intercomparison Projects (obs4MIPs) and reanalyses data (ana4MIPs).
->
-> A full list of all CMIP named variables is available here:
-> [http://clipc-services.ceda.ac.uk/dreq/index/CMORvar.html](http://clipc-services.ceda.ac.uk/dreq/index/CMORvar.html).
+> Please see section "Access to CMIP and Observational data" in [Setup]({{ page.root }}{% link setup.md %}).
 {: .callout}
diff --git a/_episodes/06-debugging.md b/_episodes/06-debugging.md
@@ -214,19 +214,13 @@ attach the run/recipe_*.yml and run/main_log_debug.txt files from the output dir
 {: .error}
 
 The values for the keys ``author``, ``maintainer``, ``projects`` and
-``references`` in the recipe should be known by ESMValTool. A list of ESMValTool
-author, maintainer, references, and projects can be found in the
-``config-references.yml``. You could download this file by typing:
+``references`` in the recipe should be known by ESMValTool:
 
-~~~bash
-  esmvaltool config get_config_reference
-~~~
-
-It will be saved to: ``{HOME}/.esmvaltool/config-references.yml``. If you modify this file, please add the directory in the ``config-user.yml``:
-
-```yaml
-config_developer_file: {HOME}/.esmvaltool/config_reference.yml
-```
+- A list of ESMValTool
+author, maintainer, and projects can be found in the
+[config-references.yml](https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/config-references.yml).
+- ESMValTool references in `BibTeX` format can be found in the
+[ESMValTool/esmvaltool/references](https://github.com/ESMValGroup/ESMValTool/tree/master/esmvaltool/references) directory.
 
 > ## ESMValTool can’t locate the data
 >
@@ -300,7 +294,7 @@ esmvalcore._task.DiagnosticError: Cannot execute script 'diag_scripts/examples/d
 
 The script path should be relative to ``diag_scripts`` directory. It means that
 the script ``diagnostic_timeseries.py`` is located in
-``<path_to_esmvaltool>/diag_scripts/ocean/``. 
+``<path_to_esmvaltool>/diag_scripts/ocean/``.
 Alternatively, the script path can be an absolute path. To examine this, we can download the script from the ``ESMValTool`` repository:
 
 ```bash
diff --git a/data/dataset.urls b/data/dataset.urls
@@ -25,6 +25,8 @@ http://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/
 http://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-ES/historical/mon/ocean/Omon/r1i1p1/v20110916/tos/tos_Omon_HadGEM2-ES_historical_r1i1p1_185912-195911.nc
 http://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-ES/historical/mon/ocean/Omon/r1i1p1/v20110916/tos/tos_Omon_HadGEM2-ES_historical_r1i1p1_195912-200512.nc
 # For Working with preprocessors episode
+# Variable = thetao, model = HadGEM2-ES
+http://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-ES/historical/mon/ocean/Omon/r1i1p1/v20110916/thetao/thetao_Omon_HadGEM2-ES_historical_r1i1p1_199912-200512.nc
 # Model = UKESM1-0-LL, project = CMIP6, Experiment = historical, ensemble= r1i1p1f2, variable = thetao, pr and tas, 1970-2000
 http://esgf-data3.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Omon/thetao/gn/v20190627/thetao_Omon_UKESM1-0-LL_historical_r1i1p1f2_gn_200001-201412.nc
 http://esgf-data3.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Omon/thetao/gn/v20190627/thetao_Omon_UKESM1-0-LL_historical_r1i1p1f2_gn_195001-199912.nc
@@ -47,3 +49,5 @@ http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCC
 http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r2i1p2f1/Amon/tas/gn/v20190429/tas_Amon_CanESM5_historical_r2i1p2f1_gn_185001-201412.nc
 http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r3i1p2f1/Amon/tas/gn/v20190429/tas_Amon_CanESM5_historical_r3i1p2f1_gn_185001-201412.nc
 http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r4i1p2f1/Amon/tas/gn/v20190429/tas_Amon_CanESM5_historical_r4i1p2f1_gn_185001-201412.nc
+# Variable = pr, model = GPCP-SG, project= obs4mips
+https://dpesgf03.nccs.nasa.gov/thredds/fileServer/obs4MIPs/observations/NASA-GSFC/Obs-GPCP/GPCP/V2.2/atmos/pr/pr_GPCP-SG_L3_v2.2_197901-201312.nc
diff --git a/data/recipe_example_thetao.yml b/data/recipe_example_thetao.yml
@@ -0,0 +1,45 @@
+# ESMValTool
+# recipe_example.yml
+---
+documentation:
+  description: Demonstrate basic ESMValTool example
+
+  authors:
+    - demora_lee
+    - mueller_benjamin
+    - swaminathan_ranjini
+
+  maintainer:
+    - demora_lee
+
+  references:
+    - demora2018gmd
+    # Some plots also appear in ESMValTool paper 2.
+
+  projects:
+    - ukesm
+
+datasets:
+  - {dataset: HadGEM2-ES, project: CMIP5, exp: historical, mip: Omon, ensemble: r1i1p1, start_year: 2000, end_year: 2005}
+
+preprocessors:
+  prep_timeseries: # For 0D fields
+    annual_statistics:
+      operator: mean
+    area_statistics:
+      operator: mean
+    depth_integration:
+
+diagnostics:
+  # --------------------------------------------------
+  # Time series diagnostics
+  # --------------------------------------------------
+  diag_timeseries_temperature:
+    description: simple_time_series
+    variables:
+      timeseries_variable:
+        short_name: thetao
+        preprocessor: prep_timeseries
+    scripts:
+      timeseries_diag:
+        script: ocean/diagnostic_timeseries.py
diff --git a/data/recipe_example_thetao_thetaoga.yml b/data/recipe_example_thetao_thetaoga.yml
diff --git a/setup.md b/setup.md
