Merge pull request #97 from ESMValGroup/bouwe

Fix links and add some clarification to configuration and recipe running episodes

Author: sverhoeven

_episodes/03-configuration.md

@@ -20,9 +20,9 @@ keypoints:
 ## The configuration file
 
 The ``config-user.yml`` configuration file contains all the global level
-information needed by ESMValTool to run. This is an [YAML
-file](https://yaml.org/spec/1.2/spec.html). An example configuration file can be
-found in the ESMValCore repository:
+information needed by ESMValTool to run.
+This is a [YAML file](https://yaml.org/spec/1.2/spec.html).
+An example configuration file can be found in the ESMValCore repository:
 [config-user-example.yml](https://github.com/ESMValGroup/ESMValCore/blob/master/esmvalcore/config-user.yml).
 
 You can generate the default configuration file by running:
@@ -31,14 +31,15 @@ You can generate the default configuration file by running:
   esmvaltool config get_config_user
 ~~~
 
-It will save the file to: ``{HOME}/.esmvaltool/config-user.yml``.
+It will save the file to: `~/.esmvaltool/config-user.yml`, where `~` is the
+path to your home directory. Note that files and directories starting with a
+period are "hidden", to see the `.esmvaltool` directory in the terminal use
+`ls -la ~`.
 
-Now, let's change our working directory in a terminal window to
-``{HOME}/.esmvaltool``. Then, we run a text editor called Nano to have a look
-inside the configuration file:
+We run a text editor called Nano to have a look inside the configuration file:
 
 ~~~bash
-  nano config-user.yml
+  nano ~/.esmvaltool/config-user.yml
 ~~~
 
 This file contains the information for:
@@ -91,12 +92,14 @@ remove_preproc_dir: true
 
 # Path to custom config-developer file, to customise project configurations.
 # See config-developer.yml for an example. Set to [null] to use the default
- config_developer_file: null
+config_developer_file: null
 # Get profiling information for diagnostics
 # Only available for Python diagnostics
 profile_diagnostic: false
 ```
 
+In general there is no need to change the settings listed above.
+
 ## Destination directory
 
 The destination directory is the rootpath where ESMValTool will store its output folders containing
@@ -169,7 +172,7 @@ for input data based on their source. The current categories in the configuratio
 file are mentioned below. For example, CMIP is used for a dataset from
 the climate model intercomparison project whereas OBS is used for an observational dataset.
 We can find more information about the projects in the ESMValTool
-[documentation](https://docs.esmvaltool.org/en/latest/input.html).
+[documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html).
 The ``rootpath`` specifies the directories where ESMValTool will look for input data.
 For each category, you can define either one path or several paths as a list.
 
@@ -181,7 +184,8 @@ rootpath:
   default: ~/default_inputpath
   CORDEX: ~/default_inputpath
 ```
-Site-specific entries for Jasmin, DKRZ and ETHZ are listed at the end of the example configuration file. 
+Site-specific entries for Jasmin, DKRZ and ETHZ are listed at the end of the
+example configuration file.
 
 In this lesson, we will work with data from
 [CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
@@ -195,10 +199,10 @@ We add the root path of the folder where  our/your data is available.
 
 > ## Setting the correct rootpath
 >
-> - To get the data (or its correct rootpath), check instruction in [Setup]({{
->   page.root }}{% link setup.md %}).
+> - To get the data (or its correct rootpath), check instruction in
+>   [Setup]({{ page.root }}{% link setup.md %}).
 > - For more information about setting the rootpath, see also the ESMValTool
->   [documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
+>   [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
 {: .callout}
 
 ## Directory structure for the data from different projects
@@ -225,7 +229,7 @@ drs:
 > ## Make your own configuration file
 >
 > It is possible to have several configuration files with different purposes,
-> for example: config-user_formalised_runs.yml, config-user_debugging.yml 
+> for example: config-user_formalised_runs.yml, config-user_debugging.yml
 {: .callout}
 
 > ## Saving preprocessed data
@@ -235,11 +239,13 @@ drs:
 >
 >> ## Solution
 >>
-> > If the option ``save_intermediary_cubes`` is set to true in the
-> > config-user.yml file, then the intermediary cubes will also be saved in the
-> > folder ``preproc``. Also, if the option ``remove_preproc_dir`` is set to
-> > ``false``, then the ``preproc/`` directory contains all the preprocessed
-> > data and the metadata interface files.
+> > If the option ``remove_preproc_dir`` is set to ``false``, then the
+> > ``preproc/`` directory contains all the pre-processed data and the
+> > metadata interface files.
+> > If the option ``save_intermediary_cubes`` is set to ``true``
+> > then data will also be saved after each preprocessor step in the folder
+> > ``preproc``. Note that saving all intermediate results to file will result
+> > in a considerable slowdown.
 > {: .solution}
 {: .challenge}
 

_episodes/04-recipe.md

@@ -64,37 +64,50 @@ diagnostics and description.
 
 The information you provide in the recipe is not only affecting the processes
 you are starting, but also the directory names your output will be structured
-in. For additional reads, please have a look at the recipe format description in
-the [ESMValTool
-manual](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#recipe-section-diagnostics).
+in.
+For additional reads, please have a look at the recipe format description in the
+[ESMValTool manual](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#recipe-section-diagnostics).
 
 ## How to run ESMValTool
 
-Once you’ve set up your conda environment and installed ESMValTool (see[ episode
-#2]({{ page.root}}{% link _episodes/02-installation.md %})) and set up your `config-user.yml` file to correctly match your local
-environment, (see [episode #3]({{ page.root}}{% link _episodes/03-configuration.md %})), ESMValTool is invoked using a simple
-command:
+Once you’ve set up your conda environment and installed ESMValTool (see
+[episode #2]({{ page.root}}{% link _episodes/02-installation.md %})
+) and set up your `config-user.yml` file to correctly match your local
+environment, (see
+[episode #3]({{ page.root}}{% link _episodes/03-configuration.md %})
+), ESMValTool is invoked using the command:
 
+~~~bash
+esmvaltool run /path/to/recipe_example.yml
 ~~~
-esmvaltool run recipe
-~~~
-{: .source}
 
-If the configuration file is not in the default {HOME}\.esmvaltool\ path you can pass its path explicitly:
+If the configuration file is not in the default location
+`~/.esmvaltool/config-user.yml`, you can pass its path explicitly:
 
+~~~bash
+esmvaltool run --config_file /path/to/config-user.yml /path/to/recipe_example.yml
 ~~~
-esmvaltool run --config_file configuration recipe
+
+Note that the path to the recipe can be either a path to a recipe file or it can
+be the name of an installed recipe.
+To view the list of installed recipes, run
+~~~bash
+esmvaltool recipes list
 ~~~
-{: .source}
 
 To try your hand with a basic recipe, please work through this episode.
 
 ## Introduction to the example recipe
 The recipe presented here is a simple, basic recipe that takes a single dataset
 and produces a time series plot.
 
-Please download the following recipe into your ESMValTool working directory with
-the name: [recipe_example.yml](https://github.com/ESMValGroup/ESMValTool_Tutorial/blob/master/data/recipe_example.yml)
+Please download
+[recipe_example.yml](https://github.com/ESMValGroup/ESMValTool_Tutorial/blob/master/data/recipe_example.yml)
+into your ESMValTool working directory:
+
+~~~bash
+wget https://raw.githubusercontent.com/ESMValGroup/ESMValTool_Tutorial/master/data/recipe_example.yml
+~~~
 
 > ## recipe_example.yml
 > ```YAML
@@ -153,14 +166,17 @@ the name: [recipe_example.yml](https://github.com/ESMValGroup/ESMValTool_Tutoria
 Please note the following sections:
 
   - documentation: lines 4-20
-  
+
     The documentation consists of the following information:
 
     - description: a short description of the recipe
     - authors: a list of authors (linked to `esmvaltool/config-references.yml`)
-    - maintainer: a list of maintainers (linked to `esmvaltool/config-references.yml`)
-    - references: a list of references (linked to a bibtexfile in `esmvaltool/references` with the same name)
-    - projects: a list of projects (linked to `esmvaltool/config-references.yml`)
+    - maintainer: a list of maintainers (linked to
+      `esmvaltool/config-references.yml`)
+    - references: a list of references (linked to a bibtexfile in
+      `esmvaltool/references` with the same name)
+    - projects: a list of projects (linked to
+      `esmvaltool/config-references.yml`)
 
 
   - datasets: lines 22-23
@@ -180,11 +196,12 @@ Please note the following sections:
 
   - preprocessors: lines 25-28
 
-    The definition for different preprocessors or preprocessor combinations. If no
-    preprocessing is needed, the preprocessor can be set to an empty python
-    dictionary (`{}`). Here, we produce annual means. The preprocessor is called
+    The definition for different preprocessors or preprocessor combinations.
+    If no preprocessing is needed, the preprocessor can be set to an empty
+    map (`{}`). Here, we produce annual means. The preprocessor is called
     with its name (here: prep_timeseries), later in the diagnostic (line 39).
-    (See [episode #5]({{ page.root}}{% link _episodes/05-preprocessor.md %}) for more details.)
+    (See [episode #5]({{ page.root}}{% link _episodes/05-preprocessor.md %})
+    for more details.)
 
   - diagnostic section: lines 30-42
 
@@ -204,8 +221,11 @@ Please note the following sections:
     - scripts: a definition of all scripts that are used in this diagnostic
     - the next indent (here: timeseries_diag) is the scripts’ names (a string
       without whitespace) for the script to use
-    - script: a executable script with a directory relative to the
-      `esmvaltool/diag_scripts/` directory
+    - script: a script that will produce the plots. The path can be either
+      relative the ESMValTool installation with subdirectory
+      [`esmvaltool/diag_scripts/`](https://github.com/ESMValGroup/ESMValTool/tree/master/esmvaltool/diag_scripts)
+      or an absolute path. In this case, it is a diagnostic script that is
+      installed with ESMValTool.
 
 > ## Please answer the following questions:
 > What is the short_name of the variable being analyzed?
@@ -222,7 +242,8 @@ Please note the following sections:
 {: .solution}
 
 > ## What is the diagnostic script being used?
-> `ocean/diagnostic_timeseries.py`
+> The installed copy of
+> [`ocean/diagnostic_timeseries.py`](https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/diag_scripts/ocean/diagnostic_timeseries.py)
 {: .solution}
 
 > ## How many years of data are being analyzed?

_episodes/06-debugging.md

@@ -50,8 +50,8 @@ conda activate esmvaltool
 
 > ## conda environment
 >
-> More information about conda environment can be found at [Installation]({{
-> page.root }}{% link _episodes/02-installation.md %})
+> More information about the conda environment can be found at
+> [Installation]({{ page.root }}{% link _episodes/02-installation.md %}).
 {: .callout}
 
 Let's change the working directory to the folder ``run`` and list its files: