Merge remote-tracking branch 'origin/master' into setup_md

Author: SarahAlidoost

LICENSE LICENSE.mdLICENSE renamed to LICENSE.md

@@ -1,7 +1,7 @@
 ---
 title: "Introduction"
-teaching: 0
-exercises: 25
+teaching: 20
+exercises: 10
 questions:
 - What is ESMValTool and when is it useful?
 - What are the main components of ESMValTool?

_episodes/01-introduction.md

@@ -244,12 +244,12 @@ to display the command line help.
 > >
 > > In my case when I run
 > > ~~~
-> > esmvaltool --version
+> > esmvaltool version
 > > ~~~
 > > {: .bash}
 > > I get that my installed ESMValTool version is
 > > ~~~
-> > 2.0.0b9
+> > ESMValCore: 2.0.0
 > > ~~~
 > > {: .output}
 > {: .solution}

_episodes/02-installation.md

@@ -1,7 +1,7 @@
 ---
 title: "Configuration"
-teaching: 0
-exercises: 0
+teaching: 10
+exercises: 10
 questions:
 - What is the user configuration file and how should I use it?
 
@@ -20,42 +20,36 @@ keypoints:
 ## The configuration file
 
 The ``config-user.yml`` configuration file contains all the global level
-information needed by ESMValTool to run. This is an [YAML
-file](https://yaml.org/spec/1.2/spec.html). An example configuration file can be
-found in the root directory of the ESMValTool repository:
-[config-user-example.yml](https://github.com/ESMValGroup/ESMValTool/blob/master/config-user-example.yml).
+information needed by ESMValTool to run.
+This is a [YAML file](https://yaml.org/spec/1.2/spec.html).
+An example configuration file can be found in the ESMValCore repository:
+[config-user-example.yml](https://github.com/ESMValGroup/ESMValCore/blob/master/esmvalcore/config-user.yml).
 
-First, we make a working directory ``esmvaltool_tutorial``.
-In a new terminal, run:
+You can generate the default configuration file by running:
 
 ~~~bash
-  mkdir esmvaltool_tutorial
-  cd esmvaltool_tutorial
+  esmvaltool config get_config_user
 ~~~
 
-Now, we download the configuration file to our working directory. To do that,
-click on [this
-link](https://raw.githubusercontent.com/ESMValGroup/ESMValTool/master/config-user-example.yml)
-to see a raw version of the file, right-click and press ``save as``, then you
-can rename it to ``config-user.yml``and save it into the working directory
-``esmvaltool_tutorial``.
+It will save the file to: `~/.esmvaltool/config-user.yml`, where `~` is the
+path to your home directory. Note that files and directories starting with a
+period are "hidden", to see the `.esmvaltool` directory in the terminal use
+`ls -la ~`.
 
-Now, let's change our working directory in a terminal window to
-``esmvaltool_tutorial``. Then, we run a text editor called Nano to have a look
-inside the configuration file:
+We run a text editor called ``nano`` to have a look inside the configuration file:
 
 ~~~bash
-  nano config-user.yml
+  nano ~/.esmvaltool/config-user.yml
 ~~~
 
 This file contains the information for:
 
-- Rootpath to input data
-- Directory structure for the data from different projects
-- Number of tasks that can be run in parallel
+- Output settings
 - Destination directory
 - Auxiliary data directory
-- Output settings
+- Number of tasks that can be run in parallel
+- Rootpath to input data
+- Directory structure for the data from different projects
 
 > ## Text editor side note
 >
@@ -67,94 +61,48 @@ This file contains the information for:
 > and then <kbd>ctrl</kbd> + <kbd>X</kbd> to exit ``nano``.
 {: .callout}
 
-## Rootpath to input data
-
-ESMValTool uses several categories (in ESMValTool, this is referred to as projects)
-for input data based on their source. The current categories in the configuration
-file are mentioned below. For example, CMIP is used for a dataset from
-the climate model intercomparison project whereas OBS is used for an observational dataset.
-We can find more information about the projects in the ESMValTool
-[documentation](https://docs.esmvaltool.org/en/latest/input.html).
-The ``rootpath`` specifies the directories where ESMValTool will look for input data.
-For each category, you can define either one path or several paths as a list.
-
-```yaml
-rootpath:
-  CMIP3: [~/cmip3_inputpath1, ~/cmip3_inputpath2]
-  CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2]
-  CMIP6: [~/cmip6_inputpath1, ~/cmip6_inputpath2]
-  OBS: ~/obs_inputpath
-  OBS6: ~/obs6_inputpath
-  obs4mips: ~/obs4mips_inputpath
-  ana4mips: ~/ana4mips_inputpath
-  native6:  ~/native6_inputpath
-  RAWOBS: ~/rawobs_inputpath
-  default: ~/default_inputpath
-```
-
-In this lesson, we will work with data from
-[CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
-We add the root path of the folder where  our/your data is available.
-
-```yaml
-  rootpath:
-  ...
-    CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2, ~/esmvaltool_tutorial/data]
-```
-
-> ## Setting the correct rootpath
->
-> - To get the data (or its correct rootpath), check instruction in [Setup]({{
->   page.root }}{% link setup.md %}).
-> - For more information about setting the rootpath, see also the ESMValTool
->   [documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
-{: .callout}
-
-## Directory structure for the data from different projects
+## Output settings
 
-Input data can be from various models, observations and reanalysis data that
-adhere to the [CF/CMOR standard](https://cmor.llnl.gov/). The ``drs`` setting
-describes the file structure. Let's use ``default`` for ``CMIP5`` in our example
-here:
+These settings are used to inform ESMValTool about your preference about
+specific actions. You can turn on or off the setting by ``true`` or ``false``
+values. Most of these settings are fairly self-explanatory, ie:
 
 ```yaml
-drs:
-  CMIP5: default
-```
-
-> ## Available drs
->
-> The ``drs`` setting describes the file structure for several projects (e.g.
-> ``CMIP6``, ``CMIP5``, ``obs4mips``, ``OBS6``, ``OBS``) on several key machines
-> (e.g. ``BADC``, ``CP4CDS``, ``DKRZ``, ``ETHZ``, ``SMHI``, ``BSC``). For more
-> information about ``drs``, you can visit the ESMValTool
-> [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html#cmor-drs).
-{: .callout}
+# Diagnostics create plots? [true]/false
+write_plots: true
+# Diagnositcs write NetCDF files? [true]/false
+write_netcdf: true
+# Set the console log level debug, [info], warning, error
+log_level: info
+# Exit on warning (only for NCL diagnostic scripts)? true/[false]
+exit_on_warning: false
+# Plot file format? [png]/pdf/ps/eps/epsi
+output_file_type: png
 
-## Number of parallel tasks
+...
 
-This option enables you to perform parallel processing.
-You can choose the number of tasks in parallel as
-1/2/3/4/... or you can set it to ``null``. That tells
-ESMValTool to use the maximum number of available CPUs:
+# Use netCDF compression true/[false]
+compress_netcdf: false
+# Save intermediary cubes in the preprocessor true/[false]
+save_intermediary_cubes: false
+# Remove the preproc dir if all fine
+remove_preproc_dir: true
 
-```yaml
+...
 
-max_parallel_tasks: null
+# Path to custom config-developer file, to customise project configurations.
+# See config-developer.yml for an example. Set to [null] to use the default
+config_developer_file: null
+# Get profiling information for diagnostics
+# Only available for Python diagnostics
+profile_diagnostic: false
 ```
 
-> ## Set the number of tasks
->
-> If you run out of memory, try setting ``max_parallel_tasks`` to 1.
-Then, check the amount of memory you need for that by inspecting
-the file ``run/resource_usage.txt`` in the output directory.
-Using the number there you can increase the number of parallel tasks
-again to a reasonable number for the amount of memory available in your system.
-{: .callout}
+In general there is no need to change the settings listed above.
 
 ## Destination directory
 
-The destination directory is the rootpath where ESMValTool will store its output,
+The destination directory is the rootpath where ESMValTool will store its output folders containing
 i.e. figures, data, logs, etc. With every run, ESMValTool automatically generates
 a new output folder determined by recipe name, and date and time using
 the format: YYYYMMDD_HHMMSS.
@@ -195,55 +143,109 @@ on.
 auxiliary_data_dir: ~/auxiliary_data
 ```
 
-## Output settings
+## Number of parallel tasks
 
-These settings are used to inform ESMValTool about your preference about
-specific actions. You can turn on or off the setting by ``true`` or ``false``
-values. Most of these settings are fairly self-explanatory, ie:
+This option enables you to perform parallel processing.
+You can choose the number of tasks in parallel as
+1/2/3/4/... or you can set it to ``null``. That tells
+ESMValTool to use the maximum number of available CPUs:
 
 ```yaml
-# Diagnostics create plots? [true]/false
-write_plots: true
-# Diagnositcs write NetCDF files? [true]/false
-write_netcdf: true
-# Set the console log level debug, [info], warning, error
-log_level: info
-# Exit on warning (only for NCL diagnostic scripts)? true/[false]
-exit_on_warning: false
-# Plot file format? [png]/pdf/ps/eps/epsi
-output_file_type: png
-# Use netCDF compression true/[false]
-compress_netcdf: false
-# Save intermediary cubes in the preprocessor true/[false]
-save_intermediary_cubes: false
-# Remove the preproc dir if all fine
-remove_preproc_dir: true
-# Path to custom config-developer file, to customise project configurations.
-# See config-developer.yml for an example. Set to [null] to use the default
-# config_developer_file: null
-# Get profiling information for diagnostics
-# Only available for Python diagnostics
-profile_diagnostic: false
+
+max_parallel_tasks: null
+```
+
+> ## Set the number of tasks
+>
+> If you run out of memory, try setting ``max_parallel_tasks`` to 1.
+Then, check the amount of memory you need for that by inspecting
+the file ``run/resource_usage.txt`` in the output directory.
+Using the number there you can increase the number of parallel tasks
+again to a reasonable number for the amount of memory available in your system.
+{: .callout}
+
+
+## Rootpath to input data
+
+ESMValTool uses several categories (in ESMValTool, this is referred to as projects)
+for input data based on their source. The current categories in the configuration
+file are mentioned below. For example, CMIP is used for a dataset from
+the climate model intercomparison project whereas OBS is used for an observational dataset.
+We can find more information about the projects in the ESMValTool
+[documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html).
+The ``rootpath`` specifies the directories where ESMValTool will look for input data.
+For each category, you can define either one path or several paths as a list.
+
+```yaml
+rootpath:
+  CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2]
+  OBS: ~/obs_inputpath
+  RAWOBS: ~/rawobs_inputpath
+  default: ~/default_inputpath
+  CORDEX: ~/default_inputpath
 ```
+Site-specific entries for Jasmin, DKRZ and ETHZ are listed at the end of the
+example configuration file.
+
+In this lesson, we will work with data from
+[CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
+We add the root path of the folder where  our/your data is available.
+
+```yaml
+  rootpath:
+  ...
+    CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2, ~/esmvaltool_tutorial/data]
+```
+
+> ## Setting the correct rootpath
+>
+> - To get the data (or its correct rootpath), check instruction in
+>   [Setup]({{ page.root }}{% link setup.md %}).
+> - For more information about setting the rootpath, see also the ESMValTool
+>   [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
+{: .callout}
+
+## Directory structure for the data from different projects
+
+Input data can be from various models, observations and reanalysis data that
+adhere to the [CF/CMOR standard](https://cmor.llnl.gov/). The ``drs`` setting
+describes the file structure. Let's use ``default`` for ``CMIP5`` in our example
+here:
+
+```yaml
+drs:
+  CMIP5: default
+```
+
+> ## Available drs
+>
+> The ``drs`` setting describes the file structure for several projects (e.g.
+> ``CMIP6``, ``CMIP5``, ``obs4mips``, ``OBS6``, ``OBS``) on several key machines
+> (e.g. ``BADC``, ``CP4CDS``, ``DKRZ``, ``ETHZ``, ``SMHI``, ``BSC``). For more
+> information about ``drs``, you can visit the ESMValTool
+> [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html#cmor-drs).
+{: .callout}
 
 > ## Make your own configuration file
 >
 > It is possible to have several configuration files with different purposes,
-> for example: config-user_formalised_runs.yml, config-user_debugging.yml {:
-> .callout}
->
+> for example: config-user_formalised_runs.yml, config-user_debugging.yml
+{: .callout}
+
 > ## Saving preprocessed data
 >
 > In the configuration file, which settings are useful to make sure preprocessed
 > data is stored when ESMValTool is run?
 >
 >> ## Solution
 >>
-> > If the option ``save_intermediary_cubes`` is set to true in the
-> > config-user.yml file, then the intermediary cubes will also be saved in the
-> > folder ``preproc``. Also, if the option ``remove_preproc_dir`` is set to
-> > ``false``, then the ``preproc/`` directory contains all the preprocessed
-> > data and the metadata interface files.
+> > If the option ``remove_preproc_dir`` is set to ``false``, then the
+> > ``preproc/`` directory contains all the pre-processed data and the
+> > metadata interface files.
+> > If the option ``save_intermediary_cubes`` is set to ``true``
+> > then data will also be saved after each preprocessor step in the folder
+> > ``preproc``. Note that saving all intermediate results to file will result
+> > in a considerable slowdown.
 > {: .solution}
 {: .challenge}