add more challenges, revise the text

Author: SarahAlidoost

_episodes/03-configuration.md

@@ -12,8 +12,9 @@ objectives:
 
 keypoints:
 - The ``config-user.yml`` tells ESMValTool where to find input data.
-- "``rootpath`` defines the root directory for the input data."
 - "``output_dir`` defines the destination directory."
+- "``rootpath`` defines the root path of the data."
+- "``drs`` defines the directory structure of the data."
 
 ---
 
@@ -98,7 +99,22 @@ config_developer_file: null
 profile_diagnostic: false
 ```
 
-In general there is no need to change the settings listed above.
+> ## Saving preprocessed data
+>
+> In the configuration file, which settings are useful to make sure preprocessed
+> data is stored when ESMValTool is run?
+>
+>> ## Solution
+>>
+>> If the option ``remove_preproc_dir`` is set to ``false``, then the
+>> ``preproc/`` directory contains all the pre-processed data and the
+>> metadata interface files.
+>> If the option ``save_intermediary_cubes`` is set to ``true``
+>> then data will also be saved after each preprocessor step in the folder
+>> ``preproc``. Note that saving all intermediate results to file will result
+>> in a considerable slowdown.
+> {: .solution}
+{: .challenge}
 
 ## Destination directory
 
@@ -108,12 +124,6 @@ a new output folder determined by recipe name, and date and time using
 the format: YYYYMMDD_HHMMSS.
 This folder contains four further subfolders: ``plots``, ``preproc``, ``run``, ``work``.
 
-Let's name our destination directory ``esmvaltool_output`` in the working directory:
-
-```yaml
-output_dir: ./esmvaltool_output
-```
-
 > ## Content of subfolders
 >
 > - ``plots``: the location for all plots, split by individual diagnostics and fields.
@@ -130,6 +140,22 @@ are not plots, e.g. files in NetCDF format (depends on the diagnostic script).
 [lesson]({{ page.root }}{% link _episodes/04-recipe.md %})
 {: .callout}
 
+> ## Set the destination directory
+>
+> Let's name our destination directory ``esmvaltool_output`` in the working directory.
+> How to tell ESMValTool about our destination directory?
+>
+>> ## Solution
+>>
+>> We use `output_dir` entry in the `config-user.yml` file as:
+>>```yaml
+>> output_dir: ./esmvaltool_output
+>>```
+>>
+>> If the `esmvaltool_output` does not exist, ESMValTool will generate it for you.
+> {: .solution}
+{: .challenge}
+
 ## Auxiliary data directory
 
 The ``auxiliary_data_dir`` setting is the path where any required additional
@@ -151,7 +177,6 @@ You can choose the number of tasks in parallel as
 ESMValTool to use the maximum number of available CPUs:
 
 ```yaml
-
 max_parallel_tasks: null
 ```
 
@@ -164,7 +189,6 @@ Using the number there you can increase the number of parallel tasks
 again to a reasonable number for the amount of memory available in your system.
 {: .callout}
 
-
 ## Rootpath to input data
 
 ESMValTool uses several categories (in ESMValTool, this is referred to as projects)
@@ -175,6 +199,7 @@ We can find more information about the projects in the ESMValTool
 [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html).
 The ``rootpath`` specifies the directories where ESMValTool will look for input data.
 For each category, you can define either one path or several paths as a list.
+For example:
 
 ```yaml
 rootpath:
@@ -184,71 +209,131 @@ rootpath:
   default: ~/default_inputpath
   CORDEX: ~/default_inputpath
 ```
-Site-specific entries for Jasmin, DKRZ and ETHZ are listed at the end of the
-example configuration file.
 
-In this lesson, we will work with data from
-[CMIP5](https://esgf-node.llnl.gov/projects/cmip5/)
-and [obs4mips](https://esgf-node.llnl.gov/projects/obs4mips/).
-We add the root path of the folder where  our/your data is available.
-
-```yaml
-  rootpath:
-  ...
-    CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2, ~/esmvaltool_tutorial/data]
-    obs4mips: ~/esmvaltool_tutorial/data
-```
+Site-specific entries for Jasmin and DKRZ are listed at the end of the
+example configuration file.
 
-> ## Setting the correct rootpath
+> ## Set the correct rootpath
 >
-> - To get the data (or its correct rootpath), check instruction in
->   [Setup]({{ page.root }}{% link setup.md %}).
-> - For more information about setting the rootpath, see also the ESMValTool
->   [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
-{: .callout}
+> In this tutorial, we will work with data from
+> [CMIP5](https://esgf-node.llnl.gov/projects/cmip5/)
+> and [obs4mips](https://esgf-node.llnl.gov/projects/obs4mips/).
+> How does ESMValTool find the data?
+>
+> Hint:
+> to get the data (or its correct rootpath), check instruction in
+> [Setup]({{ page.root }}{% link setup.md %}).
+>
+>> ## Solution
+>>
+>> - Are you working on your own local machine?
+>>   You need to add the root path of the folder where the data is available
+>> to the `config-user.yml` file as:
+>>```yaml
+>>  rootpath:
+>>  ...
+>>    CMIP5: ~/esmvaltool_tutorial/data
+>>    obs4mips: ~/esmvaltool_tutorial/data
+>>```
+>>
+>> - Are you working with on a computer cluster like Jasmin or DKRZ?
+>>   Site-specific path to the data are already listed at the end of the
+>> `config-user.yml` file. You need to uncomment the related lines.
+>> For example, on Jasmin:
+>>```yaml
+>>  # Site-specific entries: Jasmin
+>>  # Uncomment the lines below to locate data on JASMIN
+>>  rootpath:
+>>  #  CMIP6: /badc/cmip6/data/CMIP6
+>>    CMIP5: /badc/cmip5/data/cmip5/output1
+>>  #  CMIP3: /badc/cmip3_drs/data/cmip3/output
+>>  #  OBS: /group_workspaces/jasmin4/esmeval/obsdata-v2
+>>  #  OBS6: /group_workspaces/jasmin4/esmeval/obsdata-v2
+>>    obs4mips: /group_workspaces/jasmin4/esmeval/obsdata-v2
+>>  #  ana4mips: /group_workspaces/jasmin4/esmeval/obsdata-v2
+>>  #  CORDEX: /badc/cordex/data/CORDEX/output
+>>```
+>>
+>> - For more information about setting the rootpath, see also the ESMValTool
+>> [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html).
+> {: .solution}
+{: .challenge}
 
 ## Directory structure for the data from different projects
 
 Input data can be from various models, observations and reanalysis data that
 adhere to the [CF/CMOR standard](https://cmor.llnl.gov/). The ``drs`` setting
-describes the file structure. Let's use ``default`` for ``CMIP5`` in our example
-here:
+describes the file structure.
 
-```yaml
-drs:
-  CMIP5: default
-```
+The ``drs`` setting describes the file structure for several projects (e.g.
+CMIP6, CMIP5, obs4mips, OBS6, OBS) on several key machines
+(e.g. BADC, CP4CDS, DKRZ, ETHZ, SMHI, BSC). For more
+information about ``drs``, you can visit the ESMValTool documentation on
+[Data Reference Syntax (DRS)](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html#cmor-drs).
 
-> ## Available drs
+> ## Set the correct drs
 >
-> The ``drs`` setting describes the file structure for several projects (e.g.
-> ``CMIP6``, ``CMIP5``, ``obs4mips``, ``OBS6``, ``OBS``) on several key machines
-> (e.g. ``BADC``, ``CP4CDS``, ``DKRZ``, ``ETHZ``, ``SMHI``, ``BSC``). For more
-> information about ``drs``, you can visit the ESMValTool
-> [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html#cmor-drs).
-{: .callout}
-
-> ## Make your own configuration file
+> In this lesson, we will work with data from
+> [CMIP5](https://esgf-node.llnl.gov/projects/cmip5/)
+> and [obs4mips](https://esgf-node.llnl.gov/projects/obs4mips/).
+> How does ESMValTool know the data structure?
 >
-> It is possible to have several configuration files with different purposes,
-> for example: config-user_formalised_runs.yml, config-user_debugging.yml
-{: .callout}
+>> ## Solution
+>>
+>> - Are you working on your own local machine?
+>>   You need to set the `drs` of the data
+>> in the `config-user.yml` file as:
+>>```yaml
+>>   drs:
+>>     CMIP5: default
+>>     obs4mips: default
+>>```
+>>
+>> - Are you working with on a computer cluster like Jasmin or DKRZ?
+>>   Site-specific `drs` of the data are already listed at the end of the
+>> `config-user.yml` file. You need to uncomment the related lines.
+>> For example, on Jasmin:
+>>```yaml
+>>  # Site-specific entries: Jasmin
+>>  # Uncomment the lines below to locate data on JASMIN
+>>  drs:
+>>  #  CMIP6: BADC
+>>    CMIP5: BADC
+>>  #  CMIP3: BADC
+>>  #  CORDEX: BADC
+>>  #  OBS: BADC
+>>  #  OBS6: BADC
+>>    obs4mips: BADC
+>>  #  ana4mips: BADC
+>>```
+>>
+> {: .solution}
+{: .challenge}
 
-> ## Saving preprocessed data
+> ## Explain the default drs (if working on local machine)
 >
-> In the configuration file, which settings are useful to make sure preprocessed
-> data is stored when ESMValTool is run?
+> 1. In the previous exercise, we set the `drs` of CMIP5 data to `default`.
+> Can you explain why?
+>
+> 2. Have a look at the directory structure of the data.
+> There is the folder `Tier1`. What does it mean?
 >
 >> ## Solution
 >>
-> > If the option ``remove_preproc_dir`` is set to ``false``, then the
-> > ``preproc/`` directory contains all the pre-processed data and the
-> > metadata interface files.
-> > If the option ``save_intermediary_cubes`` is set to ``true``
-> > then data will also be saved after each preprocessor step in the folder
-> > ``preproc``. Note that saving all intermediate results to file will result
-> > in a considerable slowdown.
+>> 1. `drs: default` is one way to retrieve data from a ROOT directory that has no DRS-like structure.
+>> ``default`` indicates that all the files are in a folder without any structure.
+>>
+>> 2. Observational data are organized in Tiers depending on their level of public availability.
+>> Therefore the default directory must be structured accordingly with sub-directories
+>> `TierX` e.g. Tier1, Tier2 or Tier3, even when `drs: default`.
+>>
 > {: .solution}
 {: .challenge}
 
+> ## Make your own configuration file
+>
+> It is possible to have several configuration files with different purposes,
+> for example: config-user_formalised_runs.yml, config-user_debugging.yml
+{: .callout}
+
 {% include links.md %}