Merge pull request #70 from ESMValGroup/cli_update

JaroCamphuijsen · web-flow · commit 6acc1def38b4 · 2020-07-17T17:01:23.000+02:00
Add changes due to new CLI
diff --git a/_episodes/02-installation.md b/_episodes/02-installation.md
@@ -244,12 +244,12 @@ to display the command line help.
 > >
 > > In my case when I run
 > > ~~~
-> > esmvaltool --version
+> > esmvaltool version
 > > ~~~
 > > {: .bash}
 > > I get that my installed ESMValTool version is
 > > ~~~
-> > 2.0.0b9
+> > ESMValCore: 2.0.0
 > > ~~~
 > > {: .output}
 > {: .solution}
diff --git a/_episodes/03-configuration.md b/_episodes/03-configuration.md
@@ -22,26 +22,19 @@ keypoints:
 The ``config-user.yml`` configuration file contains all the global level
 information needed by ESMValTool to run. This is an [YAML
 file](https://yaml.org/spec/1.2/spec.html). An example configuration file can be
-found in the root directory of the ESMValTool repository:
-[config-user-example.yml](https://github.com/ESMValGroup/ESMValTool/blob/master/config-user-example.yml).
+found in the ESMValCore repository:
+[config-user-example.yml](https://github.com/ESMValGroup/ESMValCore/blob/master/esmvalcore/config-user.yml).
 
-First, we make a working directory ``esmvaltool_tutorial``.
-In a new terminal, run:
+You can generate the default configuration file by running:
 
 ~~~bash
-  mkdir esmvaltool_tutorial
-  cd esmvaltool_tutorial
+  esmvaltool config get_config_user
 ~~~
 
-Now, we download the configuration file to our working directory. To do that,
-click on [this
-link](https://raw.githubusercontent.com/ESMValGroup/ESMValTool/master/config-user-example.yml)
-to see a raw version of the file, right-click and press ``save as``, then you
-can rename it to ``config-user.yml``and save it into the working directory
-``esmvaltool_tutorial``.
+It will save the file to: ``{HOME}/.esmvaltool/config-user.yml``.
 
 Now, let's change our working directory in a terminal window to
-``esmvaltool_tutorial``. Then, we run a text editor called Nano to have a look
+``{HOME}/.esmvaltool``. Then, we run a text editor called Nano to have a look
 inside the configuration file:
 
 ~~~bash
@@ -50,12 +43,12 @@ inside the configuration file:
 
 This file contains the information for:
 
-- Rootpath to input data
-- Directory structure for the data from different projects
-- Number of tasks that can be run in parallel
+- Output settings
 - Destination directory
 - Auxiliary data directory
-- Output settings
+- Number of tasks that can be run in parallel
+- Rootpath to input data
+- Directory structure for the data from different projects
 
 > ## Text editor side note
 >
@@ -67,6 +60,108 @@ This file contains the information for:
 > and then <kbd>ctrl</kbd> + <kbd>X</kbd> to exit ``nano``.
 {: .callout}
 
+## Output settings
+
+These settings are used to inform ESMValTool about your preference about
+specific actions. You can turn on or off the setting by ``true`` or ``false``
+values. Most of these settings are fairly self-explanatory, ie:
+
+```yaml
+# Diagnostics create plots? [true]/false
+write_plots: true
+# Diagnositcs write NetCDF files? [true]/false
+write_netcdf: true
+# Set the console log level debug, [info], warning, error
+log_level: info
+# Exit on warning (only for NCL diagnostic scripts)? true/[false]
+exit_on_warning: false
+# Plot file format? [png]/pdf/ps/eps/epsi
+output_file_type: png
+
+...
+
+# Use netCDF compression true/[false]
+compress_netcdf: false
+# Save intermediary cubes in the preprocessor true/[false]
+save_intermediary_cubes: false
+# Remove the preproc dir if all fine
+remove_preproc_dir: true
+
+...
+
+# Path to custom config-developer file, to customise project configurations.
+# See config-developer.yml for an example. Set to [null] to use the default
+ config_developer_file: null
+# Get profiling information for diagnostics
+# Only available for Python diagnostics
+profile_diagnostic: false
+```
+
+## Destination directory
+
+The destination directory is the rootpath where ESMValTool will store its output folders containing
+i.e. figures, data, logs, etc. With every run, ESMValTool automatically generates
+a new output folder determined by recipe name, and date and time using
+the format: YYYYMMDD_HHMMSS.
+This folder contains four further subfolders: ``plots``, ``preproc``, ``run``, ``work``.
+
+Let's name our destination directory ``esmvaltool_output`` in the working directory:
+
+```yaml
+output_dir: ./esmvaltool_output
+```
+
+> ## Content of subfolders
+>
+> - ``plots``: the location for all plots, split by individual diagnostics and fields.
+> - ``preproc``: this folder contains all the preprocessed data and metadata.yml
+interface files. Note that by default this directory will be deleted after
+each run because most users will only need the results from the diagnostic scripts.
+> - ``run``: this folder includes all log files, a copy of the recipe,
+a summary of the resource usage, and the settings.yml interface files,
+resource_usage.txt and temporary files created by the diagnostic scripts.
+> - ``work``: this folder is a place for any diagnostic script results that
+are not plots, e.g. files in NetCDF format (depends on the diagnostic script).
+>
+> We explain more about output in the next
+[lesson]({{ page.root }}{% link _episodes/04-recipe.md %})
+{: .callout}
+
+## Auxiliary data directory
+
+The ``auxiliary_data_dir`` setting is the path where any required additional
+auxiliary data files are stored. This location allows us to tell the diagnostic
+script where to find the files if they can not be downloaded at runtime. This
+option should not be used for model or observational datasets, but for data
+files  (e.g. shape files) used in plotting such as coastline descriptions and so
+on.
+
+```yaml
+auxiliary_data_dir: ~/auxiliary_data
+```
+
+## Number of parallel tasks
+
+This option enables you to perform parallel processing.
+You can choose the number of tasks in parallel as
+1/2/3/4/... or you can set it to ``null``. That tells
+ESMValTool to use the maximum number of available CPUs:
+
+```yaml
+
+max_parallel_tasks: null
+```
+
+> ## Set the number of tasks
+>
+> If you run out of memory, try setting ``max_parallel_tasks`` to 1.
+Then, check the amount of memory you need for that by inspecting
+the file ``run/resource_usage.txt`` in the output directory.
+Using the number there you can increase the number of parallel tasks
+again to a reasonable number for the amount of memory available in your system.
+{: .callout}
+
+
 ## Rootpath to input data
 
 ESMValTool uses several categories (in ESMValTool, this is referred to as projects)
@@ -80,17 +175,13 @@ For each category, you can define either one path or several paths as a list.
 
 ```yaml
 rootpath:
-  CMIP3: [~/cmip3_inputpath1, ~/cmip3_inputpath2]
   CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2]
-  CMIP6: [~/cmip6_inputpath1, ~/cmip6_inputpath2]
   OBS: ~/obs_inputpath
-  OBS6: ~/obs6_inputpath
-  obs4mips: ~/obs4mips_inputpath
-  ana4mips: ~/ana4mips_inputpath
-  native6:  ~/native6_inputpath
   RAWOBS: ~/rawobs_inputpath
   default: ~/default_inputpath
+  CORDEX: ~/default_inputpath
 ```
+Site-specific entries for Jasmin, DKRZ and ETHZ are listed at the end of the example configuration file. 
 
 In this lesson, we will work with data from
 [CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
@@ -131,107 +222,12 @@ drs:
 > [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html#cmor-drs).
 {: .callout}
 
-## Number of parallel tasks
-
-This option enables you to perform parallel processing.
-You can choose the number of tasks in parallel as
-1/2/3/4/... or you can set it to ``null``. That tells
-ESMValTool to use the maximum number of available CPUs:
-
-```yaml
-
-max_parallel_tasks: null
-```
-
-> ## Set the number of tasks
->
-> If you run out of memory, try setting ``max_parallel_tasks`` to 1.
-Then, check the amount of memory you need for that by inspecting
-the file ``run/resource_usage.txt`` in the output directory.
-Using the number there you can increase the number of parallel tasks
-again to a reasonable number for the amount of memory available in your system.
-{: .callout}
-
-## Destination directory
-
-The destination directory is the rootpath where ESMValTool will store its output,
-i.e. figures, data, logs, etc. With every run, ESMValTool automatically generates
-a new output folder determined by recipe name, and date and time using
-the format: YYYYMMDD_HHMMSS.
-This folder contains four further subfolders: ``plots``, ``preproc``, ``run``, ``work``.
-
-Let's name our destination directory ``esmvaltool_output`` in the working directory:
-
-```yaml
-output_dir: ./esmvaltool_output
-```
-
-> ## Content of subfolders
->
-> - ``plots``: the location for all plots, split by individual diagnostics and fields.
-> - ``preproc``: this folder contains all the preprocessed data and metadata.yml
-interface files. Note that by default this directory will be deleted after
-each run because most users will only need the results from the diagnostic scripts.
-> - ``run``: this folder includes all log files, a copy of the recipe,
-a summary of the resource usage, and the settings.yml interface files,
-resource_usage.txt and temporary files created by the diagnostic scripts.
-> - ``work``: this folder is a place for any diagnostic script results that
-are not plots, e.g. files in NetCDF format (depends on the diagnostic script).
->
-> We explain more about output in the next
-[lesson]({{ page.root }}{% link _episodes/04-recipe.md %})
-{: .callout}
-
-## Auxiliary data directory
-
-The ``auxiliary_data_dir`` setting is the path where any required additional
-auxiliary data files are stored. This location allows us to tell the diagnostic
-script where to find the files if they can not be downloaded at runtime. This
-option should not be used for model or observational datasets, but for data
-files  (e.g. shape files) used in plotting such as coastline descriptions and so
-on.
-
-```yaml
-auxiliary_data_dir: ~/auxiliary_data
-```
-
-## Output settings
-
-These settings are used to inform ESMValTool about your preference about
-specific actions. You can turn on or off the setting by ``true`` or ``false``
-values. Most of these settings are fairly self-explanatory, ie:
-
-```yaml
-# Diagnostics create plots? [true]/false
-write_plots: true
-# Diagnositcs write NetCDF files? [true]/false
-write_netcdf: true
-# Set the console log level debug, [info], warning, error
-log_level: info
-# Exit on warning (only for NCL diagnostic scripts)? true/[false]
-exit_on_warning: false
-# Plot file format? [png]/pdf/ps/eps/epsi
-output_file_type: png
-# Use netCDF compression true/[false]
-compress_netcdf: false
-# Save intermediary cubes in the preprocessor true/[false]
-save_intermediary_cubes: false
-# Remove the preproc dir if all fine
-remove_preproc_dir: true
-# Path to custom config-developer file, to customise project configurations.
-# See config-developer.yml for an example. Set to [null] to use the default
-# config_developer_file: null
-# Get profiling information for diagnostics
-# Only available for Python diagnostics
-profile_diagnostic: false
-```
-
 > ## Make your own configuration file
 >
 > It is possible to have several configuration files with different purposes,
-> for example: config-user_formalised_runs.yml, config-user_debugging.yml {:
-> .callout}
->
+> for example: config-user_formalised_runs.yml, config-user_debugging.yml 
+{: .callout}
+
 > ## Saving preprocessed data
 >
 > In the configuration file, which settings are useful to make sure preprocessed
diff --git a/_episodes/04-recipe.md b/_episodes/04-recipe.md
@@ -76,7 +76,14 @@ environment, (see episode #3 LINK), ESMValTool is invoked using a simple
 command:
 
 ~~~
-esmvaltool -c configuration recipe
+esmvaltool run recipe
+~~~
+{: .source}
+
+If the configuration file is not in the default {HOME}\.esmvaltool\ path you can pass its path explicitly:
+
+~~~
+esmvaltool run --config_file configuration recipe
 ~~~
 {: .source}
 
@@ -235,7 +242,7 @@ Please note the following sections:
 >
 > Use the command:
 > ~~~bash
-> esmvaltool -c ./path_to_file/user-config.yml ./path_to_file/recipe_example.yml
+> esmvaltool run --config_file ./path_to_file/user-config.yml ./path_to_file/recipe_example.yml
 > ~~~
 >
 > Follow the terminal guiding you through the subprocesses that are running. Can
diff --git a/_episodes/06-debugging.md b/_episodes/06-debugging.md
@@ -27,12 +27,12 @@ directory ``esmvaltool_tutorial`` that was created during the
 [Configuration]({{ page.root }}{% link _episodes/03-configuration.md %}).
 
 In a new terminal, go to our working directory ``esmvaltool_tutorial`` where
-both files ``user-config.yml`` and ``recipe_example.yml`` are located and run
+the file ``recipe_example.yml`` is located and run
 the recipe:
 
 ~~~bash
   cd esmvaltool_tutorial
-  esmvaltool -c user-config.yml recipe_example.yml
+  esmvaltool run recipe_example.yml
 ~~~
 
 ~~~
@@ -202,7 +202,7 @@ Also, we change the ``projects`` value ``ukesm`` to ``tutorial``:
 
 Then, we save the file and run the recipe:
 ~~~bash
-  esmvaltool -c user-config.yml recipe_example.yml
+  esmvaltool run recipe_example.yml
 ~~~
 
 ~~~
@@ -216,10 +216,17 @@ attach the run/recipe_*.yml and run/main_log_debug.txt files from the output dir
 The values for the keys ``author``, ``maintainer``, ``projects`` and
 ``references`` in the recipe should be known by ESMValTool. A list of ESMValTool
 author, maintainer, references, and projects can be found in the
-``config-references.yml`` that is located in the folder ``references`` in the
-ESMValTool installation directory ``path_to_esmvaltool``. To find
-``path_to_esmvaltool`` on your system, see [Installation]({{ page.root }}{% link
-_episodes/02-installation.md %}).
+``config-references.yml``. You could download this file by typing:
+
+~~~bash
+  esmvaltool config get_config_reference
+~~~
+
+It will be saved to: ``{HOME}/.esmvaltool/config-references.yml``. If you modify this file, please add the directory in the ``config-user.yml``:
+
+```yaml
+config_developer_file: {HOME}/.esmvaltool/config_reference.yml
+```
 
 > ## ESMValTool can’t locate the data
 >
@@ -282,7 +289,7 @@ let's see if we can change the script path as:
 ```
 
 ~~~bash
-  esmvaltool -c user-config.yml recipe_example.yml
+  esmvaltool run recipe_example.yml
 ~~~
 
 ~~~
@@ -304,7 +311,7 @@ recipe:
 ```
 
 ~~~bash
-  esmvaltool -c user-config.yml recipe_example.yml
+  esmvaltool run recipe_example.yml
 ~~~
 
 > ## Available recipe and diagnostic scripts
