Skip to content

Commit 1767f75

Browse files
committed
fix markdown problem with title
1 parent 7ed2c5c commit 1767f75

File tree

1 file changed

+80
-27
lines changed

1 file changed

+80
-27
lines changed

_episodes/03-configuration.md

Lines changed: 80 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,22 @@ keypoints:
1616

1717
## The configuration file
1818

19-
The ``config-user.yml`` configuration file contains all the global level information needed by ESMValTool to run.
20-
This is an [YAML file](https://yaml.org/spec/1.2/spec.html). An example configuration file can be found in the root directory of the ESMValTool repository: [config-user-example.yml](https://github.com/ESMValGroup/ESMValTool/blob/master/config-user-example.yml).
21-
22-
Let's download it to our working directory ``esmvaltool_tutorial`` that is made during the [Setup](https://esmvalgroup.github.io/tutorial/setup.html).
23-
To do that, click on [this link](https://raw.githubusercontent.com/ESMValGroup/ESMValTool/master/config-user-example.yml) to see a raw version of the file, right-click and press ``save as``, then you can rename it to ``config-user.yml`` and save it into the working directory ``esmvaltool_tutorial``.
24-
25-
Now in a terminal, let's change our working directory to ``esmvaltool_tutorial``. Then, we run a text editor called Nano to open the configuration file:
19+
The ``config-user.yml`` configuration file contains all the global level information
20+
needed by ESMValTool to run. This is an
21+
[YAML file](https://yaml.org/spec/1.2/spec.html). An example configuration file
22+
can be found in the root directory of the ESMValTool repository:
23+
[config-user-example.yml](https://github.com/ESMValGroup/ESMValTool/blob/master/config-user-example.yml).
24+
25+
Let's download it to our working directory ``esmvaltool_tutorial``
26+
that is made during the [Setup](https://esmvalgroup.github.io/tutorial/setup.html).
27+
To do that, click on
28+
[this link](https://raw.githubusercontent.com/ESMValGroup/ESMValTool/master/config-user-example.yml)
29+
to see a raw version of the file, right-click and press ``save as``,
30+
then you can rename it to ``config-user.yml``and save it into the working directory
31+
``esmvaltool_tutorial``.
32+
33+
Now in a terminal, let's change our working directory to ``esmvaltool_tutorial``.
34+
Then, we run a text editor called Nano to open the configuration file:
2635

2736
~~~bash
2837
cd esmvaltool_tutorial
@@ -40,13 +49,22 @@ This file contains the information for:
4049

4150
> ## Which text editor
4251
>
43-
> No matter what editor you use, you will need to know where it searches for and saves files. If you start it from the shell, it will (probably) use your current working directory as its default location. We use ``nano`` in examples because it is one of the least complex text editors. Press <kbd>ctrl</kbd> + <kbd>O</kbd> to save the file, and then <kbd>ctrl</kbd> + <kbd>X</kbd> to exit ``nano``.
52+
> No matter what editor you use, you will need to know where it searches
53+
for and saves files. If you start it from the shell, it will (probably)
54+
use your current working directory as its default location. We use ``nano``
55+
in examples because it is one of the least complex text editors.
56+
Press <kbd>ctrl</kbd> + <kbd>O</kbd> to save the file,
57+
and then <kbd>ctrl</kbd> + <kbd>X</kbd> to exit ``nano``.
4458
{: .callout}
4559

4660
## Rootpath to input data
4761

48-
ESMValTool uses several categories (in ESMValTool, this is referred to as projects) for input data based on their source.
49-
The current categories in the configuration file are mentioned below. For example, CMIP is used for a dataset from the climate model intercomparison project whereas OBS for an observational dataset. The ``rootpath`` specifies the directories where ESMValTool will look for input data. For each category, you can define either one path or several paths as a list.
62+
ESMValTool uses several categories (in ESMValTool, this is referred to as projects)
63+
for input data based on their source. The current categories in the configuration
64+
file are mentioned below. For example, CMIP is used for a dataset from
65+
the climate model intercomparison project whereas OBS for an observational dataset.
66+
The ``rootpath`` specifies the directories where ESMValTool will look for input data.
67+
For each category, you can define either one path or several paths as a list.
5068

5169
~~~YAML
5270
rootpath:
@@ -62,7 +80,8 @@ rootpath:
6280
default: ~/default_inputpath
6381
~~~
6482

65-
In this lesson, we will work with data from [CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
83+
In this lesson, we will work with data from
84+
[CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
6685
We add the root path of the folder where data is available.
6786

6887
~~~YAML
@@ -73,13 +92,18 @@ We add the root path of the folder where data is available.
7392

7493
> ## Setting the correct rootpath
7594
>
76-
> * To get the data (or its correct rootpath), check instruction in [Setup](https://esmvalgroup.github.io/tutorial/setup.html).
77-
> * For more information about setting the rootpath, you can visit ESMValTool [documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
95+
> * To get the data (or its correct rootpath), check instruction in
96+
[Setup](https://esmvalgroup.github.io/tutorial/setup.html).
97+
> * For more information about setting the rootpath, you can visit the ESMValTool
98+
[documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
7899
{: .callout}
79100

80101
## Directory structure for the data from different projects
81102

82-
Input data can be from various models, observations and reanalysis data that adhere to the [CF/CMOR standard](https://cmor.llnl.gov/). To set a directory, you can use one of the values of ``default``, ``BADC``, ``DKRZ``, ``ETHZ``, .... Let's use ``default`` in our example:
103+
Input data can be from various models, observations and reanalysis data that adhere
104+
to the [CF/CMOR standard](https://cmor.llnl.gov/).
105+
To set a directory, you can use one of the values of
106+
``default``, ``BADC``, ``DKRZ``, ``ETHZ``, .... Let's use ``default`` in our example:
83107

84108
~~~YAML
85109
drs:
@@ -88,12 +112,16 @@ drs:
88112

89113
> ## Available drs
90114
>
91-
> For more information about directories, you can visit ESMValTool [documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/config.html#developer-configuration-file).
115+
> For more information about directories, you can visit the ESMValTool
116+
[documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/config.html#developer-configuration-file).
92117
{: .callout}
93118

94119
## Number of parallel tasks
95120

96-
This option enables you to perform parallel processing. You can choose the number of tasks in parallel as 1/2/3/4/... or you can set it to ``null`` that tells ESMValTool to use the number of available CPUs:
121+
This option enables you to perform parallel processing.
122+
You can choose the number of tasks in parallel as
123+
1/2/3/4/... or you can set it to ``null`` that tells
124+
ESMValTool to use the number of available CPUs:
97125

98126
~~~YAML
99127

@@ -102,12 +130,20 @@ max_parallel_tasks: null
102130

103131
> ## Set the number of tasks
104132
>
105-
> If you run out of memory, try setting ``max_parallel_tasks`` to 1. Then, check the amount of memory you need for that by inspecting the file ``run/resource_usage.txt`` in the output directory. Using the number there you can increase the number of parallel tasks again to a reasonable number for the amount of memory available in your system.
133+
> If you run out of memory, try setting ``max_parallel_tasks`` to 1.
134+
Then, check the amount of memory you need for that by inspecting
135+
the file ``run/resource_usage.txt`` in the output directory.
136+
Using the number there you can increase the number of parallel tasks
137+
again to a reasonable number for the amount of memory available in your system.
106138
{: .callout}
107139

108140
## Destination directory
109141

110-
The destination directory is the rootpath where ESMValTool will store its output, i.e. figures, data, logs, etc. With every run, ESMValTool automatically generates a new output folder determined by the recipe name, and date and time using the format: YYYYMMDD_HHMMSS. This folder contains four further subfolders: ``plots``, ``preproc``, ``run``, ``work``.
142+
The destination directory is the rootpath where ESMValTool will store its output,
143+
i.e. figures, data, logs, etc. With every run, ESMValTool automatically generates
144+
a new output folder determined by the recipe name, and date and time using
145+
the format: YYYYMMDD_HHMMSS.
146+
This folder contains four further subfolders: ``plots``, ``preproc``, ``run``, ``work``.
111147

112148
Let's name our destination directory as ``esmvaltool_output`` in the working directory:
113149

@@ -118,24 +154,36 @@ output_dir: ./esmvaltool_output
118154
> ## Content of subfolders
119155
>
120156
> * ``plots``: the location for all plots, split by individual diagnostics and fields.
121-
> * ``preproc``: this folder contains all the preprocessed data and metadata.yml interface files. Note that by default this directory will be deleted after each run because most users will only need the results from the diagnostic scripts.
122-
> * ``run``: this folder includes all log files, a copy of the recipe, a summary of the resource usage, and the settings.yml interface files, resource_usage.txt and temporary files created by the diagnostic scripts.
123-
> * ``work``: a place for any diagnostic script results that are not plots, e.g. files in NetCDF format (depends on the diagnostic script).
157+
> * ``preproc``: this folder contains all the preprocessed data and metadata.yml
158+
interface files. Note that by default this directory will be deleted after
159+
each run because most users will only need the results from the diagnostic scripts.
160+
> * ``run``: this folder includes all log files, a copy of the recipe,
161+
a summary of the resource usage, and the settings.yml interface files,
162+
resource_usage.txt and temporary files created by the diagnostic scripts.
163+
> * ``work``: a place for any diagnostic script results that are not plots, e.g.
164+
files in NetCDF format (depends on the diagnostic script).
124165
>
125-
> We explain more about output in the next [lesson](https://esmvalgroup.github.io/tutorial/04-toy-example/index.html)
166+
> We explain more about output in the next
167+
[lesson](https://esmvalgroup.github.io/tutorial/04-toy-example/index.html)
126168
{: .callout}
127169

128170
## Auxiliary data directory
129171

130-
The ``auxiliary_data_dir`` setting is the path to place any required additional auxiliary data files. This location allows us to tell the diagnostic script where to find the files if they can not be downloaded at runtime. This option is not for model or observational datasets, rather it is for data files used in plotting such as coastline descriptions and so on.
172+
The ``auxiliary_data_dir`` setting is the path to place any required
173+
additional auxiliary data files. This location allows us to tell
174+
the diagnostic script where to find the files if they can not be downloaded
175+
at runtime. This option is not for model or observational datasets, rather
176+
it is for data files used in plotting such as coastline descriptions and so on.
131177

132178
~~~YAML
133179
auxiliary_data_dir: ~/auxiliary_data
134180
~~~
135181

136182
## Output settings
137183

138-
These settings are used to inform ESMValTool about your preference. You can turn on or off the setting by ``true`` or ``false`` values. Most of these settings are fairly self-explanatory, ie:
184+
These settings are used to inform ESMValTool about your preference.
185+
You can turn on or off the setting by ``true`` or ``false`` values.
186+
Most of these settings are fairly self-explanatory, ie:
139187

140188
~~~YAML
141189
# Diagnostics create plots? [true]/false
@@ -164,16 +212,21 @@ profile_diagnostic: false
164212

165213
> ## Make your own configuration file
166214
>
167-
> It is possible to have several configuration files with different purposes, for example: config-user_formalised_runs.yml, config-user_debugging.yml
215+
> It is possible to have several configuration files with different purposes,
216+
for example: config-user_formalised_runs.yml, config-user_debugging.yml
168217
{: .callout}
169-
218+
>
170219
> ## Different settings
171220
>
172221
> In the configuration file, which settings are useful to store preprocessed data?
173222
>
174223
>> ## Solution
175224
>>
176-
>> If the option ``save_intermediary_cubes`` is set to true in the config-user.yml file, then the intermediary cubes will also be saved in the folder ``preproc``. Also, if the option ``remove_preproc_dir`` is set to ``false``, then the ``preproc/`` directory contains all the preprocessed data and the metadata interface files.
225+
>> If the option ``save_intermediary_cubes`` is set to true in
226+
the config-user.yml file, then the intermediary cubes will also be saved
227+
in the folder ``preproc``. Also, if the option ``remove_preproc_dir``
228+
is set to ``false``, then the ``preproc/`` directory contains all
229+
the preprocessed data and the metadata interface files.
177230
> {: .solution}
178231
{: .challenge}
179232

0 commit comments

Comments
 (0)