You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _episodes/03-configuration.md
+80-27Lines changed: 80 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,13 +16,22 @@ keypoints:
16
16
17
17
## The configuration file
18
18
19
-
The ``config-user.yml`` configuration file contains all the global level information needed by ESMValTool to run.
20
-
This is an [YAML file](https://yaml.org/spec/1.2/spec.html). An example configuration file can be found in the root directory of the ESMValTool repository: [config-user-example.yml](https://github.com/ESMValGroup/ESMValTool/blob/master/config-user-example.yml).
21
-
22
-
Let's download it to our working directory ``esmvaltool_tutorial`` that is made during the [Setup](https://esmvalgroup.github.io/tutorial/setup.html).
23
-
To do that, click on [this link](https://raw.githubusercontent.com/ESMValGroup/ESMValTool/master/config-user-example.yml) to see a raw version of the file, right-click and press ``save as``, then you can rename it to ``config-user.yml`` and save it into the working directory ``esmvaltool_tutorial``.
24
-
25
-
Now in a terminal, let's change our working directory to ``esmvaltool_tutorial``. Then, we run a text editor called Nano to open the configuration file:
19
+
The ``config-user.yml`` configuration file contains all the global level information
20
+
needed by ESMValTool to run. This is an
21
+
[YAML file](https://yaml.org/spec/1.2/spec.html). An example configuration file
22
+
can be found in the root directory of the ESMValTool repository:
to see a raw version of the file, right-click and press ``save as``,
30
+
then you can rename it to ``config-user.yml``and save it into the working directory
31
+
``esmvaltool_tutorial``.
32
+
33
+
Now in a terminal, let's change our working directory to ``esmvaltool_tutorial``.
34
+
Then, we run a text editor called Nano to open the configuration file:
26
35
27
36
~~~bash
28
37
cd esmvaltool_tutorial
@@ -40,13 +49,22 @@ This file contains the information for:
40
49
41
50
> ## Which text editor
42
51
>
43
-
> No matter what editor you use, you will need to know where it searches for and saves files. If you start it from the shell, it will (probably) use your current working directory as its default location. We use ``nano`` in examples because it is one of the least complex text editors. Press <kbd>ctrl</kbd> + <kbd>O</kbd> to save the file, and then <kbd>ctrl</kbd> + <kbd>X</kbd> to exit ``nano``.
52
+
> No matter what editor you use, you will need to know where it searches
53
+
for and saves files. If you start it from the shell, it will (probably)
54
+
use your current working directory as its default location. We use ``nano``
55
+
in examples because it is one of the least complex text editors.
56
+
Press <kbd>ctrl</kbd> + <kbd>O</kbd> to save the file,
57
+
and then <kbd>ctrl</kbd> + <kbd>X</kbd> to exit ``nano``.
44
58
{: .callout}
45
59
46
60
## Rootpath to input data
47
61
48
-
ESMValTool uses several categories (in ESMValTool, this is referred to as projects) for input data based on their source.
49
-
The current categories in the configuration file are mentioned below. For example, CMIP is used for a dataset from the climate model intercomparison project whereas OBS for an observational dataset. The ``rootpath`` specifies the directories where ESMValTool will look for input data. For each category, you can define either one path or several paths as a list.
62
+
ESMValTool uses several categories (in ESMValTool, this is referred to as projects)
63
+
for input data based on their source. The current categories in the configuration
64
+
file are mentioned below. For example, CMIP is used for a dataset from
65
+
the climate model intercomparison project whereas OBS for an observational dataset.
66
+
The ``rootpath`` specifies the directories where ESMValTool will look for input data.
67
+
For each category, you can define either one path or several paths as a list.
50
68
51
69
~~~YAML
52
70
rootpath:
@@ -62,7 +80,8 @@ rootpath:
62
80
default: ~/default_inputpath
63
81
~~~
64
82
65
-
In this lesson, we will work with data from [CMIP5](https://esgf-node.llnl.gov/projects/cmip5/).
We add the root path of the folder where data is available.
67
86
68
87
~~~YAML
@@ -73,13 +92,18 @@ We add the root path of the folder where data is available.
73
92
74
93
> ## Setting the correct rootpath
75
94
>
76
-
> * To get the data (or its correct rootpath), check instruction in [Setup](https://esmvalgroup.github.io/tutorial/setup.html).
77
-
> * For more information about setting the rootpath, you can visit ESMValTool [documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/datafinder.html).
95
+
> * To get the data (or its correct rootpath), check instruction in
## Directory structure for the data from different projects
81
102
82
-
Input data can be from various models, observations and reanalysis data that adhere to the [CF/CMOR standard](https://cmor.llnl.gov/). To set a directory, you can use one of the values of ``default``, ``BADC``, ``DKRZ``, ``ETHZ``, .... Let's use ``default`` in our example:
103
+
Input data can be from various models, observations and reanalysis data that adhere
104
+
to the [CF/CMOR standard](https://cmor.llnl.gov/).
105
+
To set a directory, you can use one of the values of
106
+
``default``, ``BADC``, ``DKRZ``, ``ETHZ``, .... Let's use ``default`` in our example:
83
107
84
108
~~~YAML
85
109
drs:
@@ -88,12 +112,16 @@ drs:
88
112
89
113
> ## Available drs
90
114
>
91
-
> For more information about directories, you can visit ESMValTool [documentation](https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/config.html#developer-configuration-file).
115
+
> For more information about directories, you can visit the ESMValTool
This option enables you to perform parallel processing. You can choose the number of tasks in parallel as 1/2/3/4/... or you can set it to ``null`` that tells ESMValTool to use the number of available CPUs:
121
+
This option enables you to perform parallel processing.
122
+
You can choose the number of tasks in parallel as
123
+
1/2/3/4/... or you can set it to ``null`` that tells
124
+
ESMValTool to use the number of available CPUs:
97
125
98
126
~~~YAML
99
127
@@ -102,12 +130,20 @@ max_parallel_tasks: null
102
130
103
131
> ## Set the number of tasks
104
132
>
105
-
> If you run out of memory, try setting ``max_parallel_tasks`` to 1. Then, check the amount of memory you need for that by inspecting the file ``run/resource_usage.txt`` in the output directory. Using the number there you can increase the number of parallel tasks again to a reasonable number for the amount of memory available in your system.
133
+
> If you run out of memory, try setting ``max_parallel_tasks`` to 1.
134
+
Then, check the amount of memory you need for that by inspecting
135
+
the file ``run/resource_usage.txt`` in the output directory.
136
+
Using the number there you can increase the number of parallel tasks
137
+
again to a reasonable number for the amount of memory available in your system.
106
138
{: .callout}
107
139
108
140
## Destination directory
109
141
110
-
The destination directory is the rootpath where ESMValTool will store its output, i.e. figures, data, logs, etc. With every run, ESMValTool automatically generates a new output folder determined by the recipe name, and date and time using the format: YYYYMMDD_HHMMSS. This folder contains four further subfolders: ``plots``, ``preproc``, ``run``, ``work``.
142
+
The destination directory is the rootpath where ESMValTool will store its output,
143
+
i.e. figures, data, logs, etc. With every run, ESMValTool automatically generates
144
+
a new output folder determined by the recipe name, and date and time using
145
+
the format: YYYYMMDD_HHMMSS.
146
+
This folder contains four further subfolders: ``plots``, ``preproc``, ``run``, ``work``.
111
147
112
148
Let's name our destination directory as ``esmvaltool_output`` in the working directory:
> * ``plots``: the location for all plots, split by individual diagnostics and fields.
121
-
> * ``preproc``: this folder contains all the preprocessed data and metadata.yml interface files. Note that by default this directory will be deleted after each run because most users will only need the results from the diagnostic scripts.
122
-
> * ``run``: this folder includes all log files, a copy of the recipe, a summary of the resource usage, and the settings.yml interface files, resource_usage.txt and temporary files created by the diagnostic scripts.
123
-
> * ``work``: a place for any diagnostic script results that are not plots, e.g. files in NetCDF format (depends on the diagnostic script).
157
+
> * ``preproc``: this folder contains all the preprocessed data and metadata.yml
158
+
interface files. Note that by default this directory will be deleted after
159
+
each run because most users will only need the results from the diagnostic scripts.
160
+
> * ``run``: this folder includes all log files, a copy of the recipe,
161
+
a summary of the resource usage, and the settings.yml interface files,
162
+
resource_usage.txt and temporary files created by the diagnostic scripts.
163
+
> * ``work``: a place for any diagnostic script results that are not plots, e.g.
164
+
files in NetCDF format (depends on the diagnostic script).
124
165
>
125
-
> We explain more about output in the next [lesson](https://esmvalgroup.github.io/tutorial/04-toy-example/index.html)
The ``auxiliary_data_dir`` setting is the path to place any required additional auxiliary data files. This location allows us to tell the diagnostic script where to find the files if they can not be downloaded at runtime. This option is not for model or observational datasets, rather it is for data files used in plotting such as coastline descriptions and so on.
172
+
The ``auxiliary_data_dir`` setting is the path to place any required
173
+
additional auxiliary data files. This location allows us to tell
174
+
the diagnostic script where to find the files if they can not be downloaded
175
+
at runtime. This option is not for model or observational datasets, rather
176
+
it is for data files used in plotting such as coastline descriptions and so on.
131
177
132
178
~~~YAML
133
179
auxiliary_data_dir: ~/auxiliary_data
134
180
~~~
135
181
136
182
## Output settings
137
183
138
-
These settings are used to inform ESMValTool about your preference. You can turn on or off the setting by ``true`` or ``false`` values. Most of these settings are fairly self-explanatory, ie:
184
+
These settings are used to inform ESMValTool about your preference.
185
+
You can turn on or off the setting by ``true`` or ``false`` values.
186
+
Most of these settings are fairly self-explanatory, ie:
139
187
140
188
~~~YAML
141
189
# Diagnostics create plots? [true]/false
@@ -164,16 +212,21 @@ profile_diagnostic: false
164
212
165
213
> ## Make your own configuration file
166
214
>
167
-
> It is possible to have several configuration files with different purposes, for example: config-user_formalised_runs.yml, config-user_debugging.yml
215
+
> It is possible to have several configuration files with different purposes,
216
+
for example: config-user_formalised_runs.yml, config-user_debugging.yml
168
217
{: .callout}
169
-
218
+
>
170
219
> ## Different settings
171
220
>
172
221
> In the configuration file, which settings are useful to store preprocessed data?
173
222
>
174
223
>> ## Solution
175
224
>>
176
-
>> If the option ``save_intermediary_cubes`` is set to true in the config-user.yml file, then the intermediary cubes will also be saved in the folder ``preproc``. Also, if the option ``remove_preproc_dir`` is set to ``false``, then the ``preproc/`` directory contains all the preprocessed data and the metadata interface files.
225
+
>> If the option ``save_intermediary_cubes`` is set to true in
226
+
the config-user.yml file, then the intermediary cubes will also be saved
227
+
in the folder ``preproc``. Also, if the option ``remove_preproc_dir``
228
+
is set to ``false``, then the ``preproc/`` directory contains all
229
+
the preprocessed data and the metadata interface files.
0 commit comments