Skip to content

Commit 5ff6b53

Browse files
Peter9192Peter9192
authored
Merge pull request #51 from ESMValGroup/episode06_debugging
Add episode06 debugging
2 parents a674b4c + 001e549 commit 5ff6b53

File tree

2 files changed

+335
-19
lines changed

2 files changed

+335
-19
lines changed

‎_episodes/06-creating-a-diagnostic-script.md‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎_episodes/06-debugging.md‎

Lines changed: 335 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,335 @@
1+
---
2+
title: "Debugging"
3+
teaching: 30
4+
exercises: 15
5+
questions:
6+
- "How can I handle errors/warnings?"
7+
objectives:
8+
- "Fix a broken recipe"
9+
keypoints:
10+
- "There are three different kinds of log files: ``main_log.txt``, and ``main_log_debug.txt`` and ``log.txt``."
11+
---
12+
13+
Every user encounters errors. Once you know why you get certain types of errors,
14+
they become much easier to fix.
15+
The good news is, ESMValTool creates a record of the output messages and stores them in log files.
16+
They can be used for debugging or monitoring the process.
17+
This lesson helps to understand what the different types of errors are and when you are likely to encounter them.
18+
19+
## Log files
20+
21+
Each time we run ESMValTool, it will produce a new output directory.
22+
This directory should contain the ``run`` folder that is automatically generated by ESMValTool.
23+
To examine this, we run a ``recipe_example.yml`` that can be found in
24+
[Setup]({{ page.root }}{% link setup.md %}).
25+
Let's download it to our working directory ``esmvaltool_tutorial`` that was created during
26+
the [Configuration]({{ page.root }}{% link _episodes/03-configuration.md %}).
27+
28+
In a new terminal, go to our working directory ``esmvaltool_tutorial`` where
29+
both files ``user-config.yml`` and ``recipe_example.yml`` are located and run the recipe:
30+
31+
~~~bash
32+
cd esmvaltool_tutorial
33+
esmvaltool -c user-config.yml recipe_example.yml
34+
~~~
35+
36+
~~~
37+
esmvaltool: command not found
38+
~~~
39+
{: .error}
40+
41+
ESMValTool encounters this error because
42+
the conda environment ``esmvaltool`` has not been activated.
43+
To fix the error, before running the recipe, activate the environment:
44+
45+
~~~bash
46+
conda activate esmvaltool
47+
~~~
48+
49+
> ## conda environment
50+
>
51+
> More information about conda environment can be found at
52+
[Installation]({{ page.root }}{% link _episodes/02-installation.md %})
53+
{: .callout}
54+
55+
Let's change the working directory to the folder ``run`` and list its files:
56+
57+
~~~bash
58+
cd path_to_output_directory/run
59+
ls
60+
~~~
61+
62+
~~~
63+
diag_timeseries_temperature main_log_debug.txt main_log.txt recipe_example.yml resource_usage.txt
64+
~~~
65+
{: .output}
66+
67+
In the ``main_log_debug.txt`` and ``main_log.txt``, ESMValTool writes the output messages,
68+
warnings and possible errors that might happen during pre-processings.
69+
To inspect them, we can look inside the files. For example:
70+
71+
~~~bash
72+
cat main_log.txt
73+
~~~
74+
75+
Now, let's have a look inside the folder ``diag_timeseries_temperature``:
76+
77+
~~~bash
78+
cd path_to_output_directory/run/diag_timeseries_temperature
79+
ls
80+
~~~
81+
82+
~~~
83+
diag_provenance.yml log.txt resource_usage.txt settings.yml
84+
~~~
85+
{: .output}
86+
87+
In the ``log.txt``, ESMValTool writes the output messages,
88+
warnings and possible errors that are related to the diagnostic script.
89+
90+
If you encounter an error and don’t know what it means, it is important to read the log information.
91+
Sometimes knowing where the error occurred is enough to fix it, even if you don’t entirely understand the message.
92+
However, note that you may not always be able to find the error or fix it.
93+
In that case, ESMValTool community helps you figure out what went wrong.
94+
95+
> ## Different log files
96+
>
97+
> In the ``run`` directory, there are two log files ``main_log_debug.txt`` and ``main_log.txt``.
98+
What are their differences?
99+
>
100+
>> ## Solution
101+
>>
102+
>> The ``main_log_debug.txt`` contains the output messages from the pre-processor whereas
103+
the ``main_log.txt`` shows general errors and warnings that might happen in running the recipe and diagnostics script.
104+
> {: .solution}
105+
{: .challenge}
106+
107+
Let's change some settings in the recipe to run a regional pre-processor.
108+
We use a text editor called ``nano`` to open the recipe file:
109+
110+
~~~bash
111+
cd esmvaltool_tutorial
112+
nano recipe_example.yml
113+
~~~
114+
115+
> ## Text editor side note
116+
>
117+
> No matter what editor you use, you will need to know where it searches
118+
for and saves files. If you start it from the shell, it will (probably)
119+
use your current working directory as its default location. We use ``nano``
120+
in examples here because it is one of the least complex text editors.
121+
Press <kbd>ctrl</kbd> + <kbd>O</kbd> to save the file,
122+
and then <kbd>ctrl</kbd> + <kbd>X</kbd> to exit ``nano``.
123+
{: .callout}
124+
125+
>## See the recipe_example.yml
126+
>~~~YAML
127+
>01 # ESMValTool
128+
>02 # recipe_example.yml
129+
>03 ---
130+
>04 documentation:
131+
>05 description: Demonstrate basic ESMValTool example
132+
>06
133+
>07 authors:
134+
>08 - demora_lee
135+
>09 - mueller_benjamin
136+
>10 - swaminathan_ranjini
137+
>11
138+
>12 maintainer:
139+
>13 - demora_lee
140+
>14
141+
>15 references:
142+
>16 - demora2018gmd
143+
>17 # Some plots also appear in ESMValTool paper 2.
144+
>18
145+
>19 projects:
146+
>20 - ukesm
147+
>21
148+
>22 datasets:
149+
>23 - {dataset: HadGEM2-ES, project: CMIP5, exp: historical, mip: Omon, ensemble: r1i1p1, start_year: 1859, end_year: 2005}
150+
>24
151+
>25 preprocessors:
152+
>26 prep_timeseries: # For 0D fields
153+
>27 annual_statistics:
154+
>28 operator: mean
155+
>29
156+
>30 diagnostics:
157+
>31 # --------------------------------------------------
158+
>32 # Time series diagnostics
159+
>33 # --------------------------------------------------
160+
>34 diag_timeseries_temperature:
161+
>35 description: simple_time_series
162+
>36 variables:
163+
>37 timeseries_variable:
164+
>38 short_name: thetaoga
165+
>39 preprocessor: prep_timeseries
166+
>40 scripts:
167+
>41 timeseries_diag:
168+
>42 script: ocean/diagnostic_timeseries.py
169+
>~~~
170+
{: .solution}
171+
172+
## Keys and values in recipe settings
173+
174+
The [ESMValTool pre-processors](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/preprocessor.html?highlight=preprocessor) cover a broad range of operations on the input data,
175+
like time manipulation, area manipulation, land-sea masking, variable derivation, ... .
176+
Let's add the preprocessor ``extract_region`` to the section ``prep_timeseries``:
177+
178+
~~~YAML
179+
25 preprocessors:
180+
26 prep_timeseries: # For 0D fields
181+
27 annual_statistics:
182+
28 operator: mean
183+
29 extract_region:
184+
30 start_longitude: -10
185+
31 end_longitude: 40
186+
32 start_latitude: 27
187+
33 end_latitude: 70
188+
~~~
189+
190+
Also, we change the ``projects`` value ``ukesm`` to ``tutorial``:
191+
192+
~~~YAML
193+
19 projects:
194+
20 - tutorial
195+
~~~
196+
197+
Then, we save the file and run the recipe:
198+
~~~bash
199+
esmvaltool -c user-config.yml recipe_example.yml
200+
~~~
201+
202+
~~~
203+
ValueError: Tag 'tutorial' does not exist in section 'projects' of esmvaltool/config-references.yml
204+
2020-06-29 18:09:56,641 UTC [46055] INFO If you suspect this is a bug or need help,
205+
please open an issue on https://github.com/ESMValGroup/ESMValTool/issues and
206+
attach the run/recipe_*.yml and run/main_log_debug.txt files from the output directory.
207+
~~~
208+
{: .error}
209+
210+
The values for the keys ``author``, ``maintainer``, ``projects`` and ``references`` in the recipe should
211+
be known by ESMValTool. A list of ESMValTool author, maintainer, references, and projects can be found
212+
in the ``config-references.yml`` that is located in the folder ``references``
213+
in the ESMValTool installation directory ``path_to_esmvaltool``. To find ``path_to_esmvaltool``
214+
on your system, see
215+
[Installation]({{ page.root }}{% link _episodes/02-installation.md %}).
216+
217+
> ## ESMValTool can’t locate the data
218+
>
219+
> You are assisting a colleague with ESMValTool. The colleague changes the ``project`` entry to ``CMIP6``
220+
and runs the recipe. However, ESMValTool encounters an error like:
221+
>
222+
~~~bash
223+
esmvalcore._recipe_checks.RecipeError: Missing data
224+
2020-06-29 17:26:41,303 UTC [43830] INFO If you suspect this is a bug or need help,
225+
please open an issue on https://github.com/ESMValGroup/ESMValTool/issues and
226+
attach the run/recipe_*.yml and run/main_log_debug.txt files from the output directory.
227+
~~~
228+
What suggestions would you give the researcher for fixing the error?
229+
>
230+
>> ## Solution
231+
>>
232+
>> 1. Inspect ``main_log.txt``
233+
>> 2. Check ``user-config.yml`` to see if the correct directory for input data is introduced
234+
>> 3. Check the available data, regarding exp, mip, ensemble, start_year, and end_year
235+
>> 4. Check the variable name in the ``diag_timeseries_temperature`` section in the recipe
236+
> {: .solution}
237+
{: .challenge}
238+
239+
## Check pre-processed data
240+
241+
The setting ``save_intermediary_cubes`` in the configuration file can be used
242+
to save the pre-processed data. More information about this setting can be found at
243+
[Configuration]({{ page.root }}{% link _episodes/03-configuration.md %}).
244+
245+
> ## save_intermediary_cubes
246+
>
247+
> Note that this setting should be only used for debugging, as it significantly slows down the recipe
248+
and increases disk usage because a lot of output files need to be stored.
249+
{: .callout}
250+
251+
## Check diagnostic script path
252+
253+
The result of the pre-processor is passed to the ``diagnostic_timeseries.py`` script,
254+
that is introduced in the recipe as:
255+
256+
~~~YAML
257+
40 scripts:
258+
41 timeseries_diag:
259+
42 script: ocean/diagnostic_timeseries.py
260+
~~~
261+
262+
The diagnostic scripts are located in the folder ``diag_scripts`` in
263+
the ESMValTool installation directory ``path_to_esmvaltool``.
264+
To find ``path_to_esmvaltool`` on your system,
265+
see [Installation]({{ page.root }}{% link _episodes/02-installation.md %}).
266+
267+
let's see if we can change the script path as:
268+
269+
~~~YAML
270+
40 scripts:
271+
41 timeseries_diag:
272+
42 script: diag_scripts/ocean/diagnostic_timeseries.py
273+
~~~
274+
275+
~~~bash
276+
esmvaltool -c user-config.yml recipe_example.yml
277+
~~~
278+
279+
~~~
280+
esmvalcore._task.DiagnosticError: Cannot execute script 'diag_scripts/examples/diagnostic.py' (esmvaltool/diag_scripts/diag_scripts/examples/diagnostic.py): file does not exist.
281+
2020-06-29 20:39:31,669 UTC [53008] INFO If you suspect this is a bug or need help, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues and attach the run/recipe_*.yml and run/main_log_debug.txt files from the output directory.
282+
~~~
283+
{: .error}
284+
285+
The script path should be relative to ``diag_scripts`` directory.
286+
It means that the script ``diagnostic_timeseries.py`` is located in ``path_to_esmvaltool/diag_scripts/ocean/``.
287+
Alternatively, the script path can be an absolute path.
288+
To examine this, we change the script path and run the recipe:
289+
290+
~~~YAML
291+
40 scripts:
292+
41 timeseries_diag:
293+
42 script: path_to_esmvaltool/diag_scripts/ocean/diagnostic_timeseries.py
294+
~~~
295+
296+
~~~bash
297+
esmvaltool -c user-config.yml recipe_example.yml
298+
~~~
299+
300+
> ## Available recipe and diagnostic scripts
301+
>
302+
> ESMValTool provides a broad suite of
303+
[recipes and diagnostic](https://docs.esmvaltool.org/en/latest/recipes/index.html)
304+
scripts for different disciplines like atmosphere, climate metrics, future projections,
305+
IPCC, land, ocean, ....
306+
{: .callout}
307+
308+
> ## Re-running a diagnostic
309+
>
310+
> Look at the ``main_log.txt`` file and answer the following question:
311+
How to re-run the diagnostic script?
312+
>
313+
>> ## Solution
314+
>>
315+
>> The ``main_log.txt`` file contains information on how to re-run the diagnostic script
316+
without re-running the pre-processors:
317+
>>
318+
~~~bash
319+
2020-06-29 20:36:32,844 UTC [52810] INFO To re-run this diagnostic script, run:
320+
~~~
321+
>>
322+
>> If you run the command in the next line, you will be able to re-run the diagnostic.
323+
> {: .solution}
324+
{: .challenge}
325+
326+
> ## Memory issues
327+
>
328+
> If you run out of memory, try setting ``max_parallel_tasks`` to 1 in the configuration file.
329+
Then, check the amount of memory you need for that by inspecting
330+
the file ``run/resource_usage.txt`` in the output directory.
331+
Using the number there you can increase the number of parallel tasks again to a reasonable number
332+
for the amount of memory available in your system.
333+
{: .callout}
334+
335+
{% include links.md %}

0 commit comments

Comments
 (0)