@@ -9,6 +9,7 @@ questions:
99- " Can I use different preprocessors for different variables?"
1010- " Can I use different datasets for different variables?"
1111- " How can I combine different preprocessor functions?"
12+ - " Can I run the same recipe for multiple ensemble members?"
1213objectives :
1314- " Create a recipe with multiple preprocessors"
1415- " Use different preprocessors for different variables"
@@ -18,6 +19,7 @@ keypoints:
1819- " The setting `additional_datasets` can be used to add a different dataset."
1920- " Variable groups are useful for defining different settings for different
2021 variables."
22+ - " Multiple ensemble members and experiments can be analysed in a single recipe through concatenation."
2123---
2224
2325## Introduction
@@ -26,7 +28,7 @@ One of the key strengths of ESMValTool is in making complex analyses reusable
2628and reproducible. But that doesn't mean everything in ESMValTool needs to be
2729complex. Sometimes, the biggest challenge is in keeping things simple. You
2830probably know the 'warming stripes' visualization by Professor Ed Hawkins. On
29- the site < https://showyourstripes.info > you can find the same visualization for
31+ the site < https://showyourstripes.info > { : target =" _ blank"} you can find the same visualization for
3032many regions in the world.
3133
3234![ Warming stripes] ( ../fig/warming_stripes.png ) * Shared by Ed Hawkins under a
@@ -42,16 +44,16 @@ The diagnostic script that we will use is called `warming_stripes.py` and
4244can be downloaded [ here] ( ../files/warming_stripes.py ) .
4345
4446Download the file and store it in your working directory. If you want, you may
45- also have a look at the contents, but it is not necessary to follow along .
47+ also have a look at the contents, but it is not necessary to do so for this lesson .
4648
4749We will write an ESMValTool recipe that takes some data, performs the necessary
4850preprocessing, and then runs this Python script.
4951
5052> ## Drawing up a plan
5153>
5254> Previously, we saw that running ESMValTool executes a number of
53- > tasks. Write down what tasks we will need to execute in this episode and what
54- > each of these tasks does ?
55+ > tasks. What tasks do you think we will need to execute and what
56+ > should each of these tasks do to generate the warming stripes ?
5557>
5658> > ## Answer
5759> >
@@ -69,11 +71,11 @@ preprocessing, and then runs this Python script.
6971
7072The easiest way to make a new recipe is to start from an existing one, and
7173modify it until it does exactly what you need. However, in this episode we will
72- start from scratch. This forces us to think about all the steps. We will deal
73- with common errors as they occur throughout the development .
74+ start from scratch. This forces us to think about all the steps involved in processing the data.
75+ We will also deal with commonly occurring errors through the development of the recipe .
7476
75- Remember the basic structure of a recipe, and notice that each of them is
76- extensively described in the documentation following the header
77+ Remember the basic structure of a recipe, and notice that each component is
78+ extensively described in the documentation under the section,
7779 [ "Overview"] [ recipe-overview ] {: target ="_ blank"}:
7880
7981- [ documentation] [ recipe-section-documentation ] {: target ="_ blank"}
@@ -102,11 +104,11 @@ documentation:
102104
103105```
104106
105- Notice that ` yaml ` always requires 2 spaces indentation between the different
107+ Notice that ` yaml ` always requires ` two spaces` indentation between the different
106108levels. Pressing ` ctrl+o ` will save the file. Verify the filename at the bottom
107109and press enter. Then use ` ctrl+x ` to exit the editor.
108110
109- We will try to run the recipe after every modification we make, to see if it (still) works.
111+ We will try to run the recipe after every modification we make, to see if it (still) works!
110112
111113``` bash
112114esmvaltool run recipe_warming_stripes.yml
@@ -132,9 +134,13 @@ files run/recipe_*.yml and run/main_log_debug.txt from the output directory.
132134```
133135{: .error}
134136
135- Here, ESMValTool is telling us that it is missing a required field, namely the
136- authors. We see that ESMValTool always tries to validate the recipe
137- at an early stage.
137+ We can use the the log message above, to understand why ESMValTool failed. Here, this is because
138+ we missed a required field with author names. The text ` documentation.authors: Required field missing `
139+ tells us that. We see that ESMValTool always tries to validate the recipe
140+ at an early stage. Note also the suggestion to open a GitHub issue if
141+ you need help debugging the error message. This is something most
142+ users do when they cannot understand the error or are not able to fix it
143+ on their own.
138144
139145Let's add some additional information to the recipe. Open the recipe file again,
140146and add an authors section below the description. ESMValTool expects the authors
@@ -238,8 +244,8 @@ Let's add a datasets section.
238244> {: .solution}
239245 {: .challenge}
240246
241- We start with the BCC-ESM1 dataset and add a datasets section to the recipe,
242- listing a single dataset, as shown below. Note that key fields such
247+ Let us start with the BCC-ESM1 dataset and add a datasets section to the recipe,
248+ listing this single dataset, as shown below. Note that key fields such
243249as ` mip ` or ` start_year ` are included in the ` datasets ` section here but are part
244250of the ` diagnostic ` section in the recipe example seen in
245251[ Running your first recipe] ({{ page.root }}{% link _ episodes/04-recipe.md %}).
@@ -277,7 +283,7 @@ in ESMValTool.
277283> You can select all available models for processing using
278284> ` glob ` patterns or wildcards. An example ` datasets ` section that uses all
279285> available CMIP6 models and ensemble members for the ` historical ` experiment
280- > will look is available [ here] [ include-all-datasets] {: target ="_ blank"}.
286+ > is available [ here] [ include-all-datasets] {: target ="_ blank"}.
281287> Note that you will have to set the ` search_esgf ` option in the ` config_file ` to
282288> ` always ` so that you can download data from ESGF nodes as needed.
283289 {: .callout}
@@ -302,15 +308,16 @@ standard, gridded temperature data to a timeseries of temperature anomalies.
302308> > ## Solution
303309> >
304310> > We need to calculate anomalies and global means. There is an ` anomalies `
305- > > preprocessor which needs a granularity , a reference period, and whether or
311+ > > preprocessor which takes in as arguments, a time period , a reference period, and whether or
306312> > not to standardize the data. The global means can be calculated with the
307313> > ` area_statistics ` preprocessor, which takes an operator as argument (in our
308314> > case we want to compute the ` mean ` ).
309315> >
310316> > The default order in which these preprocessors are applied can be seen
311317> > [ here] [ preprocessor-functions ] {: target ="_ blank"}:
312318> > ` area_statistics ` comes before ` anomalies ` . If you want to change this, you
313- > > can use the ` custom_order ` preprocessor. We will keep it like this.
319+ > > can use the ` custom_order ` preprocessor as described [ here] [ recipe-section-preprocessors ] {: target ="_ blank"}.
320+ >> For this example, we will keep the default order..
314321> >
315322> > Let's name our preprocessor ` global_anomalies ` .
316323> {: .solution}
@@ -448,8 +455,9 @@ for each of the modifications we'll make below.
448455> # # Different time periods
449456>
450457> Split the diagnostic in two with two different time periods for the same variable.
451- > You can choose the time periods yourself. For example, the recent past and
452- > the 20th century. You can do this by using variable groups.
458+ > You can choose the time periods yourself. In the example below, we have
459+ > chosen the recent past and
460+ > the 20th century and have used variable grouping.
453461>
454462> > # # Solution
455463> >
@@ -552,17 +560,20 @@ for each of the modifications we'll make below.
552560>
553561{:.challenge}
554562
555- >
563+
564+
565+
556566> # # Pro-tip: YAML anchors
557- > If you want to avoid repetition, you can use YAML anchors as seen in the recipe above.
567+ > If you want to avoid retyping the arguments used in your preprocessor, you can use YAML anchors as
568+ seen in the `anomalies` preprocessor specifications in the recipe above.
558569{:.callout}
559570
560571> # # Additional datasets
561572>
562573> So far we have defined the datasets in the datasets section of the recipe.
563574> However, it's also possible to add specific datasets only for specific
564- > variable groups. Look at the documentation to learn about the
565- > ' additional_datasets' keyword, and add a second dataset only for one of the
575+ > variables or variable groups. Take a look at the documentation to learn about the
576+ > ` additional_datasets` keyword [here][additional-datasets]{:target="_blank"} , and add a second dataset only for one of the
566577> variable groups.
567578>
568579> > # # Solution
0 commit comments