Made changes from Birgit's revies, added section on grouping variables, altered timing numbers at the top

Author: rswamina

_episodes/05-preprocessor.md

@@ -1,17 +1,18 @@
 ---
 title: "Working with preprocessors"
-teaching: 20?
-exercises: 15?
+teaching: 20
+exercises: 20
 questions:
 - "How do I set up a preprocessor?"
 - "Can I use different preprocessors for different variables?"
 - "Can I use different datasets for different variables?"
+- "How can I combine different preprocessor functions?"
 objectives:
 - "Create a recipe with multiple preprocessors"
 - "Use different preprocessors for different variables"
 - "Run a recipe with variables from different datasets"
 keypoints:
-- "A recipe can run different preprocessors at the same time."
+- "A recipe can work with different preprocessors at the same time."
 - "The setting additional_datasets can be used to add a different dataset."
 - "Variable groups are useful for defining different settings for different variables."
 ---
@@ -38,7 +39,7 @@ Each preprocessor section includes a preprocessor name, a list of preprocessor s
 
 For instance, the 'annual_statistics' with the  'operation: mean' argument preprocessor receives an iris cube, takes the annual average for each year of data in the cube, and returns the processed cube. 
 
-You could use several preprocessor steps listed in the [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html). The standardised interface between the preprocessors allows them to be used modularly - like lego blocks. Almost any conceivable preprocessing order of operations can be performed using ESMValTool preprocessors. 
+You may use one or more of several preprocessors listed in the [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html). The standardised interface between the preprocessors allows them to be used modularly - like lego blocks. Almost any conceivable preprocessing order of operations can be performed using ESMValTool preprocessors. 
 
 > ## The 'custom order' command.
 >
@@ -50,16 +51,16 @@ You could use several preprocessor steps listed in the [documentation](https://d
 >
 >The default preprocessor order is listed in the [ESMValCore preprocessor API page](>https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.preprocessor.html).
 >
-> Also note that not preprocessor operations aren't always commutative - meaning that the order of operations matters. For instance, if you run the preprocessor 'extract_volume' to extract the top 100m of the ocean surface, then 'volume_statistics' to calculate the volume-weighted mean of the data, then your result will differ depending on the order of these two preprocessors. In fact, the 'extract_volume' preprocessor will fail if you try to run it on a 2D dataset.
+> Also note that preprocessor operations aren't always commutative - meaning that the order of operations matters. For instance, when you run the two preprocessors --  'extract_volume' to extract the top 100m of the ocean surface and  'volume_statistics' to calculate the volume-weighted mean of the data, your result will differ depending on the order of these two preprocessors. In fact, the 'extract_volume' preprocessor will fail if you try to run it on a 2D dataset.
 >
 > Changing the order of preprocessors can also speed up your processing. For instance, if you want to extract a two-dimensional layer from a 3D field and re-grid it, the layer extraction should be done first. If you did it the other way around, then the regridding function would be applied to all the layers of your 3D cube and it would take much more time.
 {: .callout}
 
-Some preprocessor stages are always applied and do not need to be called. This includes the preprocessors that load the data, apply the fixes, and save the data file afterwards. They do not need to be explicitly included in recipes. 
+Some preprocessor modeules are always applied and do not need to be called. This includes the preprocessors that load the data, apply any fixes and save the data file afterwards. These do not need to be explicitly included in recipes. 
 
 > ## Exercise: Adding more preprocessor steps
 >
-> Edit the [example recipe](LINK to episode #4) to change the variable in ?thetao? and to add preprocessors which take the average over the latitude and longitude dimensions and the average over the depth. And then run the recipe.
+> Edit the [example recipe](LINK to episode #4) to first change the variable thetao, then add preprocessors to average over the latitude and longitude dimensions and finally average over the depth. Now run the recipe.
 >
 >> ## Solution
 >> 
@@ -78,7 +79,7 @@ Some preprocessor stages are always applied and do not need to be called. This i
 
 ## Using different preprocessors for different variables
 
-You could define different preprocessors with several preprocessor sections (setting different preprocessor names). In the variable section you call the specific preprocessor which should be applied.
+You can also define different preprocessors with several preprocessor sections (setting different preprocessor names). In the variable section you call the specific preprocessor which should be applied.
 
 > ## Example
 >~~~YAML
@@ -91,7 +92,7 @@ You could define different preprocessors with several preprocessor sections (set
 >       operator: mean
 >     area_statistics:
 >       operator: mean
->     Depth_integration:
+>     depth_integration:
 > ---
 > diagnostics:
 >   # --------------------------------------------------      
@@ -114,14 +115,14 @@ You could define different preprocessors with several preprocessor sections (set
 >         short_name: thetao
 >         preprocessor: prep_timeseries_2
 >     scripts:
->          timeseries_diag:
+>       timeseries_diag:
 >         script: ocean/diagnostic_timeseries.py
 >~~~
 >{: .source}
 {: .solution}
 
 >## Challenge : How to write a recipe with multiple preprocessors
-> We now know that a recipe can have more than one diagnostic, variable or preprocessor. As we saw in the examples so far, we can group preprocessors with a single user defined name and can have more than one such preprocessor group in the recipe as well. Write two different preprocessors - one to regrid the data to a 1x1 resolution and the second preprocessor to mask out sea and ice grid cells before regridding to the saem resolution. In the second case, ensure you perform the masking first before regridding (hint: custom order your operations). Now, use the two preprocessors in different diagnostics within the same recipe. You may use any variable(s) of your choice. Once you succeed, try to add new datasets to the same recipe. Placeholders for the different components are provided below:
+> We now know that a recipe can have more than one diagnostic, variable or preprocessor. As we saw in the examples so far, we can group preprocessors with a single user defined name and can have more than one such preprocessor group in the recipe as well. Write two different preprocessors - one to regrid the data to a 1x1 resolution and the second preprocessor to mask out sea and ice grid cells before regridding to the same resolution. In the second case, ensure you perform the masking first before regridding (hint: custom order your operations). Now, use the two preprocessors in different diagnostics within the same recipe. You may use any variable(s) of your choice. Once you succeed, try to add new datasets to the same recipe. Placeholders for the different components are provided below:
 >
 >> ## Recipe
 >>
@@ -168,8 +169,7 @@ You could define different preprocessors with several preprocessor sections (set
 >
 >> ## Solution: 
 >> 
->> Here is one possible way to use two different preprocessors including a 
->> group of preprocessors on different variables.
+>> Here is one solution to complete the challenge above using  two different preprocessors
 >> 
 >>~~~YAML
 >>
@@ -184,7 +184,7 @@ You could define different preprocessors with several preprocessor sections (set
 >>       scheme: linear  #how to interpolate for regridding
 >> 
 >>   prep_map_land:
->>     custom_order: true #ensure order follows a given
+>>     custom_order: true #ensure that given order of preprocessing is followed
 >>     mask_landsea:    #apply a mask
 >>       mask_out: sea   #mask out sea grid cells
 >>     regrid:    # now apply the preprocessor to regrid
@@ -205,7 +205,6 @@ You could define different preprocessors with several preprocessor sections (set
 >>         grid: gn #can change for variables from the same model
 >>         start_year: 1970 
 >>         end_year: 2000
->>         additional_datasets:
 >>     scripts: null
 >> 
 >>   diag_land_only_plot:
@@ -217,7 +216,6 @@ You could define different preprocessors with several preprocessor sections (set
 >>       grid: gn #can change for variables from the same model
 >>       start_year: 1970 
 >>       end_year: 2000
->>       additional_datasets:
 >>     scripts: null
 >> ~~~
 >> {: .source}
@@ -226,7 +224,7 @@ You could define different preprocessors with several preprocessor sections (set
 
 ## Adding different datasets for different variables
 
-Sometimes, we may want to include specific datasets for certain variables. An example is when we use observations for two different variables in a diagnostic. While the CMIP dataset details for the two variables may be common, the observations will likely not be so. It would be useful to know how to include different datasets for different variables. Here is an example of a simple preprocessor and diagnostic setup for that:
+Sometimes, we may want to include specific datasets only for certain variables. An example is when we use observations for two different variables in a diagnostic. While the CMIP dataset details for the two variables may be common, the observations will likely not be so. It would be useful to know how to include different datasets for different variables. Here is an example of a simple preprocessor and diagnostic setup for that:
 
 > ## Example
 >~~~YAML
@@ -254,7 +252,7 @@ Sometimes, we may want to include specific datasets for certain variables. An ex
 >         mip: Amon
 >         grid: gn #can change for variables from the same model
 >         start_year: 1970  
->         end_year: 2000 #  start and end years for a30 year period, 
+>         end_year: 2000 #  start and end years for a 30 year period, 
 >                        # we assume this is common and exists for all
 >                        # model and obs data 
 >         additional_datasets:
@@ -269,16 +267,81 @@ Sometimes, we may want to include specific datasets for certain variables. An ex
 >         end_year: 2000
 >         additional_datasets:
 >           - {dataset: HadCRUT4, project: OBS, type: ground, 
->              version: 1, tier: 2} #dataset specific to the temperature var
+>              version: 1, tier: 2} #dataset specific to the temperature variable
 > 
 >     scripts: null
 >~~~
 >{: .source}
 {: .solution}
 
+## Creating variable groups
+
+Variable grouping can be used to preprocess different clusters of data for the same variable. For instance, the example below illustrates how we can compute separate multimodel means for CMIP5 and CMIP6 data given the same variable. Additionally we can also preprocess observed data for evaluation.
+
+> ## Example
+>~~~YAML
+>
+>preprocessors:
+>  prep_mmm:
+>    custom_order: true
+>    regrid:
+>      target_grid: 2.5 x 2.5
+>      scheme: linear
+>    multi_model_statistics:
+>      span: full
+>      statistics: [mean, median]
+>
+>  prep_obs:
+>    mask_landsea:
+>      mask_out: sea
+>    regrid:
+>      target_grid: 2.5 x 2.5
+>      scheme: linear
+>
+> #note that there is no field called datasets anymore
+> #note how multiple ensembles are added by using (1:4)
+>cmip5_datasets: &cmip5_datasets 
+>  - {dataset: CanESM2, ensemble: "r(1:4)i1p1", project: CMIP5} 
+>  - {dataset: MPI-ESM-LR, ensemble: "r(1:2)i1p1", project: CMIP5}
+>
+>cmip6_datasets: &cmip6_datasets
+>  - {dataset: UKESM1-0-LL, ensemble: "r(1:4)i1p1f2", grid: gn, project: CMIP6}
+>  - {dataset: CanESM5, ensemble: "r(1:4)i1p2f1", grid: gn, project: CMIP6}
+>
+>diagnostics:
+>
+>  diag_variable_groups:
+>    description: Demonstrate the use of variable groups.
+>    variables:
+>      tas_cmip5: &variable_settings # need a key name for the grouping
+>        short_name: tas # specify variable to look for
+>        preprocessor: prep_mmm
+>        mip: Amon
+>        exp: historical
+>        start_year: 2000
+>        end_year: 2005
+>        tag: TAS_CMIP5 #tag is optional if you are using these settings just once
+>        additional_datasets: *cmip5_datasets
+>      tas_obs:
+>        <<: *variable_settings
+>        preprocessor: prep_obs
+>        tag: TAS_OBS
+>        additional_datasets:
+>          - {dataset: HadCRUT4, project: OBS, type: ground, version: 1, tier: 2}
+>      tas_cmip6:
+>        <<: *variable_settings
+>        tag: TAS_CMIP6 
+>        additional_datasets: *cmip6_datasets #nothing changes from cmip5 except the data set
+>    scripts: null
+>~~~
+>{: .source}
+{: .solution}
+
+You should be able to see the variables grouped under different subdirectories under your output preproc directory. The different groupings can be accessed in your diagnostic by selecting the key name of the field variable_group  such as tas_cmip5, tas_cmip6 or tas_obs.
+>
 > ## How to find what CMIP data is available?
 >
-> [CMIP5](https://pcmdi.llnl.gov/mips/cmip5/index.html) and [CMIP6](https://pcmdi.llnl.gov/CMIP6/Guide/dataUsers.html) data obey the [CF-conventions](http://cfconventions.org/). Available variables could be found under the [CMIP5 data request](https://pcmdi.llnl.gov/mips/cmip5/docs/standard_output.pdf?id=28) and the [CMIP6 Data Request](http://clipc-services.ceda.ac.uk/dreq/index.html).
+> [CMIP5](https://pcmdi.llnl.gov/mips/cmip5/index.html) and [CMIP6](https://pcmdi.llnl.gov/CMIP6/Guide/dataUsers.html) data obey the [CF-conventions](http://cfconventions.org/). Available variables can be found under the [CMIP5 data request](https://pcmdi.llnl.gov/mips/cmip5/docs/standard_output.pdf?id=28) and the [CMIP6 Data Request](http://clipc-services.ceda.ac.uk/dreq/index.html).
 >
 > CMIP data is widely available via the Earth System Grid Federation ([ESGF](https://esgf.llnl.gov/)) and is accessible to users either via download from the ESGF portal or through the ESGF data nodes hosted by large computing facilities (like [CEDA-Jasmin](https://esgf-index1.ceda.ac.uk/), [DKRZ](https://esgf-data.dkrz.de/), etc). The ESGF also hosts observations for Model Intercomparison Projects (obs4MIPs) and reanalyses data (ana4MIPs).
 >