Skip to content

Commit aaade54

Browse files
flicj191flicj191
committed
clean up
1 parent e002f43 commit aaade54

File tree

2 files changed

+152
-143
lines changed

2 files changed

+152
-143
lines changed

‎_episodes/11-esmvalcoreapi.md‎

Lines changed: 149 additions & 143 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,21 @@ keypoints:
1717
- "Use `datasets_to_recipe` helper to start making recipes"
1818
---
1919

20-
In this episode we will introduce the ESMValCore API in a jupyter notebook. This is reformatted from material from
21-
this [blog post](https://blog.esciencecenter.nl/easy-ipcc-powered-by-esmvalcore-19a0b6366ea7){:target="_blank"}
22-
by Peter Kalverla. There's also material from the [example notebooks][docs-notebooks]{:target="_blank"} and the
20+
In this episode we will introduce the ESMValCore API in a jupyter notebook. This is reformatted
21+
from material from this [blog post][easy-ipcc-blog]{:target="_blank"}
22+
by Peter Kalverla. There's also material from the
23+
[example notebooks][docs-notebooks]{:target="_blank"} and the
2324
[API reference documentation][api-reference]{:target="_blank"}.
2425

2526
## Start JupyterLab
26-
A [jupyter notebook](https://jupyter.org/){:target="_blank"} is an interactive document where you can run code.
27+
A [jupyter notebook](https://jupyter.org/){:target="_blank"} is an interactive document where
28+
you can run code.
2729
You will need to use a python environment with ESMValTool and ESMValCore installed.
2830

2931
## Find Datasets with facets
30-
We have seen from running available recipes that ESMValTool is able to find data from facets that were given in
31-
the recipe. We can use this in a Notebook, including filling out the facets for data definition.
32+
We have seen from running available recipes that ESMValTool is able to find data from facets that
33+
were given in the recipe. We can use this in a Notebook, including filling out the facets for
34+
data definition.
3235
To do this we will use the `Dataset` object from the API. Let's look at this example.
3336

3437
```python
@@ -47,7 +50,8 @@ dataset.augment_facets()
4750
print(dataset)
4851
```
4952
> ## Pro tip: Augmented facets in the output
50-
> When running a recipe there is a `_filled` recipe `yml` file in the output `/run` folder which augments the facets.
53+
> When running a recipe there is a `_filled` recipe `yml` file in the output `/run` folder which
54+
> augments the facets.
5155
> > ## Example recipe output folder
5256
> > ```output
5357
> > esmvaltool_output/flato13ipcc_figure914_20240729_043707/run
@@ -63,7 +67,8 @@ print(dataset)
6367
{: .callout}
6468
6569
> ## Search available
66-
> Search from files available locally with wildcard functionality `'*'` to get all the available datasets.
70+
> Search from files available locally with wildcard functionality `'*'` to get all the available
71+
> datasets.
6772
> - How can you search for all available ensembles?
6873
>
6974
> > ## Solution
@@ -84,8 +89,8 @@ print(dataset)
8489
> > ```
8590
> {: .solution}
8691
> There is also the ability to search on ESGF nodes and download. See
87-
> [reference](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.esgf.html){:target="_blank"}
88-
> for more details. Check the configuration settings for this.
92+
> [reference][api-esgf]{:target="_blank"} for more details.
93+
> Check the configuration settings for this.
8994
>```python
9095
>from esmvalcore.config import CFG
9196
>CFG['search_esgf'] = 'always'
@@ -253,7 +258,7 @@ print(da)
253258
The output from the preprocessor functions are Iris cubes.
254259
[Iris](https://scitools-iris.readthedocs.io/en/latest/index.html){:target="_blank"}
255260
has wrappers for [matplotlib](https://matplotlib.org/){:target="_blank"} to [plot the processed
256-
cubes](https://scitools-iris.readthedocs.io/en/latest/userguide/plotting_a_cube.html#iris-cube-plotting){:target="_blank"}.
261+
cubes][iris-plot]{:target="_blank"}.
257262
This is useful in a notebook to help develop your recipe with the esmvalcore preprocessors.
258263
```python
259264
from iris import quickplot
@@ -270,70 +275,71 @@ quickplot.plot(cube)
270275
> - Apply the preprocessors to each dataset and plot the result
271276
>
272277
> > ## Solution
273-
> > ```python
274-
> > import cf_units
275-
> > import matplotlib.pyplot as plt
276-
> > from iris import quickplot
277-
> >
278-
> > from esmvalcore.config import CFG
279-
> > from esmvalcore.dataset import Dataset
280-
> > from esmvalcore.preprocessor import annual_statistics, anomalies, area_statistics
281-
> >
282-
> >
283-
> > # Settings for automatic ESGF search
284-
> > CFG['search_esgf'] = 'when_missing'
285-
> >
286-
> > # Declare common dataset facets
287-
> > template = Dataset(
288-
> > short_name='tos',
289-
> > mip='Omon',
290-
> > project='CMIP6',
291-
> > exp= '*', # We'll fill this below
292-
> > dataset='*', # We'll fill this below
293-
> > ensemble='r4i1p1f1',
294-
> > grid='gn',
295-
> > )
296-
> >
297-
> > # Substitute data sources and experiments
298-
> > datasets = []
299-
> > for dataset_id in ["CESM2", "MPI-ESM1-2-LR", "ACCESS-ESM1-5"]:
300-
> > for experiment_id in ['ssp126', 'ssp585']:
301-
> > dataset = template.copy(dataset=dataset_id, exp=['historical', experiment_id])
302-
> > dataset.add_supplementary(short_name='areacello', mip='Ofx', exp='historical')
303-
> > dataset.augment_facets()
304-
> > datasets.append(dataset)
305-
> >
306-
> > # Set the reference period for anomalies
307-
> > reference_period = {
308-
> > "start_year": 1950, "start_month": 1, "start_day": 1,
309-
> > "end_year": 1979, "end_month": 12, "end_day": 31,
310-
> > }
311-
> >
312-
> > # (Down)load, pre-process, and plot the cubes
313-
> > for dataset in datasets:
314-
> > cube = dataset.load()
315-
> > cube = area_statistics(cube, operator='mean')
316-
> > cube = anomalies(cube, reference=reference_period, period='month') # notice 'month'
317-
> > cube = annual_statistics(cube, operator='mean')
318-
> > cube.convert_units('degrees_C')
319-
> >
320-
> > # Make sure all datasets use the same calendar for plotting
321-
> > tcoord = cube.coord('time')
322-
> > tcoord.units = cf_units.Unit(tcoord.units.origin, calendar='gregorian')
323-
> >
324-
> > # Plot
325-
> > quickplot.plot(cube, label=f"{dataset['dataset']} - {dataset['exp']}")
326-
> >
327-
> > # Show the plot
328-
> > plt.legend()
329-
> > plt.show()
330-
> > ```
278+
> >```python
279+
> >import cf_units
280+
> >import matplotlib.pyplot as plt
281+
> >from iris import quickplot
282+
> >
283+
> >from esmvalcore.config import CFG
284+
> >from esmvalcore.dataset import Dataset
285+
> >from esmvalcore.preprocessor import annual_statistics, anomalies, area_statistics
286+
> >
287+
> >
288+
> ># Settings for automatic ESGF search
289+
> >CFG['search_esgf'] = 'when_missing'
290+
> >
291+
> ># Declare common dataset facets
292+
> >template = Dataset(
293+
> > short_name='tos',
294+
> > mip='Omon',
295+
> > project='CMIP6',
296+
> > exp= '*', # We'll fill this below
297+
> > dataset='*', # We'll fill this below
298+
> > ensemble='r4i1p1f1',
299+
> > grid='gn',
300+
> >)
301+
> >
302+
> ># Substitute data sources and experiments
303+
> >datasets = []
304+
> >for dataset_id in ["CESM2", "MPI-ESM1-2-LR", "ACCESS-ESM1-5"]:
305+
> > for experiment_id in ['ssp126', 'ssp585']:
306+
> > dataset = template.copy(dataset=dataset_id, exp=['historical', experiment_id])
307+
> > dataset.add_supplementary(short_name='areacello', mip='Ofx', exp='historical')
308+
> > dataset.augment_facets()
309+
> > datasets.append(dataset)
310+
> >
311+
> ># Set the reference period for anomalies
312+
> >reference_period = {
313+
> > "start_year": 1950, "start_month": 1, "start_day": 1,
314+
> > "end_year": 1979, "end_month": 12, "end_day": 31,
315+
> >}
316+
> >
317+
> ># (Down)load, pre-process, and plot the cubes
318+
> >for dataset in datasets:
319+
> > cube = dataset.load()
320+
> > cube = area_statistics(cube, operator='mean')
321+
> > cube = anomalies(cube, reference=reference_period, period='month') # notice 'month'
322+
> > cube = annual_statistics(cube, operator='mean')
323+
> > cube.convert_units('degrees_C')
324+
> >
325+
> > # Make sure all datasets use the same calendar for plotting
326+
> > tcoord = cube.coord('time')
327+
> > tcoord.units = cf_units.Unit(tcoord.units.origin, calendar='gregorian')
328+
> >
329+
> > # Plot
330+
> > quickplot.plot(cube, label=f"{dataset['dataset']} - {dataset['exp']}")
331+
> >
332+
> ># Show the plot
333+
> >plt.legend()
334+
> >plt.show()
335+
> >```
331336
> {: .solution}
332337
{: .challenge}
333338
334339
> ## Pro tip: Convert to recipe
335-
> We can use the helper to start making the recipe. A recipe can be used for reproducibility of an
336-
> analysis. This list the datasets in a recipe format and we would then have to create the preprocessors
340+
> We can use the helper to start making the recipe. A recipe can be used for
341+
> reproducibility of an analysis. This list the datasets in a
342+
> recipe format and we would then have to create the preprocessors
337343
> and diagnostic script.
338344
> ```python
339345
> from esmvalcore.dataset import datasets_to_recipe
@@ -418,87 +424,87 @@ quickplot.plot(cube)
418424
>
419425
> > ## 1. Define datasets:
420426
> >
421-
> > ```python
422-
> > from esmvalcore.dataset import Dataset
423-
> > obs = Dataset(
424-
> > short_name='siconc', mip='SImon', project='OBS6', type='reanaly',
425-
> > dataset='NSIDC-G02202-sh', tier='3', version='4', timerange='1979/2018',
426-
> > )
427-
> > # Add areacello as supplementary dataset
428-
> > obs.add_supplementary(short_name='areacello', mip='Ofx')
429-
> >
430-
> > model = Dataset(
431-
> > short_name='siconc', mip='SImon', project='CMIP6', activity='CMIP',
432-
> > dataset='ACCESS-ESM1-5', ensemble='r1i1p1f1', grid='gn', exp='historical',
433-
> > timerange='1960/2010', institute = '*',
434-
> > )
435-
> >
436-
> > om_facets={'dataset' :'ACCESS-OM2', 'exp':'omip2', 'activity':'OMIP', 'timerange':'0306/0366' }
437-
> >
438-
> > model.add_supplementary(short_name='areacello', mip='Ofx')
439-
> >
440-
> > model_om = model.copy(**om_facets)
441-
> > ```
427+
> >```python
428+
> >from esmvalcore.dataset import Dataset
429+
> >obs = Dataset(
430+
> > short_name='siconc', mip='SImon', project='OBS6', type='reanaly',
431+
> > dataset='NSIDC-G02202-sh', tier='3', version='4', timerange='1979/2018',
432+
> >)
433+
> ># Add areacello as supplementary dataset
434+
> >obs.add_supplementary(short_name='areacello', mip='Ofx')
435+
> >
436+
> >model = Dataset(
437+
> > short_name='siconc', mip='SImon', project='CMIP6', activity='CMIP',
438+
> > dataset='ACCESS-ESM1-5', ensemble='r1i1p1f1', grid='gn', exp='historical',
439+
> > timerange='1960/2010', institute = '*',
440+
> >)
441+
> >
442+
> >om_facets={'dataset' :'ACCESS-OM2', 'exp':'omip2', 'activity':'OMIP', 'timerange':'0306/0366' }
443+
> >
444+
> >model.add_supplementary(short_name='areacello', mip='Ofx')
445+
> >
446+
> >model_om = model.copy(**om_facets)
447+
> >```
442448
> {: .solution}
443449
> > ## Tip: Check dataset files can be found
444450
> > The observational dataset used is a Tier 3, so with some licensing restrictions.
445451
> > Check files can be found for all the datasets:
446452
> >
447-
> > ```python
448-
> > for ds in [model, model_om, obs]:
449-
> > print(ds['dataset'],' : ' ,ds.files)
450-
> > print(ds.supplementaries[0].files)
451-
> > ```
452-
> > You can try to find and add another observational dataset. eg:
453-
> > ```python
454-
> > obs_other = Dataset(
455-
> > short_name='siconc', mip='*', project='OBS', type='*',
456-
> > dataset='*', tier='*', timerange='1979/2018'
457-
> > )
458-
> > obs_other.files
459-
> > ```
453+
> >```python
454+
> >for ds in [model, model_om, obs]:
455+
> > print(ds['dataset'],' : ' ,ds.files)
456+
> > print(ds.supplementaries[0].files)
457+
> >```
458+
> >You can try to find and add another observational dataset. eg:
459+
> >```python
460+
> >obs_other = Dataset(
461+
> > short_name='siconc', mip='*', project='OBS', type='*',
462+
> > dataset='*', tier='*', timerange='1979/2018'
463+
> >)
464+
> >obs_other.files
465+
> >```
460466
> {: .solution}
461467
> > ## 2. Use esmvalcore API preprocessors on the datasets and plot results
462468
> >
463-
> > ```python
464-
> > import iris
465-
> > import matplotlib.pyplot as plt
466-
> > from iris import quickplot
467-
> > from esmvalcore.preprocessor import (
468-
> > mask_outside_range,
469-
> > extract_region,
470-
> > area_statistics,
471-
> > annual_statistics
472-
> > )
473-
> > # model_om - at index 1 to offset years
469+
> >```python
470+
> >import iris
471+
> >import matplotlib.pyplot as plt
472+
> >from iris import quickplot
473+
> >from esmvalcore.preprocessor import (
474+
> > mask_outside_range,
475+
> > extract_region,
476+
> > area_statistics,
477+
> > annual_statistics
478+
> >)
479+
> ># model_om - at index 1 to offset years
480+
>
481+
> >load_data = [model, model_om, obs]
474482
> >
475-
> > load_data = [model, model_om, obs]
476-
> >
477-
> > # function to use for both min and max ['max','min']
478-
> >
479-
> > def trends_seaicearea(min_max):
480-
> > plt.clf()
481-
> > for i,data in enumerate(load_data):
482-
> > cube = data.load()
483-
> > cube = mask_outside_range(cube, 15, 100)
484-
> > cube = extract_region(cube,0,360,-90,0)
485-
> > cube = area_statistics(cube, 'sum')
486-
> > cube = annual_statistics(cube, min_max)
487-
> >
488-
> > iris.util.promote_aux_coord_to_dim_coord(cube, 'year')
489-
> > cube.convert_units('km2')
490-
> > if i == 1: ## om years 306/366 apply offset
491-
> > cube.coord('year').points = [y + 1652 for y in cube.coord('year').points]
492-
> > label_name = data['dataset']
493-
> > print(label_name, cube.shape)
494-
> > quickplot.plot(cube, label=label_name)
495-
> >
496-
> > plt.title(f'Trends in Sea-Ice {min_max.title()}ima')
497-
> > plt.ylabel('Sea-Ice Area (km2)')
498-
> > plt.legend()
499-
> >
500-
> > trends_seaicearea('min')
501-
> > ```
483+
> ># function to use for both min and max ['max','min']
484+
> >
485+
> >def trends_seaicearea(min_max):
486+
> > plt.clf()
487+
> > for i,data in enumerate(load_data):
488+
> > cube = data.load()
489+
> > cube = mask_outside_range(cube, 15, 100)
490+
> > cube = extract_region(cube,0,360,-90,0)
491+
> > cube = area_statistics(cube, 'sum')
492+
> > cube = annual_statistics(cube, min_max)
493+
> >
494+
> > iris.util.promote_aux_coord_to_dim_coord(cube, 'year')
495+
> > cube.convert_units('km2')
496+
> > if i == 1: ## om years 306/366 apply offset
497+
> > cube.coord('year').points = [y + 1652 for y in cube.coord('year').points]
498+
> > label_name = data['dataset']
499+
> > print(label_name, cube.shape)
500+
> > quickplot.plot(cube, label=label_name)
501+
> >
502+
> > plt.title(f'Trends in Sea-Ice {min_max.title()}ima')
503+
> > plt.ylabel('Sea-Ice Area (km2)')
504+
> > plt.legend()
505+
> >
506+
> >trends_seaicearea('min')
507+
> >```
502508
> {: .solution}
503509
{: .challenge}
504510

‎_includes/links.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,10 @@
8484
[workshop-repo]: {{ site.workshop_repo }}
8585
[yaml]: http://yaml.org/
8686
[api-config]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.config.html
87+
[easy-ipcc-blog]: https://blog.esciencecenter.nl/easy-ipcc-powered-by-esmvalcore-19a0b6366ea7
8788
[experimental-output]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.experimental.recipe_output.html
8889
[docs-notebooks]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/example-notebooks.html
8990
[api-reference]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.html#api
91+
[api-esgf]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.esgf.html
9092
[api-preprocessors]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.preprocessor.html
93+
[iris-plot]: https://scitools-iris.readthedocs.io/en/latest/userguide/plotting_a_cube.html#iris-cube-plotting

0 commit comments

Comments
 (0)