Skip to content

Commit f83b917

Browse files
author
Remi Kazeroni
committed
review suggestions added
1 parent 57a9832 commit f83b917

File tree

1 file changed

+44
-20
lines changed

1 file changed

+44
-20
lines changed

_episodes/09-cmorization.md

Lines changed: 44 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,9 @@ the data. This process is called "CMORization".
4040
> Concretely, the CMOR standards dictate e.g. the variable names and units,
4141
coordinate information, how the data should be structured (e.g. 1 variable per
4242
file), additional metadata requirements, but also file naming conventions a.k.a.
43-
the data reference syntax (DRS). All this information is stored in so-called
44-
CMOR tables. As an example, the CMOR tables for the CMIP6 project can be found
43+
the data reference syntax ([DRS](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html)).
44+
> All this information is stored in so-called CMOR tables.
45+
> As an example, the CMOR tables for the CMIP6 project can be found
4546
[here](https://github.com/PCMDI/cmip6-cmor-tables).
4647
{: .callout}
4748

@@ -58,6 +59,7 @@ ESMValTool offers two ways to CMORize data:
5859
In this lesson, we will re-implement a CMORizer script for the FLUXCOM dataset
5960
that contains observations of the Gross Primary Production (GPP), a variable
6061
that is important for calculating components of the global carbon cycle.
62+
See the next section on how to obtain data.
6163

6264
We will assume that you are using a development installation of ESMValTool as
6365
explained in the [Development and Contribution episode](/07-development-setup).
@@ -73,7 +75,7 @@ button on the "FLUXCOM (RS+METEO) Global Land Carbon Fluxes using CRUNCEP
7375
climate data". You'll receive an email with the FTP address to access the
7476
server. Connect to the server, follow the path in your email, and look for the
7577
file `raw/monthly/GPP.ANN.CRUNCEPv6.monthly.2000.nc`. Download that file and
76-
save it in a folder called `/RAWOBS/Tier3/FLUXCOM`.
78+
save it in a folder called `~/data/RAWOBS/Tier3/FLUXCOM`.
7779

7880
Note: you'll need a user-friendly ftp client. On Linux, `ncftp` works okay.
7981

@@ -116,10 +118,20 @@ run the CMORizer scripts:
116118
cmorize_obs -c <config-user.yml> -o <dataset-name>
117119
```
118120

119-
The ``config-user-yml`` is the file in which we define the different data
120-
paths, e.g. where the ESMValTool would find the "RAWOBS" folder. The
121-
``dataset-name`` needs to be identical to the folder name that was created
122-
to store the raw observation data files, in our case this would be "FLUXCOM".
121+
The ``config-user.yml`` is the file in which we define the different data
122+
paths, see the episode on [Configuration](03-configuration).
123+
In the ``rootpath`` of your ``config-user.yml`, make sure to add the right
124+
directory for "RAWOBS" data in which you downloaded the FLUXCOM dataset:
125+
126+
```yaml
127+
rootpath:
128+
RAWOBS: ~/data/RAWOBS
129+
```
130+
131+
This enables ESMValTool to find the raw observational datasets stored in the
132+
"RAWOBS" folder. The ``dataset-name`` needs to be identical to the folder
133+
name that was created to store the raw observation data files, i.e.
134+
``RAWOBS/TierX/dataset-name``. In our case this would be "FLUXCOM".
123135
124136
If everything is okay, the output should look something like this:
125137
@@ -157,15 +169,21 @@ If everything is okay, the output should look something like this:
157169
{: .output}
158170
159171
So you can see that several fixes are applied, and the CMORized file is written
160-
to the ESMValTool output directory. In order to use it, we'll have to copy it
161-
from the output directory to a folder called
162-
`<path_to_your_data>/OBS/Tier3/FLUXCOM` and make sure the path to ``OBS`` is set
163-
correctly in our config-user file.
172+
to the ESMValTool output directory, i.e.
173+
`~/esmvaltool_output/cmorize_obs_YYYYMMDD_HHMMSS/TierX/dataset-name/filename.nc`
174+
In order to use it, we'll have to copy it from the output directory to a folder
175+
called `~/data/OBS/Tier3/FLUXCOM` and make sure the path to ``OBS`` is set
176+
correctly in our config-user file:
177+
178+
```yaml
179+
rootpath:
180+
OBS: ~/data/OBS
181+
```
164182

165183
You can also see the path where ESMValTool stores the reformatting script:
166-
`<path to esmvaltool>/esmvaltool/cmorizers/obs/cmorize_obs_fluxcom.py`. You may
184+
`~/ESMValTool/esmvaltool/cmorizers/obs/cmorize_obs_fluxcom.py`. You may
167185
have a look at this file if you want. The script also uses a configuration file:
168-
`<path to esmvaltool>/esmvaltool/cmorizers/obs/cmor_config/FLUXCOM.yml`.
186+
`~/ESMValTool/esmvaltool/cmorizers/obs/cmor_config/FLUXCOM.yml`.
169187

170188
## Make a test recipe
171189

@@ -247,9 +265,9 @@ going on. We'll also remove the CMORized data that we've just created, so our
247265
test recipe will not be able to use it anymore.
248266

249267
```bash
250-
rm <path_to_your_data>/OBS/Tier3/FLUXCOM/OBS_FLUXCOM_reanaly_ANN-v1_Lmon_gpp_200001-200012.nc
251-
rm <path_to_esmvaltool>/esmvaltool/cmorizers/obs/cmorize_obs_fluxcom.nc
252-
rm <path to esmvaltool>/esmvaltool/cmorizers/obs/cmor_config/FLUXCOM.yml
268+
rm ~/data/OBS/Tier3/FLUXCOM/OBS_FLUXCOM_reanaly_ANN-v1_Lmon_gpp_200001-200012.nc
269+
rm ~/ESMValTool/esmvaltool/cmorizers/obs/cmorize_obs_fluxcom.py
270+
rm ~/ESMValTool/esmvaltool/cmorizers/obs/cmor_config/FLUXCOM.yml
253271
```
254272

255273
If you now run the test recipe again it should fail, and somewhere in the output
@@ -301,7 +319,7 @@ shortly. When you type the command ``cmorize_obs`` in the terminal, ESMValTool
301319
will call this function with the settings found in your configuration files.
302320

303321
The ESMValTool CMORizer also needs a dataset configuration file. Create a file
304-
called `<path_to_esmvaltool>/esmvaltool/cmorizers/obs/cmor_config/FLUXCOM.yml`
322+
called `~/ESMValTool/esmvaltool/cmorizers/obs/cmor_config/FLUXCOM.yml`
305323
and fill it with the following boilerplate:
306324

307325
```yaml
@@ -323,7 +341,7 @@ attributes:
323341
# mip: ???
324342
```
325343

326-
**Note**: the name of this file *must* be identical to dataset name.
344+
**Note**: the name of this file *must* be identical to ``dataset-name``.
327345

328346
As you can see, the configuration file contains information about the original
329347
filename of the dataset, and some additional metadata that you might recognize
@@ -556,9 +574,15 @@ address in the next section.
556574

557575
### 3. Implementing additional fixes
558576

559-
Copy the output of the CMORizer to your folder `<path to your data>/OBS6/Tier3/`
577+
Copy the output of the CMORizer to your folder `~/data/OBS6/Tier3/`
560578
and change the test recipe to look for OBS6 data instead of OBS (note: we're
561-
upgrading the CMORizer to newer standards here!).
579+
upgrading the CMORizer to newer standards here!). Make sure the path to ``OBS6``
580+
is set correctly in our config-user file:
581+
582+
```yaml
583+
rootpath:
584+
OBS6: ~/data/OBS6
585+
```
562586

563587
If we now run the test recipe on our newly 'CMORized' data,
564588

0 commit comments

Comments
 (0)