Skip to content

Commit 1169a07

Browse files
hb326hb326
authored
Update 09-cmorization.md
Finishing touches on Section 5, and combining Section 5 with Section 6.
1 parent c4a26f2 commit 1169a07

File tree

1 file changed

+148
-20
lines changed

1 file changed

+148
-20
lines changed

‎_episodes/09-cmorization.md‎

Lines changed: 148 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@ know we have completed our task.
186186
Try to run the example recipe with
187187
188188
```bash
189-
esmvaltool run recipe_check_fluxnet.yml --log_level debug
189+
esmvaltool run recipe_check_fluxcom.yml --log_level debug
190190
```
191191
192192
The `log_level` flag ensures that all relevant information is included in the
@@ -850,43 +850,171 @@ iris.exceptions.UnitConversionError: Cannot convert from unknown units. The
850850
"units" attribute may be set directly.
851851
```
852852
853-
Ok, so let's fix the units of the "GPP" variable in the CMORizer.
853+
Ok, so let's fix the units of the "GPP" variable in the CMORizer. For that
854+
we add the following three lines to the code in the section
855+
``_extract_variable``:
854856
857+
```python
858+
# convert data from gc/m2/day to kg/m2/s
859+
cube = cube / (1000 * 86400)
860+
cube.units = 'kg m-2 s-1'
861+
```
855862
856-
.. utilities.py: https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/cmorizers/obs/utilities.py
863+
The whole section should not look then look like this:
857864
865+
```python
866+
def _extract_variable(cmor_info, attrs, filepath, out_dir):
867+
"""Extract variable."""
868+
var = cmor_info.short_name
869+
logger.info("Var is %s", var)
870+
cubes = iris.load(filepath)
871+
for cube in cubes:
872+
# convert data from gc/m2/day to kg/m2/s
873+
cube = cube / (1000 * 86400)
874+
cube.units = 'kg m-2 s-1'
858875
859-
## 6. Run the CMORizer script
876+
logger.info("Saving file")
877+
utils.save_variable(cube,
878+
var,
879+
out_dir,
880+
attrs,
881+
unlimited_dimensions=['time'])
860882
861-
The cmorizing script for the given dataset can be run with:
883+
```
884+
885+
If we run the CMORizer script now again with the ``cmorize_obs`` call we
886+
get a NetCDF file that has fixed units, but has an "unknown" variable name.
862887
863888
```bash
864-
cmorize_obs -c <config-user.yml> -o <dataset-name>
889+
...
890+
variables:
891+
float unknown(time, lat, lon) ;
892+
unknown:_FillValue = 1.e+20f ;
893+
unknown:units = "kg m-2 s-1" ;
894+
double time(time) ;
895+
time:axis = "T" ;
896+
time:units = "days since 1582-10-15 00:00:00" ;
897+
time:standard_name = "time" ;
898+
time:calendar = "gregorian" ;
899+
double lat(lat) ;
900+
double lon(lon) ;
901+
...
865902
```
866903
867-
> ## Note
904+
We will have to do some more code tweaking then. But we also have not fixed
905+
the problem with the coordinates ``lat`` and ``lon`` yet. There is no "units"
906+
or "standard_name" given for either of these coordinates which will cause a
907+
problem for the ESMValTool. Such some smaller formatting problems can occur
908+
relatively often for coordinates like ``lat``, ``lon`` or ``time``. This means
909+
that these problems need fixing in many CMORizers. Therefore there are common
910+
functions available within the ESMValTool that one can import and use in the
911+
new CMORizer script. The functions, written in python, are stored in the folder
912+
[utilities.py](https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/cmorizers/obs/utilities.py)
913+
914+
> ## Finalizing the "FLUXCOM" CMORizer
868915
>
869-
> The output path given in the configuration file is the path where
870-
> your cmorized dataset will be stored. The ESMValTool will create a folder
871-
> with the correct tier information (see Section `2. Edit your configuration file`) if that tier folder is not
872-
> already available, and then a folder named after the data set. In this
873-
> folder the cmorized data set will be stored as a netCDF file.
874-
{: .callout}
875-
876-
If your run was successful, one or more NetCDF files are produced in your
877-
output directory.
878-
879-
> ## Check that your cmoriziation was successful
916+
> The task is now to work with the functions in the file "utilities.py" to
917+
> complete the CMORizer for the "FLUXCOM" dataset so that it can be read by
918+
> the ESMValTool. The definitions of the coordinates and the variable "gpp"
919+
> should look like the following after the successful CMORization:
920+
>
921+
> ```bash
922+
> variables:
923+
> float gpp(time, lat, lon) ;
924+
> gpp:_FillValue = 1.e+20f ;
925+
> gpp:standard_name = "gross_primary_productivity_of_carbon" ;
926+
> gpp:long_name = "Carbon Mass Flux out of Atmosphere due to Gross Primary Production on Land" ;
927+
> gpp:units = "kg m-2 s-1" ;
928+
> double time(time) ;
929+
> time:axis = "T" ;
930+
> time:bounds = "time_bnds" ;
931+
> time:units = "days since 1950-1-1 00:00:00" ;
932+
> time:standard_name = "time" ;
933+
> time:calendar = "gregorian" ;
934+
> double time_bnds(time, bnds) ;
935+
> double lat(lat) ;
936+
> lat:axis = "Y" ;
937+
> lat:bounds = "lat_bnds" ;
938+
> lat:units = "degrees_north" ;
939+
> lat:standard_name = "latitude" ;
940+
> lat:long_name = "latitude coordinate" ;
941+
> double lat_bnds(lat, bnds) ;
942+
> double lon(lon) ;
943+
> lon:axis = "X" ;
944+
> lon:bounds = "lon_bnds" ;
945+
> lon:units = "degrees_east" ;
946+
> lon:standard_name = "longitude" ;
947+
> lon:long_name = "longitude coordinate" ;
948+
> double lon_bnds(lon, bnds) ;
949+
> ```
880950
>
881-
> Check your output directory for the freshly produced NetCDF file.
951+
> For that to happen, you will have to fix/work on the following things:
952+
> - adding standard names to the dimensions ``lat`` and ``lon``
953+
> - fix the metadata for the variable "gpp"
954+
> - change the time units to start in the year 1950
955+
> - make the coordinates CMOR-compliant
956+
> - set global attributes for the data file
882957
>
883958
> > ## Answers
884959
> >
885-
> > Write the answer here.
960+
> > To fully CMORize the "FLUXCOM" dataset, you need to add the following
961+
> > lines of code to the ``_extract_variable`` function of your CMORizer
962+
> > script:
963+
> >
964+
> > ```python
965+
> > def _extract_variable(cmor_info, attrs, filepath, out_dir):
966+
> > """Extract variable."""
967+
> > var = cmor_info.short_name
968+
> > logger.info("Var is %s", var)
969+
> > cubes = iris.load(filepath)
970+
> > for cube in cubes:
971+
> > # convert data from gc/m2/day to kg/m2/s
972+
> > cube = cube / (1000 * 86400)
973+
> > cube.units = 'kg m-2 s-1'
974+
> >
975+
> > # The following two lines are needed for iris.util.guess_coord_axis
976+
> > cube.coord('lat').standard_name = 'latitude'
977+
> > cube.coord('lon').standard_name = 'longitude'
978+
> > utils.fix_var_metadata(cube, cmor_info)
979+
> > utils.convert_timeunits(cube, 1950)
980+
> > utils.fix_coords(cube)
981+
> > utils.set_global_atts(cube, attrs)
982+
> > utils.flip_dim_coord(cube, 'latitude')
983+
> > coord = cube.coord('latitude')
984+
> > coord.bounds = np.flip(coord.bounds, axis=1)
985+
> > logger.info("Saving file")
986+
> > utils.save_variable(cube,
987+
> > var,
988+
> > out_dir,
989+
> > attrs,
990+
> > unlimited_dimensions=['time'])
991+
> >
992+
> > ```
886993
> >
887994
> {: .solution}
888995
{: .challenge}
889996
997+
So now we can finally run our new CMORizing script to produce an ESMValTool
998+
readable datafile of the "FLUXCOM" dataset. As already mentioned, the call
999+
to run a CMORizing script is as follows:
1000+
1001+
```bash
1002+
cmorize_obs -c <config-user.yml> -o <dataset-name>
1003+
```
1004+
1005+
If your run was successful, you should have gotten no error message in the
1006+
ESMValTool logging information and one or more NetCDF files should have been
1007+
produced in your output directory.
1008+
1009+
To make sure that the CMORization has really worked as planned, we run the
1010+
CMOR checker on this newly produced NetCDF file. If the ESMValTool can read
1011+
the data without problems, you should see as one of the last lines of
1012+
ESMValTool output:
1013+
1014+
```bash
1015+
INFO Run was successful
1016+
```
1017+
8901018
8911019
8921020
## Conclusion

0 commit comments

Comments
 (0)