CMIP data

[briochemc]

Any recommended workflow for processing/analyzing CMIP data in Julia?

I thought I'd ask here for tips, which will undoubtedly be useful to me, and maybe help others too :)

• cloud compute/download. I've been using Zarr.jl and AWS.jl (using code found in notebooks by @gaelforget, Ryan Abernathey, and others) with some success, but I thought I'd ask here if you'd recommend anything else? I usually go

   # Some config I don't really understand
   AWS.global_aws_config(AWSConfig(creds=nothing, region=""))
   # table with all the URLs to the data on google cloud
   CMIP6_stores = CSV.read(download("https://storage.googleapis.com/cmip6/cmip6-zarr-consolidated-stores.csv"), DataFrame)
   # Some filtering of the DataFrame to pinpoint whatever data I need...
   store = filter(..., CMIP6_stores)
   # Then use Zarr to open it lazily
   data = zopen(store) 
   # Do some (hopefully lazy) computations on the data
   output = f(data)

• dealing with inconsistencies between models (e.g., different coordinate names like nav_lon vs longitude). The Python xmip package by Julius Busecke is specifically designed to address a lot of these issues. Is there anyone planning to port its functionality to Julia somewhere in this org? Or is it easy enough to use PyCall? Or do you do this all manually?

• xarray-like package. Which of the many packages do/would you use to manipulate gridded data from CMIP? For example, if I want to make some tracer budgets, I can deal with the different model grids (e.g., staggered or not) by hand (doing some tedious checks, interpolations/extrapolations, and so on), but surely there is a better way that leverages existing code?

Maybe this is all already answered somewhere and I just missed it, in which case sorry for the noise!

[Alexander-Barth]

dealing with inconsistencies between models

If you are using NCDatasets (NetCDF_jll should support Zarr), one can also index a dataset via the NetCDF standard name (assuming that this attribute is defined). If there is only one longitude NetCDF variable, then ds[CF"longitude"] should return it.
For some model (like ROMS, a C-grid staggered model), there are different longitude arrays for different variables. To get the one associated (via shared dimension names) to e.g. "temp", one can use:

julia> ds["temp"][CF"longitude"]
lon_rho (150 × 94)
  Datatype:    Float64
  Dimensions:  xi_rho × eta_rho
  Attributes:
   long_name            = longitude of RHO-points
   units                = degree_east
   standard_name        = longitude
   field                = lon_rho, scalar

julia> ds["u"][CF"longitude"]
lon_u (149 × 94)
  Datatype:    Float64
  Dimensions:  xi_u × eta_u
  Attributes:
   long_name            = longitude of U-points
   units                = degree_east
   standard_name        = longitude
   field                = lon_u, scalar

Maybe this helps...

[briochemc]

@Alexander-Barth I'm not sure how to use NCDatasets here. Do you have a MWE of using NCDatasets to grab a variable from a CMIP dataset in the cloud?

[Alexander-Barth]

Unfortunately, I don't have a MWE, just the general information here:
https://docs.unidata.ucar.edu/nug/current/nczarr_head.html

Note however, NetCDF_jll does not jet have S3 supported activated.
Ref: JuliaPackaging/Yggdrasil#6192

[Balinus]

I don't have code nearby, but I had success in the past using Zarr and YAXArray :

https://juliadatacubes.github.io/YAXArrays.jl/dev/examples/generated/UserGuide/openZarr/