Which NWP variables and vertical levels to select for ICON-EU?

[JackKelly]

We plan to put a reduced set of the variables into the ICON-EU Zarr on Dynamical.

And we currently plan to not have a separate dimension for vertical levels. Instead, we'll use a suffix in the variable name to denote different vertical levels (e.g. u_10m for the 10 metre U wind component). And we don't plan to store many different vertical levels. (Update: But see the discussion below about the importance of vertical levels for the AI NWP community!)

Note that OCF's ICON-EU archive from 2020 to 2023 includes only 27 of the 63 ICON-EU variables. (See here for a list of the 27 variables included in the 2020 to 2023 archive; and see here for a list of all 63 variables). We'd like the ICON-EU dataset on Dynamical to be as easy to use as possible, and to have as little missing data as possible. So, ideally, the ICON-EU variables exposed on Dynamical would be selected from the 27 variables included in OCF's ICON-EU archive from 2020 to 2023.

[jacobbieker]

I do know that the ICON-EU/ICON-Global archives are/have been used for ML weather forecasting training datasets. For those, including the pressure level variables would be necessary to have. Pretty much all the pressure level variables are used by the main ML weather models, so including geopotential, U/V wind, vertical velocity, relative/specific humidity and temperature would be really helpful.

Other variables used, but not as uniformly, would include the total column water vapor, and surface pressure.

[JackKelly]

Very good point, thank you Jacob! At the moment, I think we're working on the assumption that we might not have the capacity to expose all the pressure levels in the Zarr.

But I'd be keen to support the AI NWP community.

Jacob, would it be sufficient if we only put single-level data into the Zarr, but also archived DWD's original GRIB2 files on cloud object storage (so the AI NWP community could use those GRIBs?)

What do you think, @aldenks?

[jacobbieker]

Yeah, if its not feasible to have them included, archiving the gribs would work, just having a continuing archive somewhere would be super useful.

[JackKelly]

Two relevant discussions in the ICON-EU discussion on Hugging Face:

• A discussion where a user asks for P (pressure) and QV (water vapor specific humidity) at different pressure levels. The user says: "P and QV are used in humidity calculations and it also lets us run our own models. Without having P & QV at different pressure levels, we would have a hard time running our own models."
• A discussion where two users ask for model levels (instead of or as well as pressure levels) because the model levels are at constant altitudes, and so are useful for wind power forecasting.

[aldenks]

Thanks @jacobbieker @JackKelly! I like the idea of archiving the complete gribs (e.g. on Source Coop) and then starting with a more limited surface variable/level set in the Zarr and seeing what people ask to add in. For AI WP model training I'd imagine almost everyone wants the whole domain at each timestep, meaning the native "chunking" of the grib files is actually optimal and we could expose it as a virtualzarr/virtual icechunk dataset and get equivalent read performance.

That said @jacobbieker if you know orgs that would want to financially support the storage costs in Zarr for the whole thing, have them email me (alden@) or feedback@dynamical.org. For the dynamical.org Zarr i want to make sure we are resourced to support whatever we publish in a long-term, ongoing way.

[jacobbieker]

Yeah, for the weather models, its always the full domain, so a virtualzarr dataset would probably work just as well!

[rusmarko]

Hi,

I use these variables:

'clch',
'clcl',
'clcm',
'clct',
'alb_rad',
'relhum_2m',
't_2m',
'u_10m',
('u', 50),
('u', 500),
('u', 850),
('u', 950),
'v_10m',
('v', 50),
('v', 500),
('v', 850),
('v', 950),

Where numbers are pressure levels.

[JackKelly]

That's really useful, thank you, @rusmarko! Please may I ask two quick follow-up questions:

• Do you have any preference between pressure levels versus model levels?
• I haven't been able to find any documentation about alb_rad (surface albedo) in ICON-EU. Is alb_rad just the average surface albedo from climatology? Is it the same across NWP init times and steps?

TBH, I'm afraid that we're currently thinking that we might not include vertical levels as a dimension in the Zarr on dynamical. So the Zarr probably won't include wind at different pressure levels, I'm sorry.

The good news is that - as well as creating the Zarr - we're hoping to archive DWD's raw GRIB files in public cloud object storage. But we would probably only store either the model-level GRIBs, or the pressure-level GRIBs.

Update: I've just found documentation about ICON's outputs. For alb_rad, the docs say:

"Ratio of upwelling to downwelling diffuse radiative flux for wavelength interval [0.3 µm, 5.0 µm]. Values over snow-free land points are based on a monthly mean MODIS climatology. MODIS values have been limited to a minimum value of 2 %."

[rusmarko]

I only used pressure levels, which works, however, I never used model levels. I guess the best option would be the one that is the most compatible to other NWP models.

No problem about not including pressure levels in zarr on dynamical, thanks for your work!

[jacobbieker]

There are some variables on present on model levels, but not pressure levels, like specific humidity. If you are archiving only one, I'd probably recommend the model levels, as they can then be translated to pressure levels, and it wouldn't be missing any weather variables.

[stefanwayon]

+1 to @jacobbieker, though the only way I’ve found to get pressure levels from ICON model levels (cdo ap2pl) does not result in exactly the same values as the published pressure-level values, and for parts of the data that need extrapolation instead of interpolation, the values are way off.

Will post updates if I find a reliable way to get pressure levels from model levels across variables, and then you’d only need to store the model levels.

[lorenzo-solcast]

Hi, thanks for doing this work! We currently use these variables:

alb_rad
aswdifd_s
aswdir_s
asob_s_cs
aswdifu_s
asob_s
cape_con
clch
clcl
clcm
clct
h_snow
htop_con
ps
rain_con
rain_gsp
relhum_2m
snow_con
snow_gsp
t_2m
td_2m
tot_prec
u_10m
v_10m
w_snow
u: levels=(600, 900)
v: levels=(600, 900)
hsurf
clat
clon

[JackKelly]

Should we include surface_albedo (AKA alb_rad, al) in the Zarr?

ICON's documentation defines alb_rad as:

"Ratio of upwelling to downwelling diffuse radiative flux for wavelength interval [0.3 µm, 5.0 µm]. Values over snow-free land points are based on a monthly mean MODIS climatology. MODIS values have been limited to a minimum value of 2 %."

I assume most people are interested in alb_rad for forecasting solar PV power output. And I assume most PV systems are installed on snow-free land. And so it sounds like it might be far far more data efficient for folks to just download the monthly mean MODIS climatology, rather than for us to include alb_rad in the Zarr. What do folks think? (I may have misunderstood!)