DWD ICON-EU forecast task list & general discussion (for live-updating Zarr)

[JackKelly]

Goal: Add DWD's ICON-EU NWP to Dynamical as a single sharded Zarr v3. Update the Zarr directly from the GRIB files published on DWD's HTTP server. DWD deletes the GRIB files after 24 hours. Backfill from Open Climate Fix's ICON-EU archive on Hugging Face.

My tasks:

• Email everyone who downloaded ICON-EU from OCF's archive on Hugging Face
  • Draft email. Ask them which NWP variables they'd like. Invite their feedback on Which NWP variables and vertical levels to select for ICON-EU? #176. Say that the ICON-EU archive on HF will be turned off 6 months after the backfilled ICON-EU dataset appears on Dynamical. Explain that the Dynamical dataset will make their lives much easier 🙂: ARCO FTW.
  • Share draft with @aldenks & @devsjc & Emily (at OCF).
  • Send email after Alden & Sol & Emily have approved the draft.
  • Update the README on the ICON-EU Hugging Face archive with text from the email.
  • Post to LinkedIn.
• Email DWD
  • Draft email. (Note to self: Check notes on meeting minutes for what to include in this email).
  • Share draft with @aldenks & @devsjc.
  • Send email.
• How to handle the slightly different shapes of OCF's ICON-EU archive on Hugging Face? #183
• Decide on a chunking & sharding scheme for the ICON-EU Zarr.
• ICON-EU step 1: Implement template_config.py #184
• ICON-EU step 2: Implement region_job.py #226
• Copy files from DWD FTP using rclone #360 (which would replace PRs Calculate which FTP files to transfer to object storage for ICON-EU #331 & Copy files from FTP to an ObjectStore #318)
• Save ICON-EU GRIB files from DWD's FTP server to cloud object storage #258
• Research into speeding up rclone copy from DWD's FTP server to Source Co-Op #390
• Finish RegionJob for ICON-EU #381
• Speed up rclone copy from DWD's FTP server to Source Co-Op #392
• PR to instantiate and return ReformatCronJob and ValidationCronJob from DwdIconEuForecastDataset.operational_kubernetes_resources and test
• Update the start date in template_config.py just before launching the live updates
• Launch live-updating 4-inits-per-day ICON-EU Zarr on dynamical.org!
• Write more unit-tests (especially for the code introduced in PR Production code to copy ICON-EU grib files from FTP to Source Co-Op #370)
• Update the README on Hugging Face with an exact date for when we'll stop updating ICON-EU on HF
• Update the code to handle DWD's planned new file naming conventions, and check DWD's Open Data news for an update on when DWD will actually change their naming convention. See this comment for more info.
• If I have time, then also Backfill DWD ICON-EU #343

Please shout if I've missed anything!

(CC'ing @jacobbieker for visibility, because Jacob wrote the original code to archive ICON-EU to HF! Thanks Jacob!)

[aldenks]

Wahoo! Excited for your hands on support

[jacobbieker]

Excited to see this in an easier-to-use format! Is there plans for ICON-Global as well?

[JackKelly]

We're going to see how things go with ICON-EU, and then consider ICON-Global!

[JackKelly]

Slightly off topic, but: I've just learnt that DWD have an AI weather model (AICON) in the works.

(DWD are giving a talk titled "AICON - Introducing ML-based weather forecasting at DWD" on Wednesday 17 Sept at the ECMWF UEF2025 workshop.)

[JackKelly]

(Sorry for not starting work on the PR to implement region_job yet. Work got busy! I haven't forgotten about it, I promise!)

[jacobbieker]

Just a note, but DWD is going to be changing how the open data is presented on the open data server. The main changes will be removing the bz2 compression, and changing everything to follow this format:

The following new variants exist for the deterministic models:

/v1/m/<model-name>/p/<parameter-shortname>/r/<run>/s/<step>.grib2
/v1/m/<model-name>/p/<parameter-shortname>/lvt1/<level-type>/lv1/<level>/r/<run>/s/<step>.grib2
/v1/m/<model-name>/p/<parameter-shortname>/wvl1/<wavelength>/lvt1/<level-type>/lv1/<level>/r/<run>/s/<step>.grib2
There are the following new variants for the ensemble models:

/v1/m/<model-name>/p/<parameter-shortname>/lvt1/<level-type>/lv1/<level>/r/<run>/e/<ensemble-member>/s/<step>.grib2
/v1/m/<model-name>/p/<parameter-shortname>/r/<run>/e/<ensemble-member>/s/<step>.grib2
/v1/m/<model-name>/p/<parameter-shortname>/wvl1/<wavelength>/lvt1/<level-type>/lv1/<level>/r/<run>/e/<ensemble-member>/s/<step>.grib2
Here is <level-type> the numeric GRIB-header typeOfFirstFixedSurface. . .. Examples of <level-type>:

100: Isobaric Surface / "Pressure"
106: Depth Below Land Surface
150: Generalized Vertical Height Coordinate

Currently, the ICON-ART models follow this on the server, but the plan to move everything to that same structure. Not sure of the timeline when.

[JackKelly]

OK, thanks for flagging!

[JackKelly]

Ah yes, I can see the new URLs used here (for example): https://opendata.dwd.de/weather/nwp/v1/m/icon-art-eu/p/CAPE_ML/r/

The changes to the URL structure isn't a concern. That's easy to change.

But what does break us is that new "data sets [will] generally only [be published on the DWD HTTP server] in the native triangular grid." At present, we're using the regular lat-lon grids. (Update: After talking to DWD, it sounds like they may keep ICON-EU (specifically) in the same lat lon grid as the current archive)