-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Open
Copy link
Labels
enhancementNew feature or requestNew feature or request
Description
Goal: Add DWD's ICON-EU NWP to Dynamical as a single sharded Zarr v3. Update the Zarr directly from the GRIB files published on DWD's HTTP server. DWD deletes the GRIB files after 24 hours. Backfill from Open Climate Fix's ICON-EU archive on Hugging Face.
My tasks:
- Email everyone who downloaded ICON-EU from OCF's archive on Hugging Face
- Draft email. Ask them which NWP variables they'd like. Invite their feedback on Which NWP variables and vertical levels to select for ICON-EU? #176. Say that the ICON-EU archive on HF will be turned off 6 months after the backfilled ICON-EU dataset appears on Dynamical. Explain that the Dynamical dataset will make their lives much easier 🙂: ARCO FTW.
- Share draft with @aldenks & @devsjc & Emily (at OCF).
- Send email after Alden & Sol & Emily have approved the draft.
- Update the README on the ICON-EU Hugging Face archive with text from the email.
- Post to LinkedIn.
- Email DWD
- How to handle the slightly different shapes of OCF's ICON-EU archive on Hugging Face? #183
- Decide on a chunking & sharding scheme for the ICON-EU Zarr.
- ICON-EU step 1: Implement
template_config.py#184 - ICON-EU step 2: Implement
region_job.py#226 - Copy files from DWD FTP using
rclone#360 (which would replace PRs Calculate which FTP files to transfer to object storage for ICON-EU #331 & Copy files from FTP to anObjectStore#318) - Save ICON-EU GRIB files from DWD's FTP server to cloud object storage #258
- Research into speeding up
rclone copyfrom DWD's FTP server to Source Co-Op #390 - Finish
RegionJobfor ICON-EU #381 - Speed up
rclonecopy from DWD's FTP server to Source Co-Op #392 - PR to instantiate and return
ReformatCronJobandValidationCronJobfromDwdIconEuForecastDataset.operational_kubernetes_resourcesand test - Update the start date in
template_config.pyjust before launching the live updates - Launch live-updating 4-inits-per-day ICON-EU Zarr on dynamical.org!
- Write more unit-tests (especially for the code introduced in PR Production code to copy ICON-EU grib files from FTP to Source Co-Op #370)
- Update the README on Hugging Face with an exact date for when we'll stop updating ICON-EU on HF
- Update the code to handle DWD's planned new file naming conventions, and check DWD's Open Data news for an update on when DWD will actually change their naming convention. See this comment for more info.
- If I have time, then also Backfill DWD ICON-EU #343
Please shout if I've missed anything!
(CC'ing @jacobbieker for visibility, because Jacob wrote the original code to archive ICON-EU to HF! Thanks Jacob!)
jacobbieker and aldenks
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request