Corr diff - testing CI/CD - DO NOT SUBMIT by marymcglo · Pull Request #17 · MeteoSwiss/HiRAD-Gen

Commits on May 12, 2025

add inference
PetarStam
committed
Plot absolute error onto a projection for a given date

Mary McGlohon
authored and
Mary McGlohon
committed
Adjust how reshaping is done before error calcs

Mary McGlohon
authored and
Mary McGlohon
committed
plot for all channels, and against baseline

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on May 13, 2025

Add MAE output

Mary McGlohon
authored and
Mary McGlohon
committed
Fix indexing error

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on May 14, 2025

Try adding spectral graph

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jun 16, 2025

add crps code

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jun 17, 2025

restructure utils
Petar Stamenkovic
committed
restructure inference utils and samplers
Petar Stamenkovic
committed

Commits on Jul 1, 2025

refactor generation
Petar Stamenkovic
committed
Add some initial CRPS code

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 2, 2025

Fixes to CRPS

Mary McGlohon
authored and
Mary McGlohon
committed
Adds time capability to crps

Mary McGlohon
authored and
Mary McGlohon
committed
Plot CRPS per channel

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 4, 2025

Adds plotting CRPS over time (not yet tested)

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 8, 2025

Add plotting CRPS across time as well as area.

Mary McGlohon
authored and
Mary McGlohon
committed
add visualization during training
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed

Commits on Jul 9, 2025

fix calculating crps only for diffusion output
Petar Stamenkovic
committed
environment files
Petar Stamenkovic
committed
small fix environment
Petar Stamenkovic
committed

Commits on Jul 11, 2025

Make some paths relative

Mary McGlohon
authored and
Mary McGlohon
committed
Fix errors from merge process

Mary McGlohon
authored and
Mary McGlohon
committed
Move environments to ci/ directory

Mary McGlohon
authored and
Mary McGlohon
committed
Update paths for CRPS plotting

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 14, 2025

Change slurm account to new project 🎉

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 16, 2025

Starting plotting script

Mary McGlohon
authored and
Mary McGlohon
committed
add time point to logging

Mary McGlohon
authored and
Mary McGlohon
committed
eval script

Mary McGlohon
authored and
Mary McGlohon
committed
add ensemble mean MAE, and add a comparison plot over time

Mary McGlohon
authored and
Mary McGlohon
committed
Do some extra plotting tests to ensure that CRPS reduces to MAE

Mary McGlohon
authored and
Mary McGlohon
committed
Remove superfluous plotting, now that we know CRPS reduvces to MAE

Mary McGlohon
authored and
Mary McGlohon
committed
fix filename

Mary McGlohon
authored and
Mary McGlohon
committed
Move script methods into script

Mary McGlohon
authored and
Mary McGlohon
committed
Cleanup eval script a bit

Mary McGlohon
authored and
Mary McGlohon
committed
Fully separate CRPS and MAE plots for long time ranges

Mary McGlohon
authored and
Mary McGlohon
committed
Minor improvements to plotting

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 21, 2025

Update paths to environment for training scripts

Mary McGlohon
authored and
Mary McGlohon
committed
Make label opt argument

Mary McGlohon
authored and
Mary McGlohon
committed
Add test scripts/configs for regression, diffusion, and generation

Mary McGlohon
authored and
Mary McGlohon
committed
Update CI to include regression test script

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 24, 2025

Test script should not be srun if it's going to work with ci/cd

Mary McGlohon
authored and
Mary McGlohon
committed
Try to directly run from cscs.yml

Mary McGlohon
authored and
Mary McGlohon
committed
plot diurnal cycles of precip amount and wet-hour frequency over time period
leuty
committed

Commits on Jul 25, 2025

add mlflow logging
Petar Stamenkovic
committed
Merge remote-tracking branch 'origin/corr-diff' into corr-diff
Petar Stamenkovic
committed
mlflow init bug fix
Petar Stamenkovic
committed
add option to continue same run mlflow
Petar Stamenkovic
committed
add separate visualization frequency setting
Petar Stamenkovic
committed
add plots of temperature and windspeed
leuty
committed
Add script to plot the 99th all-hour percentile
leuty
committed
add visualization logging option for mlflow
Petar Stamenkovic
committed

Commits on Jul 28, 2025

enable mlflow logging to remote server
Petar Stamenkovic
committed
fix logging bug for average loss
Petar Stamenkovic
committed
update readme
Petar Stamenkovic
committed
fix indents in readme
Petar Stamenkovic
committed
simplify cscs.yml to try to get ci/cd to run

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jul 29, 2025

logs
leuty
committed

Commits on Jul 30, 2025

Commits on Jul 31, 2025

comment
leuty
committed
fix bug in std
leuty
committed
add patched diffusion configs
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed
small fixes
leuty
committed

Commits on Aug 4, 2025

Commits on Aug 5, 2025

Commits on Aug 6, 2025

Update model compilation
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed
turn on optimizations
Petar Stamenkovic
committed
add some common untility functions
leuty
committed
use get_channel_indices function
leuty
committed
remove slurm account variables

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
put more constants in plotting, use land-mask from plotting
leuty
committed
re-use concat_and_group_diurnal
leuty
committed
remove unneeded function
leuty
committed
cleanup dcycles
leuty
committed
cleanup
leuty
committed

Commits on Aug 7, 2025

Commits on Aug 13, 2025

Merge pull request #19 from leuty/dcycles
marymcglo
authored

Commits on Aug 20, 2025

Merge branch 'main' into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'main' into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Aug 25, 2025

remove dependencies from pyproject

Mary McGlohon
authored and
Mary McGlohon
committed
change conversion factor to get mm instead of cm

Mary McGlohon
authored and
Mary McGlohon
committed
add __init__ package for input_data

Mary McGlohon
authored and
Mary McGlohon
committed
Clean up copernicus processing

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Aug 26, 2025

Generalize copernicus script to work with grib/netcdf

Mary McGlohon
authored and
Mary McGlohon
committed
add copy anemoi script for copying from catalogue

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
move anemoi script to input_data directory

Mary McGlohon
authored and
Mary McGlohon
committed
Set training parameters in configs; add input/output channel names selection to dataset config and update era5cosmo dataset class; replace image saving function with results saving during training;…
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed
Enhance evaluation scripts to include regression predictions
Petar Stamenkovic
committed

Commits on Aug 27, 2025

one-off script to regrid copernicus data

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
Add input_data dependencies to container

Mary McGlohon
authored and
Mary McGlohon
committed
add grid variable

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Sep 1, 2025

more on processing copernicus data

Mary McGlohon
authored and
Mary McGlohon
committed
removing old file

Mary McGlohon
authored and
Mary McGlohon
committed
Log config into training script logs

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Sep 9, 2025

Make InfiniteSampler start from where it left off when resuming training and not restart from beginning.
Petar Stamenkovic
committed

Commits on Sep 10, 2025

Move model compilation after checkpoint loading and fix checkpoint to strip off optimization layer.
Petar Stamenkovic
committed
Refactor image_batching and image_fuse to handle input tensor dtype conversion and remove _cast_type function
Petar Stamenkovic
committed
Validate max_patch_per_gpu against batch_size_per_gpu
Petar Stamenkovic
committed

Commits on Sep 16, 2025

Add method to load static variables

Mary McGlohon
authored and
Mary McGlohon
committed
Add static variable config

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Sep 18, 2025

Add a script to check input data for missing/corrupt/nan data

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Sep 19, 2025

Commits on Sep 24, 2025

Add a less heavyweight Dockerfile that doesn't use nvidia stuff, for input data processing.

Mary McGlohon
authored and
Mary McGlohon
committed
Modifications to input data tests to optionally load the torch files

Mary McGlohon
authored and
Mary McGlohon
committed
Scripts used to reprocess data for various runs

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Oct 2, 2025

Add reprocessing script

Mary McGlohon
authored and
Mary McGlohon
committed
add functionality to have hour of the day and month of the year embedding as conditioning inputs
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed
update config to latest features
Petar Stamenkovic
committed

Commits on Oct 8, 2025

Commits on Oct 9, 2025

Eval More 10m winds (#23)
leuty
authored

Commits on Oct 20, 2025

first draft of regridding REA-L-CH1

Mary McGlohon
authored and
Mary McGlohon
committed
remove unused function

Mary McGlohon
authored and
Mary McGlohon
committed
add conversion to geo coords

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Oct 21, 2025

use plotting function from interpolate_baisic

Mary McGlohon
authored and
Mary McGlohon
committed
pip install meteodata-lab

Mary McGlohon
authored and
Mary McGlohon
committed
config file for REA-L-CH1

Mary McGlohon
authored and
Mary McGlohon
committed
skeleton for realch1 interpolatoin task

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Oct 22, 2025

Enhance generation process with randomized sampler over time.
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed

Commits on Oct 27, 2025

Wind eval streaming (#24)
leuty
authored

Commits on Oct 28, 2025

some updates to regridding (still not working)

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
regridding is actually working now.

Mary McGlohon
authored and
Mary McGlohon
committed
skeleton of full regrid script

Mary McGlohon
authored and
Mary McGlohon
committed
Regridding REA-L-CH1 script complete

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Oct 29, 2025

Add trim_edge functionality to regridding realch1

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Oct 30, 2025

do rea-l-ch1 regridding in batches to improve performance/memory usage

Mary McGlohon
authored and
Mary McGlohon
committed
small refactoring and cleanup so it is easier to re-use era5-cosmo interpolation methods

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 3, 2025

bit more skeleton to new inteprolation

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 4, 2025

working script

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 5, 2025

changes to copernicus regrid

Mary McGlohon
authored and
Mary McGlohon
committed
config updates

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 7, 2025

updates to CRPS eval

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 10, 2025

fix shading
leuty
committed

Commits on Nov 11, 2025

remove unused libraries

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 14, 2025

fix mlflow logging of learning rate
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed

Commits on Nov 17, 2025

hdf5

Mary McGlohon
authored and
Mary McGlohon
committed
cleanup hdf5

Mary McGlohon
authored and
Mary McGlohon
committed
lmdb

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 18, 2025

Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
corrected dimensions in hdf db (and new path)

Mary McGlohon
authored and
Mary McGlohon
committed
un-comment the write db commands

Mary McGlohon
authored and
Mary McGlohon
committed
Make cosmo dataset optional.

Mary McGlohon
authored and
Mary McGlohon
committed
update era location

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff

Mary McGlohon
authored and
Mary McGlohon
committed
correct dims for output grid

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 20, 2025

add bunch of configs for batches

Mary McGlohon
authored and
Mary McGlohon
committed
correct some things

Mary McGlohon
authored and
Mary McGlohon
committed
remove unused script

Mary McGlohon
authored and
Mary McGlohon
committed
add interpolate script

Mary McGlohon
authored and
Mary McGlohon
committed
oops commented line in the middle of the script isn't good

Mary McGlohon
authored and
Mary McGlohon
committed
refactor interpolate_basic script to be less complex

Mary McGlohon
authored and
Mary McGlohon
committed
add balfrin scripts

Mary McGlohon
authored and
Mary McGlohon
committed
Merge branch 'interpolate' of https://github.com/MeteoSwiss/HiRAD-Gen into interpolate

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 25, 2025

Add comments and make interpolate_basic a bit more generalizable.

Mary McGlohon
authored and
Mary McGlohon
committed
remove ensemble dimension from processing

Mary McGlohon
authored and
Mary McGlohon
committed
separate out era5-cosmo specific main method into its own script

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 26, 2025

correct error in anemoi open dataset

Mary McGlohon
authored and
Mary McGlohon
committed
wrong arg name

Mary McGlohon
authored and
Mary McGlohon
committed
small fixes, script runs now

Mary McGlohon
authored and
Mary McGlohon
committed
add methods for netcdf interpolation (draft)

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Nov 27, 2025

mostly working netcdf regridding refactor. However, there is an off-by-one error in the time parsing I am still chasing down.

Mary McGlohon
authored and
Mary McGlohon
committed
fixed off-by-one error

Mary McGlohon
authored and
Mary McGlohon
committed
cleanup and comments

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Dec 4, 2025

add logging, and small refactor of regrid_realc1 to more closely match style in interpolate_basic

Mary McGlohon
authored and
Mary McGlohon
committed
a few processing scripts

Mary McGlohon
authored and
Mary McGlohon
committed
start of era5->realch1 processing script

Mary McGlohon
authored and
Mary McGlohon
committed
add a bunch of configs

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Dec 5, 2025

fix bug in channel indexing in dataset
Petar Stamenkovic
committed
Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed

Commits on Dec 8, 2025

add dependencies (attempt)

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Dec 9, 2025

attempted fix

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Dec 11, 2025

regridding dependencies and bug fixes

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Dec 16, 2025

stash some stuff

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Dec 17, 2025

add static and stats calc to ralch1

Mary McGlohon
authored and
Mary McGlohon
committed
reorg files

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Dec 19, 2025

adapt snapshots to be able to select channels to plot
Petar Stamenkovic
committed
fix bug in snapshots
Petar Stamenkovic
committed
update dataset for realch1
Petar Stamenkovic
committed
update plotting to use new channel metadata to string functions
Petar Stamenkovic
committed

Commits on Dec 23, 2025

add script to make video from snapshots
Petar Stamenkovic
committed
add script to calculate datasets stats after transforming data
Petar Stamenkovic
committed

Commits on Dec 24, 2025

enable torch compile during distributed training
Petar Stamenkovic
committed
enable time logging of data fetching
Petar Stamenkovic
committed
add realch1 dataset training configs
Petar Stamenkovic
committed

Commits on Jan 5, 2026

Refactor evaluation scripts for improved configuration handling and usability across different target datasets
Petar Stamenkovic
committed
changed config files
Petar Stamenkovic
committed
modify dataset configs
Petar Stamenkovic
committed
clean up configs further
Petar Stamenkovic
committed

Commits on Jan 6, 2026

add config for cosmo model evaluation
Petar Stamenkovic
committed

Commits on Jan 14, 2026

Merge branch 'interpolate' into corr-diff. Most of this will be obsolete soon, but want to go ahead and merge the latest version before we delete it, to preserve history and keep records of how tra…

Mary McGlohon
authored and
Mary McGlohon
committed

Commits on Jan 15, 2026

Merge branch 'corr-diff' of https://github.com/MeteoSwiss/HiRAD-Gen into corr-diff
Petar Stamenkovic
committed

Commits

Commits on May 12, 2025

Commits on May 13, 2025

Commits on May 14, 2025

Commits on May 15, 2025

Commits on May 16, 2025

Commits on May 21, 2025

Commits on May 22, 2025

Commits on May 23, 2025

Commits on May 26, 2025

Commits on May 27, 2025

Commits on Jun 5, 2025

Commits on Jun 12, 2025

Commits on Jun 13, 2025

Commits on Jun 16, 2025

Commits on Jun 17, 2025

Commits on Jul 1, 2025

Commits on Jul 2, 2025

Commits on Jul 4, 2025

Commits on Jul 8, 2025

Commits on Jul 9, 2025

Commits on Jul 11, 2025

Commits on Jul 14, 2025

Commits on Jul 16, 2025

Commits on Jul 21, 2025

Commits on Jul 24, 2025

Commits on Jul 25, 2025

Commits on Jul 28, 2025

Commits on Jul 29, 2025

Commits on Jul 30, 2025

Commits on Jul 31, 2025

Commits on Aug 4, 2025

Commits on Aug 5, 2025

Commits on Aug 6, 2025