Skip to content

Fix plot_train #1607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
clessig wants to merge 113 commits into
base: develop
Choose a base branch
from
Draft

Fix plot_train #1607

clessig wants to merge 113 commits into from

Conversation

@clessig
Copy link
Collaborator

@clessig clessig commented Jan 14, 2026

Description

Adapt plot_train to new, more complex structure of loss terms

Issue Number

Closes #1606

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

clessig added 30 commits December 30, 2025 23:14
@clessig
Partially revised config; model is still missing but proper setup of …
b98d074 
…training_config and validation_config
@clessig
Changes necessary due to changed position of time keys and of run_id
bb12d5a
@clessig
Handling of multiple loss terms / target_aux_calculators and non-Loss…
d510956 
…Physical ones.
@clessig
Changed position of run_id in config
793578e
@clessig
Add function to extract batch size from mode_cfg
6eabd27
@clessig
Changed position of run_id in config
d350b38
@clessig
Changes due to revised config. Also proper handling of target_aux_cal…
78b17af 
…culator and various other details cleaned up
@clessig
Revised config structure, in particular for losses, and related changes
d8a1291
@clessig
Add missing copyright and minor changed to to_device()
0d4e471
@clessig
Moved sanity checking from trainer here. Also learning_rate sub_part …
b99b5c9 
…of config is passed to LRScheduler, which leads to major simplifications
@clessig
Minor cleanups
9ef940b
@clessig
Changes due to changed structure of losses in config
868e595
@clessig
Changes due to changed structure of losses in config
f005ef0
@clessig
Minor changes due to changed position of run_id in config
7b1d189
@clessig
Minor changes to accomodate new config, in particular target_aux_calc…
53eb0d0 
…ulator config
@clessig
Support batch_size > 1. Clean up of various smaller parts
cdbb696
@clessig
Clean up and implementation for batch_size > 1.
0b99f3e
@clessig
Fix to sharding problem with FSDP2
7d1226f
@clessig
Removed scatter offset computation which now happens on the fly in th…
0ca381d 
…e model
@clessig
Changes for revised config, simplify overall where possible
4d67ad2
@clessig
Fix issues with source-target sample generation and matching. Work in…
66c83a2 
… progress
@clessig
Linting
fff5749
@clessig
Linting
f28874b
@clessig
Linting
192930a
@clessig
Linting
32243f3
@clessig
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into cles…
0d11f87 
…sig/develop/fix_config_1534
@clessig
Type hint
0b900d3
@clessig
Linting
132a2be
@clessig
Linting
070e859
@clessig
Linting
39d02ef
clessig and others added 27 commits January 11, 2026 19:18
@clessig
Restoring default for decoder_type
a60aab5
@iluise
Merge branch 'develop' into iluise/fix-lead-time
c335d97
@iluise
update to develop
33755c7
@iluise
lint
ecbd897
@clessig
Merge branch 'iluise/fix-lead-time' of github.com:ecmwf/WeatherGenera…
c714be2 
…tor into clessig/develop/fix_config_1534
@clessig
Fixed problem where parameter was expected in old config place
da42ad6
@clessig
Fixed linting
e4e3922
@clessig
Simplified interface
d741e2f
@clessig
Re-enabled forecast step and location weighting
bcd561f
@clessig
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into cles…
8b0bb12 
…sig/develop/fix_config_1534
@clessig
Merge branches 'develop' and 'clessig/develop/fix_config_1534' of git…
72d92f9 
…hub.com:ecmwf/WeatherGenerator into clessig/develop/fix_config_1534
@clessig
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into cles…
41c85a3 
…sig/develop/fix_config_1534
@clessig
Linting
a4f3eed
@clessig
Using new option to have validate_before_training as an int arg that …
78a7cd4 
…allows to specify number of samples;

Added copyright statement
@clessig
Added option to have validate_before_training as int argument (specif…
4bfea4b 
…yiung the number of samples).

Fixed some minor subtle problems in validate() to fully distinguish validation and testing.
@clessig
Refactored correspondence parsing
9123bae
@sophie-xhonneux @clessig
Sophiex/dev/teacher overrides (#1557)
a218319 
* Add option to modify teacher TODO fix ema update

* Fix EMA under teacher and student model differences

* Attempt to revert newline

* Raise error if teacher has weights not in student

* Clessig/sophiex/dev/teacher overrides (#1585)

* Simplified error message

* Added support for target_and_aux configs

* Fix bug that validation EMA params are not used

* Removing unused/superfluous function

* Removed debug statement

* Changed config so that target_aux params are specified as dict at the appropriate place

---------

Co-authored-by: Christian Lessig <[email protected]>
@clessig
Fixed missing default value
841e027
@sophie-xhonneux @clessig
Bilinear decoder: adapt code for batchsize > 1 (#1592)
5f2cb75 
* Adapt code for batchsize > 1

* Fixed comment

---------

Co-authored-by: Christian Lessig <[email protected]>
@clessig
Changed defaults
71d43df
@clessig
Linting
80e9181
@clessig
Fixed linting issue
c4bd337
@clessig
Reverting to ERA5-only as default
aae83c0
@clessig
Fixed problem with train_continue
eb235de
@clessig
Adding filtering of config based on enabled/disabled
b3cba32
@clessig
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into cles…
6541f06 
…sig/develop/fix_enabled_flag_1578
@clessig
Fixed very hacking to get some plots
0e8c739
@github-actions github-actions bot added the model Related to model training or definition (not generic infra) label Jan 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

model Related to model training or definition (not generic infra)

Projects

WeatherGen-dev
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

plot_train is broken

4 participants

@clessig @iluise @sophie-xhonneux