Conversation
… we want to factor it out of the rule
* Distinguish between primary runs ('candidates') and secondary runs
* Docstrings
* Adopt forecast intervals including the end point * Fix parsing * Experiments work * Update config/forecasters.yaml * Align init times to availabiliy of COE * run pre-commit * Change README to COSMO-E availability --------- Co-authored-by: Jonas Bhend <jonasbhend@users.noreply.github.com> Co-authored-by: Jonas Bhend <jonas.bhend@meteoswiss.ch>
* draft changes * rename workspace resources dir * working for config/forecasters.yaml * improve logging * works for interpolators.yaml * re-add get_leadtime function * refactor run directives into script
* add region averages * add regions to config * Add regions to verification module, scripts, and rules * add stratification to forecaster config and fix typo * fix dict indexing * fix append error * read lon/lat from obs dataset * Add inner verification domain * Add missing dependency * add plots by region * Add regions to dashboard * Fix dashboard * Add region name and initializations to plot title (and remove header div) * Add support for multiple regions * Fix legend
…e-to-generate-namelist
dnerini
left a comment
dnerini
left a comment
There was a problem hiding this comment.
looking very nice @andreaspauling ! I have added a few initial thoughts form a quick look into your changes, but I plan to have a closer look soon!
| rule prepare_mec_input: | ||
| input: | ||
| src_dir=OUT_ROOT / "data/runs/{run_id}/{init_time}/grib", | ||
| inference_ok=OUT_ROOT / f"run_inference_all.{EXPERIMENT_HASH}.ok", |
There was a problem hiding this comment.
| inference_ok=OUT_ROOT / f"run_inference_all.{EXPERIMENT_HASH}.ok", | |
| inference_okfile=rules.execute_inference.output.okfile,, |
| # prepare_mec_input: setup run dir, gather observations and model data in the run dir for the actual init time | ||
| rule prepare_mec_input: | ||
| input: | ||
| src_dir=OUT_ROOT / "data/runs/{run_id}/{init_time}/grib", |
There was a problem hiding this comment.
There is no rule giving this an output, so this should to the very list trigger some warnings from snakemake. You could specify it as a parameter instead.
| set -euo pipefail | ||
|
|
||
| # Run MEC inside sarus container | ||
| # Note: pull command currently needed only once to download the container |
There was a problem hiding this comment.
the pull command could then be factored out into a separate rule that is run only once before launching all the parallel MEC jobs
| RESULTS_DIR = OUT_ROOT / "results" / EXPERIMENT_NAME | ||
|
|
||
| # prefer one rule because snakemake complains about ambiguous rules (same output) | ||
| ruleorder: prepare_inference_forecaster > prepare_inference_interpolator |
There was a problem hiding this comment.
need to have a closer look at this, I don0't understand why this problem would appear with your changes
There was a problem hiding this comment.
Without it snakemake complains. Thanks for having a closer look at it.
| expand( | ||
| OUT_ROOT / "data/runs/{run_id}/fdbk_files/verSYNOP_{init_time}.nc", | ||
| init_time=[t.strftime("%Y%m%d%H%M") for t in REFTIMES_MEC], | ||
| run_id=collect_all_candidates(), |
There was a problem hiding this comment.
| run_id=collect_all_candidates(), | |
| run_id=CANDIDATES, |
workflow/Snakefile
Outdated
| run_id=CANDIDATES, | ||
| ), | ||
| output: | ||
| inference_ok=touch(OUT_ROOT / f"run_inference_all.{EXPERIMENT_HASH}.ok") |
There was a problem hiding this comment.
This is thought to ensure all inference output is there before the MEC rules start
5 participants
Add the MEC workflow. The new parts are in green in the DAG: snakemake_dag.pdf
For each valid date a MEC case is set up and run. This includes:
All MEC cases can be removed once the final feedback file is produced (removal not yet implemented).
Remarks/Questions: