Examples, validation cases, and CI

[glwagner]

Should the new validation script be run in CI? Maybe after #67, because that touches the Validation workflow

One issue with always running validation in CI is that there will probably need to be some heavy validation cases. So we may not be able to run them all (or at least not all the time... maybe infrequently, eg nightly or weekly)

Another thought is that I think its nice to implement validation cases as "examples" which get built into the documentation (Oceananigans does this to some extent, though we may be able to do an even better job here).

However one may want to have additional validation cases that are not examples. Some of the non-example validation cases could be lightweight enough to run regularly in CI.

Maybe we can flesh out a strategy that incorporates these considerations?

Originally posted by @glwagner in #66 (comment)

[glwagner]

I'd like to collect a list of simulations that we would like to run for documenting simulation behavior and for testing.

I think we should classify simulations into 2 or 3 cases:

1. Examples, which are built into the documentation via Literate. Oceananigans has 12 examples. Our examples should both illustrate how to use the code, but also validate its behavior to some extent.

2. Non-example "validation" that can be run in CI. These would be cheaper validation cases that might be feasible to run in CI, which we want to have available as stand-alone scripts with visualization but which we don't want to build into the documentation (perhaps because we want to curate the examples carefully rather than overwhelming users with too many examples).

3. More expensive validation cases that we may want to keep scripts for in the repo, but which might not be feasible to run in CI --- or at least, not too regularly.

[glwagner]

Here are some off the top of my head (I'll work on coming up with references for these gradually, others should feel free to edit this and add):

• thermal bubble
• supercell
• squall line
• mountain wave
• LES intercomparison cases: bomex, TRMM, dycoms, gables, GATE, what else?
• cloud botany cases (Jansson, Janssens et al 2023)
• TC world setup (Cronin and Chavas 2019)
• mountain wave (Schar et al 2002)

[bischtob]

@kaiyuan-cheng can you chime in here?

[jtbuch]

We should also consider a deep convection case. One potential option is this one based on the CAIPEEX seeding campaign with cloud microphysics observations: https://iccp2020.tropmet.res.in/fnd/assets/case_study_CMW_2020-CAIPEEX.pdf with all the relevant soundings, forcings, and aircraft data here: https://drive.google.com/drive/folders/10JF2CCzOv4Nqj2G_6EJENuV4exMbIAiM

I think this will also be a useful case to have once we switch on seeding in the model because CAIPEEX has several field campaigns worth of good cloud seeding data that we can also leverage.