GCM-Driven SCM Calibration Pipeline: v1 #3765
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
costachris
force-pushed
the
cc/edmf_scm_cal_v2
branch
from
bd0483c to
d62f6ae
Compare
costachris
force-pushed
the
cc/edmf_scm_cal_v2
branch
2 times, most recently
from
0f6d3fd to
44cf596
Compare
…r both shallow and deep convection. Includes option for defining stretched calibration grid. Add cal grid to edmf_ensemble_stats gcm driven calibration updates: restart, plotting support for variable grids, add plot prior and normalize var scripts, add microphysics cal option Add NN mixing length closure, add load pretrain weights logic in prior Add clippings for NN inputs, dev calibration Add scm_runner tools to run SCMs (in parallel) across cases for a given parameter set scm runner updates limit nn mixing length by smag length + (1/z), and add new dz input variable. Remove problematic deep convective cases from library. Update minibatcher to specify cases for each epoch. Add serialize_std_model with default in run_cal to allow different priors for NN weights and biases. Increase request limits for top-level slurm sbatch script, preventing minor memory leak from killing calibration. Increase noise by an order of mag. Update, improve restart calibration + allow to work with NNs. Limit number of processes per ensemble member by 5 to avoid queue waits . Add back deep deep conv cases, clip ql, qi above 0 in les cases. Fix get_optimal_particle nearest neighbor mean getter. Simplify nn prior creation at beginning of run_calibration. Request less time on nodes for scm runs. Limit num_cpu to 5 in restart script. Increase noise, use batch size = 20. And cfsite info to plot_ensemble plots. Add radiation metrics to runner model config. Add nn_helper functions. Add precal prior toml. Increase noise and batch size (30). Increase prior sigma for NN (0.05 for weights). Lower prior std for mixing_length_diss_coeff. Turn off accelerator. Make shallow library default in get_les_metadata. Make leaky relu NN default. Change default noise with leaky relu (std_weight = 0.1, std_bias = 0.00001). Increase default runtime for restart for ens members to 180 mins. Increase resource requests for scm_runner. Make leaky relu NN default (and write to scratch). Pass args to scm_runner in sbatch script. Add CLI option for scm_runner, more flexible parsing of cfsite (for different AMIP experiments). Switch between NN and standard priors in restart calibration script Add batch size logic in edmf_ensemble_stats Add forcing parameters as additional tomls
costachris
force-pushed
the
cc/edmf_scm_cal_v2
branch
from
44cf596 to
ecf5302
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Contains improvements and updates to the EDMF SCM pipeline
Data-driven Closure Support:
Tooling Improvements: