Add regression test setup, scripts, and HISTORY templates

Author: biljanaorescanin

GEOSldas_App/ldas_setup

@@ -3,8 +3,10 @@
 import sys
 import argparse
 import resource
+import os, shutil
 from setup_utils import *
 from ldas        import *
+from pathlib     import Path
 
 def parseCmdLine():
     """
@@ -176,3 +178,27 @@ if __name__=='__main__':
     print ("creating batch Run scripts")
     status = ldasObj.createBatchRun()
     assert (status)
+
+    # --- Install regression driver into this experiment (copy from source tree) ---
+def _copy_regression_from_source(expdir: Path):
+    """
+    Copy util/postproc/regression from the GEOSldas source tree
+    into <EXPDIR>/regress.
+    """
+    src = Path(__file__).resolve().parents[2] / "src" / "Components" / "@GEOSldas_GridComp" / "GEOSldas_App" / "util" / "postproc" / "regression"
+    dst = expdir / "regress"
+    if not src.is_dir():
+        print(f"WARNING: regression source not found: {src}")
+        return
+
+    for root, dirs, files in os.walk(src):
+        rel = Path(root).relative_to(src)
+        (dst / rel).mkdir(parents=True, exist_ok=True)
+        for f in files:
+            srcf = Path(root) / f
+            dstf = dst / rel / f
+            shutil.copy2(srcf, dstf)
+            if dstf.suffix == ".sh":
+                mode = os.stat(dstf).st_mode
+                os.chmod(dstf, mode | 0o111)
+

GEOSldas_App/util/postproc/regression/README.md

@@ -0,0 +1,165 @@
+GEOSldas Global Regression: Model Start/Stop & Layout (6-hour tavg profile)
+
+Overview
+
+This regression is run after you have already built and executed a working GEOSldas experiment.
+
+You must have:
+
+A complete experiment directory containing:
+
+run/, input/, build/, and output/<DOMAIN>/
+
+Valid restart files under output/<DOMAIN>/rs/ens0000/
+(e.g., CURRENT.catch_internal_rst.*, CURRENT.landice_internal_rst.*)
+
+A run/LDAS.rc that defines your grid type (CF or EASE)
+
+The regression does not modify your experiment.
+It makes a self-contained sandbox copy, runs start/stop tests, and compares results.
+
+This regression runs GEOSldas in an isolated sandbox cloned from your experiment,
+forces a 6-hour time-averaged HISTORY profile (small & fast), and verifies that:
+
+Restarts are identical for a 24 h run vs 12 h + 12 h split.
+
+HISTORY (6-hour centers) is identical for the same 24 h window.
+
+It is grid-agnostic:
+
+CF (cubed-sphere): tavg24_2d_*_Nx
+
+EASE (1-D grids): tavg24_1d_*_Nt
+
+Both are normalized to 6-hour frequency: 060000 with ref_time: 000000.
+
+Your real experiment is not modified.
+Everything runs in regress/sandbox/<EXPID> (comment out the cleanup line to keep it).
+
+Regression package layout
+util/postproc/regression/
+├─ start_stop_model.sh        # regression driver
+├─ templates/
+│   ├─ HISTORY_2d.rc          # CF (2d/Nx) 6-hour tavg only
+│   └─ HISTORY_1d.rc          # EASE (1d/Nt) 6-hour tavg only
+├─ README.md                  # this file
+
+
+When a regression run starts, this structure appears under your experiment:
+
+<EXPID>/
+├─ run/                       # original job files (unchanged)
+├─ input/                     # restart, tile, forcing, etc.
+├─ build/                     # model binaries
+├─ output/<DOMAIN>/           # real experiment outputs
+│   ├─ rs/ens0000/            # restarts (catch, land-ice)
+│   ├─ cat/ens0000/           # HISTORY (tavg24_*.nc4)
+│   └─ rc_out/                # category files
+└─ regress/
+    ├─ logs/                  # regression stdout/stderr with timestamps
+    ├─ sets/                  # collected results per segment:
+    │   ├─ T1_*               # 24 h run
+    │   ├─ T2_*               # 12 h first half
+    │   └─ T3_*               # 12 h second half
+    └─ sandbox/<EXPID>/       # isolated copy used for the run
+        ├─ run/               # patched job/rc files
+        ├─ build/             # symlink to ../build
+        ├─ output/<DOMAIN>/   # new outputs written here
+        └─ scratch/           # Slurm log/stdout/err for sandbox runs
+
+
+To inspect the sandbox after a run, comment out the final cleanup line
+in start_stop_model.sh.
+By default, the sandbox is deleted after a PASS.
+
+Quick start
+
+Run your experiment once so that restart files and outputs exist.
+The regression uses these restarts as inputs.
+
+Run the regression driver
+
+cd util/postproc/regression
+./start_stop_model.sh
+
+
+Run with layout test
+
+To check layout invariance (different 1-D axis decomposition):
+
+RUN_LAYOUT=1 ALT_1D=120 ./start_stop_model.sh
+
+
+where ALT_1D can be 84, 120, 126, etc., depending on grid resolution.
+
+What the regression does
+
+Creates regress/sandbox/<EXPID> and copies your run directory.
+
+Detects grid type (CF or EASE) and applies the correct 6-hour HISTORY template.
+
+Adjusts environment variables:
+
+DO_HISTORY=TRUE
+DO_HIST=TRUE
+POSTPROC_HIST=0
+
+
+Runs:
+
+T1 – single 24-hour job
+
+T2 – 12-hour run to mid-time
+
+T3 – 12-hour run to final time
+
+Compares:
+
+RESTARTS: T1 (24 h) vs T3 (12 h + 12 h)
+
+HISTORY: T1 vs [T2 ∪ T3] at 03/09/15/21 Z centers
+
+Environment variables
+Variable	Description	Default
+EXPDIR	Experiment root (run/, input/, build/, output/)	auto-detected
+EXPDOMAIN	Domain under output/	auto-detected
+SUBMIT	Batch command (Slurm only)	sbatch
+ALT_1D	Alternate 1-D task count for layout test	required if RUN_LAYOUT=1
+NCCMP_FLAGS_TOL	Tolerant compare flags	-dmfgqMNS -t 1e-12 -T 1e-6
+HIST_STEP_SEC	Step for HISTORY collect	21600 (6 h)
+HIST_STEP_OFFSET_SEC	Center offset (+3 h)	10800
+
+Example:
+
+export EXPDIR=/discover/nobackup/borescan/par/global_regress_test/CURRENT
+export EXPDOMAIN=CF0090x6C_GLOBAL
+RUN_LAYOUT=0 ./start_stop_model.sh
+
+Comparison logic
+
+Restarts are compared with nccmp -dmfgqMNS.
+If strict compare fails, the script performs a tolerant comparison.
+
+HISTORY compares all 6-hour stamps in the same 24-hour window.
+
+Notes
+
+The 6-hour profile is used for both CF (2d/Nx) and EASE (1d/Nt).
+It reduces runtime and I/O while staying bit-for-bit safe for segmented runs.
+
+For EASE daily tavg24 tests, use day-aligned 24 h jobs only.
+Do not test sub-day segments in one job with daily tavg.
+
+If restart diffs appear only in diagnostic counters, enable:
+
+MAPL_R8_BFB=1
+MAPL_BFB_REDUCTIONS=1
+
+
+or restrict comparison to prognostic variables.
+
+
+Maintenance
+
+Templates (templates/HISTORY_1d.rc, templates/HISTORY_2d.rc) are version-controlled.
+If land-ice is disabled, the glc stream is ignored automatically by GEOSldas.