NGWPC NWM PI-3 Formulation Selection Delivery 2026-02-25

.github/workflows/documentation.yml

@@ -0,0 +1,34 @@
+name: documentation
+
+on:
+  push:
+    branches: [ "documentation" ]
+
+permissions:
+  contents: write
+
+jobs:
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: |
+          pip install -r docs/source/requirements.txt
+          pip install .
+      - name: Debug Python imports
+        run: |
+          python -c "import nwm_region_mgr.parreg.config_schema; print('Import OK')"
+      - name: Sphinx build
+        run: |
+          sphinx-build docs/source docs/build
+      - name: Deploy to GitHub Pages
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_branch: gh-pages
+          publish_dir: docs/build/
+          force_orphan: true

.gitignore

@@ -25,7 +25,11 @@ __pycache__/
 # Editor configs
 .vscode/
 
+# Documentation
+docs/build/*
+docs/source/API/*
+!docs/source/API/index.rst
 # Specific project paths
 pkgs/parreg/build
 pkgs/parreg/tests/test_data/test_outputs
-nwm-rte
+nwm-rte

LICENSE

@@ -0,0 +1,14 @@
+Copyright 2025 Raytheon Company
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation
+and/or other materials provided with the distribution.
+
+Licensed under: https://opensource.org/license/bsd-2-clause
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+All rights reserved. Based on Government sponsored work under contract GS-35F-204GA.
+
+-----------------

README.md

@@ -1,90 +1,105 @@
-# nwm-region-mgr
+# Formulation and Parameter Regionalization for the NextGen Framework
+[![Build](https://img.shields.io/github/actions/workflow/status/ngwpc/nwm-region-mgr/ci.yaml?branch=main)](.github/workflows/ci.yml)
+[![License: BSD 2-Clause](https://img.shields.io/badge/License-BSD%202--Clause-orange.svg)](https://opensource.org/license/bsd-2-clause)
+[![Release](https://img.shields.io/github/v/release/ngwpc/nwm-region-mgr)](https://github.com/NGWPC/nwm-region-mgr)
+![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-orange.svg)
+![Linter: Ruff](https://img.shields.io/badge/linter-ruff-orange)
 
-## Name
-NGEN/NWM Regionalization
+<img src="docs/source/_images/overview.png" alt="overview" width="600"/>
 
-## Description
-This repository includes packages for conducting formulation and parameter regionalizations for NextGen modules,
-as well as scripts/workflows to run ngen simulations and evaluation.
-- parreg: parameter regionalization
-- formreg: formulation regionalization
-- utils: utility functions shared by parreg and formreg
+
+`nwm_region_mgr` is a Python package for identifying optimal model formulations and parameter values in ungauged catchments. It leverages calibration data from gauged catchments and catchment attributes to improve hydrologic modeling and forecasting skill across regions, playing a key role in the NextGen and NWM ecosystem.
+
+
+## Key Features
+
+- **Formulation Regionalization** – Ranks NextGen model formulations for ungauged catchments based on performance at gauged sites in a region.
+- **Parameter Regionalization** – Estimates parameter values by intelligently transferring calibrations across catchments based on similarity.
+- **Clustering Methods** – Groups catchments with shared hydrologic characteristics using multiple clustering approaches.
+- **Distance methods** – Identify donors using distance metrics that characterize catchment similarity/dissimilarity.
+- **Diagnostic Plots** – Generates diagnostic plots and maps that explain formulation and parameter choices.
+- **Scalable Workflows** – Efficiently supports studies from individual watersheds to CONUS-wide applications.
+- **Customizable Configurations** – Full control of workflows via human-readable config files.
 
 ## Docker Run Time Environment (RTE)
-### Step 0. Build Docker image
-`Note` This step is only necessary if a docker image doesn't already exists or if updates to the code base have been implemented. The short flag `-d` can also be used in place of `--docker`
+### Step 0. Build Docker images and download data
+Follow NMW-RTE [README](https://github.com/NGWPC/nwm-rte/blob/development/README.md) to build Docker images and download sample data.
+
+> **Note:** A single ngen RTE Docker image needs to be built before continuing to the regionalization steps below. 
+> Make sure that RTE's build script runs successfully.  RTE's default run script (for forecasting) is not used by the regionalization workflow and should be skipped.
 
-```bash
-cd nwm-region-mgr
-```
-```bash
-./region-mgr.sh -docker
-```
 ### Step 1. Run regionalization
-#### a) Run both formulation and parameter regionalization together:
+
+First, navigate to nwm-rte directory, e.g.,
 ```bash
-./region-mgr.sh --formreg --parreg
+cd ~/ngwpc/nwm-rte
 ```
-#### b) Run formulation regionalization alone:
-The short flag `-f` can also be used in place of `--formreg`.
+
+#### a) Run formulation regionalization alone (no parreg):
+The short flag `-f` can also be used in place of `--formreg`. Prior to running, configure the settings in `configs/config_general.yaml` and `configs/config_formreg.yaml`.
 ```bash
-./region-mgr.sh --formreg
+time ./ngen_rte_run_region.sh --formreg
 ```
-#### c) Run parameter regionalization alone:
-The short flag `-p` can also be used in place of `--parreg`.
+Typically this step can be skipped since parameter regionalization also runs formulation regionalization as a prerequisite.
+
+#### b) Run parameter regionalization (formreg is also ran as a prerequisite):
+The short flag `-p` can also be used in place of `--parreg`. Prior to running, configure the settings in `configs/config_general.yaml`, `configs/config_formreg.yaml` and `configs/config_parreg.yaml`.
 ```bash
-./region-mgr.sh --parreg
+time ./ngen_rte_run_region.sh --parreg
 ```
 ### Step 2. Run NGEN
 Run a NGEN simulation:
 
-The short flag `-n` can also be used in place of `--ngen`.
+The short flag `-n` can also be used in place of `--ngen`. Prior to running, configure the settings in `configs/config_ngen.yaml`.
 ```bash
-./region-mgr.sh --ngen
+time ./ngen_rte_run_region.sh --ngen
 ```
 ### Step 3. Run Evaluation
 Run an evaluation:
 
-The short flag `-e` can also be used in place of `--eval`.
+The short flag `-e` can also be used in place of `--eval`. Prior to running, configure the settings in `configs/config_eval.yaml`.
 ```bash
-./region-mgr.sh --eval
+time ./ngen_rte_run_region.sh --eval
 ```
-### Steps 0-4
-Alternatively the user can run steps 0-4 in all at once in series:
+
+### To run all steps in one command
 ```bash
-./region-mgr.sh -dfpne
+time ./ngen_rte_run_region.sh --parreg --ngen --eval
 ```
-## Desktop Run Time Environment (RTE)
+
+## Desktop/Workspace
 ### Clone & Build
 
 #### 1. clone ngen-region-mgr from Github
 
 ```bash
-cd [NGEN_REG_ROOT]
-git clone -b development --recurse-submodules https://github.com/NGWPC/nwm-region-mgr.git
+git clone https://github.com/NGWPC/nwm-region-mgr.git
+cd nwm-region-mgr
 ```
 
 #### 2. create python venv
-
+Create a virtual environment to isolate the dependencies of this library from your base Python environment.
 ```bash
-cd [VENV_ROOT]
-/usr/bin/python3.11 -m venv venv
-source venv/bin/activate
+python3.11 -m venv .venv
+source .venv/bin/activate
 pip install --upgrade pip
 ```
+
 #### 3. install nwm-region-mgr
 
+You will then be able to install nwm_region_mgr. There are a few variants that users may be interested in.
+
 ```bash
-cd [NGEN_REG_ROOT]
-pip install . # to install formreg and utils only
-# OR
-pip install .[parreg] # to install formreg, utils, and parreg 
+# Regular package install
+pip install .
+# Install the package in edit mode (for development)
+pip install -e .
+# Install additional dependencies for parameter regionalization
+pip install .[parreg]
+# Install dependencies for docs and dev
+pip install .[docs,dev]
 ```
 
-where [VENV_ROOT] and [NGEN_REG_ROOT] refer to the directory to install python venv and nwm-region-mgr 
-in your local workspace, respectively
-
-
 ### STEP 1: Run regionalization to produce regionalized parameters and formulations
 
 #### 1) Set up configuration yaml files
@@ -94,55 +109,30 @@ Three yaml config files are needed to run regionalization
 - **onfig_formreg.yaml**: specific settings for the formulation regionalization process.
 - **config_parreg.yaml**: specific settings for the parameter regionalization process.
 
-Follow the sample [config files](https://github.com/NGWPC/nwm-region-mgr/tree/development/sample_files/configs) to set up the configurations for your regionalization application as needed.
-
-Sample input data can be downloaded from **s3://ngwpc-dev/Yuqiong.Liu/repos/nwm-region-mgr**
+Follow the sample config files (nwm_region_mgr/configs) to set up the configurations
+for your regionalization application as needed.
 
+Sample input data can be downloaded from **s3://ngwpc-dev/regionalization/inputs**
 
 #### 2) Run the regionalization script
 
 ```bash
-python [NGEN_REG_ROOT]/nwm-region-mgr/regionalization.py [COFIG_DIR]
+python [NGEN_REG_ROOT]/nwm-region-mgr/regionalization.py [COFIG_DIR] [REG_TYPE]
 ```
-Where: 
+Where:
 - [NGEN_REG_ROOT] refers to the directory where nwm-region-mgr is installed
 - [COFIG_DIR] refers to the directory containing the three config files as noted in 1), e.g.,
+- [REG_TYPE] refers to the type of regionalization to run, either 'formreg' (formulation regionalization only) or 'region' (parameter regionalization, which also runs formulation regionalization first if not done already). If not specified, the default is 'region'.
 
 ```bash
-python regionalization.py sample_files/configs
+python regionalization.py configs formreg # to run formulation regionalization only
+python regionalization.py configs region # to run parameter regionalization (and formulation regionalization if not done already)
 ```
 
 ### STEP 2: Run NGEN simulation with regionalized parameters
 
-- #### Run natively in workspace
-    - ##### 1) Install [ngen](https://github.com/NGWPC/ngen) and all submodules in its own venv
-        You may want to follow the following Confluence pages:
-        - [Clone ngen](https://confluence.nextgenwaterprediction.com/display/NGWPC/Clone+NGWPC+GitHub+Code)
-        - [Build ngen](https://confluence.nextgenwaterprediction.com/display/NGWPC/Build+ngen+completely)
-
-    - ##### 2) Install [mswm](https://github.com/NGWPC/nwm-msw-mgr) in its own venv
-
-    - ##### 3) Activate MSWM venv, e.g.
-        ```bash
-        source ~/repos/nwm-msw-mgr/venv/bin/activate
-        ```
-    - ##### 4) Set up MSWM configuration as shown in [run_ngen_vpu.sh](https://github.com/NGWPC/nwm-region-mgr/blob/development/run_ngen_vpu.sh)
-
-    - ##### 5) Run MSWM and ngen simulation
-        ```bash
-        cd ~/repos/nwm-region-mgr
-        ./run_ngen_vpu.sh
-        ```
-    - ##### 6) Check inputs, outputs and logs
-    All input, output and log files from running MSWM and NGEN can be found in *[work_dir]/regionalization/[run_name]/[vpu]* 
-(as defined in **run_ngen_vpu.sh**)
-
-    - ##### 7) If ngen fails at t-route
-    Check if all NGEN cat-*.csv and nex-*.csv output files are generated; if yes,
-    run t-route separately from the Output directory where ngen outputs are located, e.g.,
-        ```bash
-        python -m nwm_routing -f -V4 ../Input/vpu_09_troute_config_region.yaml
-        ```
+To void complications from building ngen and its submodules locally, we recommend you always run NGEN simulation 
+with regionalized parameters and formulations from a Docker container. Follow instructions from the **Docker Run Time Environment (RTE)** section above. 
 
 ### STEP 3: Evaluate NGEN simulation with nwm.verf
 
@@ -154,7 +144,7 @@ Follow example config at [config_eval.yaml](https://github.com/NGWPC/nwm-region-
 
 Check out what metrics are currently supported [here](https://confluence.nextgenwaterprediction.com/display/NGWPC/Forecast+Verification+%28ngen-verf%29%3A+Configuration)
 
-Sample input data can be downloaded from **s3://ngwpc-dev/Yuqiong.Liu/repos/nwm-verf** and are also available in [Github](https://github.com/NGWPC/nwm-verf/tree/development/data)
+Sample input data can be downloaded from **s3://ngwpc-dev/regionalization/data/inputs/eval** 
 
 #### 3) Activate venv for nwm.verf
 ```bash

sample_files/configs/config_eval.yaml → configs/config_eval.yaml

@@ -7,25 +7,27 @@ general:
     compute_metrics: true
     plot_metrics: true
   domain: conus  # Define NWM domain. Supported options: conus, ak, hi, prvi
-  location_set_name: vpu_09  # user-specified name for the set of locations
+  location_set_name: vpu_03S  # user-specified name for the set of locations
   location_type: usgs_gage # location type
   variable_name: streamflow # currently only 'streamflow' is supported
   nwm_configuration: ngen_simulation    
-  dataset_name: [hdbscan] # user-specified name of datasets (e.g., formualtion or algorithm name)
-  nwm_version: [ngen, ngen] # list of NWM versions; must have the same length as 'dataset_name'
-  forecast_start_date: ['2022-10-01 00:00:00', '2022-10-01 00:00:00'] # list of start dates for simulation; must be same length as 'dataset_name'
-  forecast_end_date: ['2022-10-01 10:00:00', '2022-10-01 10:00:00'] # list of end dates for simulation; must be same length as 'dataset_name'
-  eval_start_date: ['2022-10-01 03:00:00', '2022-10-01 03:00:00'] # start date for evaluation period
-  eval_end_date: ['2022-10-01 10:00:00', '2022-10-01 10:00:00'] # end date for evaluation period
+  dataset_name: [test_kmeans] # user-specified name of datasets (e.g., formualtion or algorithm name)
+  nwm_version: [ngen] # list of NWM versions; must have the same length as 'dataset_name'
+  forecast_start_date: ['2022-10-01 00:00:00'] # list of start dates for simulation; must be same length as 'dataset_name'
+  forecast_end_date: ['2022-10-01 10:00:00'] # list of end dates for simulation; must be same length as 'dataset_name'
+  eval_start_date: ['2022-10-01 03:00:00'] # start date for evaluation period
+  eval_end_date: ['2022-10-01 10:00:00'] # end date for evaluation period
+  separate_calibrated: True # whether to distinguish calibrated and regionalized locations in the evaluation
 
 #### configurations for various file paths
 file_paths:
-  base_dir: ngen-app/nwm-region-mgr/data/nwm-verf/data # root directory to store data/outputs for verification
-  location_list_file: '{base_dir}/inputs/regionalization/gage_list_conus_{location_set_name}.csv' # text file containing list of usgs gage IDs (one gage ID per line)
-  gage_hydrofabric_file: '{base_dir}/inputs/gage_files/gage_hydrofabric_all_domains.parquet' # gage hydrofabric file (including geometry)
-  crosswalk_file: '{base_dir}/inputs/regionalization/usgs_{nwm_version}_crosswalk_{domain}.parquet' # crosswalk file mapping gage IDs to NWM/ngen link IDs
-  fcst_data_file: 'ngen-app/nwm-region-mgr/data/regionalization/{dataset_name}/{location_set_name}/Output/troute_output_{forecast_start_date}.nc' # forecast data directory (must be specified when nwm_forecast.data_source is set to ngenCERF)
-  output_dir: '{base_dir}/outputs/{location_set_name}' # output directory to store all output datasets, metrics and plots
+  #base_dir: data # when running from local
+  base_dir: /ngen-app/nwm-region-mgr/data/ # root directory to store inputs/outputs for verification
+  location_list_file: '{base_dir}/inputs/eval/gage_list_conus_{location_set_name}.csv' # text file containing list of usgs gage IDs (one gage ID per line)
+  crosswalk_file: '{base_dir}/inputs/eval/usgs_{nwm_version}_crosswalk_{domain}.parquet' # crosswalk file mapping gage IDs to NWM/ngen link IDs
+  fcst_data_file: '{base_dir}/outputs/ngen/regionalization/{dataset_name}/{location_set_name}/Output/troute_output_{forecast_start_date}.nc' # forecast data directory (must be specified when nwm_forecast.data_source is set to ngenCERF)
+  calib_param_file: '{base_dir}/inputs/region/pseudo_calib_params/sampled_params_{domain}.csv' # calibration parameters file (used when separate_calibrated is True); must have column 'gage_id'
+  output_dir: '{base_dir}/outputs/eval/{location_set_name}' # output directory to store all output datasets, metrics and plots
 
 nwm_forecast:
   data_source: ngenSIM   # Specifies the remote location from which to fetch the data ("GCS", "NOMADS", "DSTOR", "ngenCERF")

sample_files/configs/config_formreg.yaml → configs/config_formreg.yaml

@@ -2,8 +2,9 @@
 
 # General settings for the formulation regionalization application
 general:
-  huc12_hydrofabric_file: '{base_dir}/inputs/NHDPlusV21/NHDPlusNationalData/NationalWBDSnapshot.gdb'
-  divide_huc12_cwt_file: '{base_dir}/inputs/cwt_divide_huc12/cwt_divide_huc12_{domain}.csv' # Path to crosswalk file for HUC12 and NextGen catchments.
+  huc12_hydrofabric_file: '{base_dir}/inputs/region/NHDPlusV21/NHDPlusNationalData/NationalWBDSnapshot.gdb' #non-AK domains
+  #huc12_hydrofabric_file: '{base_dir}/inputs/region/NHDPlusV21/NHD_H_Alaska_State_GPKG.gpkg'  # AK domain
+  divide_huc12_cwt_file: '{base_dir}/inputs/region/cwt_divide_huc12/cwt_divide_huc12_{domain}.csv' # Path to crosswalk file for HUC12 and NextGen catchments.
 
   calib_basins_only: False  # whether to run formulation selection only for calibrated basins (based on summary score)
   formulation_to_include: # List of competing formulations required to be considered in the selection process
@@ -63,7 +64,7 @@ summary_score:
 formulation_cost:
 
   #score_tolerance: 0.05 # Tolerance in percentage for summary score. 
-  file:  '{base_dir}/inputs/formulation_costs_secs_per_catchment.csv' # Path to the file containing the computational cost of each formulation
+  file:  '{base_dir}/inputs/region/formulation_costs_secs_per_catchment.csv' # Path to the file containing the computational cost of each formulation
   costs: # List of formulations and their corresponding computational costs
     "noah-owp-modular ueb cfe-x t-route": 10
     # "noah-owp-modular sac-sma t-route": 20
@@ -76,7 +77,7 @@ output:
 
   formulation: # Output configurations for the selected formulations
     save: True
-    path: '{base_dir}/outputs/{run_name}/formulations'
+    path: '{base_dir}/outputs/region/{run_name}/formulations'
     stem: 'form_{domain}_vpu{vpu_list}' 
     stem_suffix: '_pars'  # suffix for the formulation file with parameters
     format: parquet
@@ -86,11 +87,11 @@ output:
 
   config_final: # Final configuration file after processing and validation
     save: True
-    path: '{base_dir}/outputs/{run_name}/config_formreg_final.yaml' 
+    path: '{base_dir}/outputs/region/{run_name}/config_formreg_final.yaml' 
 
   summary_score: # Output configurations for the summary score
     save: True
-    path: '{base_dir}/outputs/{run_name}/summary_score'
+    path: '{base_dir}/outputs/region/{run_name}/summary_score'
     stem: 'score_{domain}_vpu{vpu_list}'
     stem_suffix: '_all_gages'  # suffix for the summary score file containing all gages in the domain
     format: parquet