NGWPC NWM PI-3 Formulation Selection Delivery 2025-12-18

.github/workflows/documentation.yml

@@ -0,0 +1,34 @@
+name: documentation
+
+on:
+  push:
+    branches: [ "documentation" ]
+
+permissions:
+  contents: write
+
+jobs:
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: |
+          pip install -r docs/source/requirements.txt
+          pip install .
+      - name: Debug Python imports
+        run: |
+          python -c "import nwm_region_mgr.parreg.config_schema; print('Import OK')"
+      - name: Sphinx build
+        run: |
+          sphinx-build docs/source docs/build
+      - name: Deploy to GitHub Pages
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_branch: gh-pages
+          publish_dir: docs/build/
+          force_orphan: true

.gitignore

@@ -25,7 +25,11 @@ __pycache__/
 # Editor configs
 .vscode/
 
+# Documentation
+docs/build/*
+docs/source/API/*
+!docs/source/API/index.rst
 # Specific project paths
 pkgs/parreg/build
 pkgs/parreg/tests/test_data/test_outputs
-nwm-rte
+nwm-rte

LICENSE

@@ -0,0 +1,14 @@
+Copyright 2025 Raytheon Company
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation
+and/or other materials provided with the distribution.
+
+Licensed under: https://opensource.org/license/bsd-2-clause
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+All rights reserved. Based on Government sponsored work under contract GS-35F-204GA.
+
+-----------------

README.md

@@ -1,36 +1,45 @@
-# nwm-region-mgr
+# Formulation and Parameter Regionalization for the NextGen Framework
+[![Build](https://img.shields.io/github/actions/workflow/status/ngwpc/nwm-region-mgr/ci.yaml?branch=main)](.github/workflows/ci.yml)
+[![License: BSD 2-Clause](https://img.shields.io/badge/License-BSD%202--Clause-orange.svg)](https://opensource.org/license/bsd-2-clause)
+[![Release](https://img.shields.io/github/v/release/ngwpc/nwm-region-mgr)](https://github.com/NGWPC/nwm-region-mgr)
+![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-orange.svg)
+![Linter: Ruff](https://img.shields.io/badge/linter-ruff-orange)
 
-## Name
-NGEN/NWM Regionalization
+<img src="docs/source/_images/overview.png" alt="overview" width="600"/>
 
-## Description
-This repository includes packages for conducting formulation and parameter regionalizations for NextGen modules,
-as well as scripts/workflows to run ngen simulations and evaluation.
-- parreg: parameter regionalization
-- formreg: formulation regionalization
-- utils: utility functions shared by parreg and formreg
 
+`nwm_region_mgr` is a Python package for identifying optimal model formulations and parameter values in ungauged catchments. It leverages calibration data from gauged catchments to improve hydrologic modeling and forecasting skill across regions, playing a key role in the NextGen and NWM ecosystem.
+
+
+## Key Features
+
+- **Formulation Regionalization** – Ranks NextGen model formulations for ungauged catchments based on similarity to gauged sites.
+- **Parameter Regionalization** – Estimates parameter values by intelligently transferring calibrations across catchments.
+- **Clustering Methods** – Groups catchments with shared hydrologic characteristics using multiple clustering approaches.
+- **Diagnostic Plots** – Generates clear plots and maps that explain formulation and parameter choices.
+- **Scalable Workflows** – Efficiently supports studies from individual watersheds to CONUS-wide applications.
+- **Customizable Configurations** – Full control of workflows via human-readable config files.
 ## Docker Run Time Environment (RTE)
-### Step 0. Build Docker image
+### Step 0. Build Docker image and download data
+#### a) Sample input data can be downloaded from: **s3://ngwpc-dev/regionalization/data**
+
+#### b) Build Docker image
 `Note` This step is only necessary if a docker image doesn't already exists or if updates to the code base have been implemented. The short flag `-d` can also be used in place of `--docker`
 
 ```bash
 cd nwm-region-mgr
 ```
 ```bash
-./region-mgr.sh -docker
+./region-mgr.sh --docker
 ```
 ### Step 1. Run regionalization
-#### a) Run both formulation and parameter regionalization together:
-```bash
-./region-mgr.sh --formreg --parreg
-```
-#### b) Run formulation regionalization alone:
+
+#### a) Run formulation regionalization alone (no parreg):
 The short flag `-f` can also be used in place of `--formreg`.
 ```bash
 ./region-mgr.sh --formreg
 ```
-#### c) Run parameter regionalization alone:
+#### b) Run parameter regionalization (formreg is also ran as a prerequisite):
 The short flag `-p` can also be used in place of `--parreg`.
 ```bash
 ./region-mgr.sh --parreg
@@ -52,39 +61,41 @@ The short flag `-e` can also be used in place of `--eval`.
 ### Steps 0-4
 Alternatively the user can run steps 0-4 in all at once in series:
 ```bash
-./region-mgr.sh -dfpne
+./region-mgr.sh -dpne
 ```
-## Desktop Run Time Environment (RTE)
+## Desktop/Workspace Run Time Environment (RTE)
 ### Clone & Build
 
 #### 1. clone ngen-region-mgr from Github
 
 ```bash
-cd [NGEN_REG_ROOT]
-git clone -b development --recurse-submodules https://github.com/NGWPC/nwm-region-mgr.git
+git clone https://github.com/NGWPC/nwm-region-mgr.git
+cd nwm-region-mgr
 ```
 
 #### 2. create python venv
-
+Create a virtual environment to isolate the dependencies of this library from your base Python environment.
 ```bash
-cd [VENV_ROOT]
-/usr/bin/python3.11 -m venv venv
-source venv/bin/activate
-pip install --upgrade pip
+python3.11 -m venv .venv
+source .venv/bin/activate
+pip pip install --upgrade pip
 ```
+
 #### 3. install nwm-region-mgr
 
+You will then be able to install nwm_region_mgr. There are a few variants that users may be interested in.
+
 ```bash
-cd [NGEN_REG_ROOT]
-pip install . # to install formreg and utils only
-# OR
-pip install .[parreg] # to install formreg, utils, and parreg 
+# Regular package install
+pip install .
+# Install the package in edit mode (for development)
+pip install -e .
+# Install additional dependencies for parameter regionalization
+pip install .[parreg]
+# Install dependencies for docs and dev
+pip install .[docs,dev]
 ```
 
-where [VENV_ROOT] and [NGEN_REG_ROOT] refer to the directory to install python venv and nwm-region-mgr 
-in your local workspace, respectively
-
-
 ### STEP 1: Run regionalization to produce regionalized parameters and formulations
 
 #### 1) Set up configuration yaml files
@@ -94,17 +105,18 @@ Three yaml config files are needed to run regionalization
 - **onfig_formreg.yaml**: specific settings for the formulation regionalization process.
 - **config_parreg.yaml**: specific settings for the parameter regionalization process.
 
-Follow the sample [config files](https://github.com/NGWPC/nwm-region-mgr/tree/development/sample_files/configs) to set up the configurations for your regionalization application as needed.
+Follow the sample config files (nwm_region_mgr/sample_files/configs) to set up the configurations
+for your regionalization application as needed.
 
-Sample input data can be downloaded from **s3://ngwpc-dev/Yuqiong.Liu/repos/nwm-region-mgr**
+Sample input data can be downloaded from **s3://ngwpc-dev/regionalization/**
 
 
 #### 2) Run the regionalization script
 
 ```bash
 python [NGEN_REG_ROOT]/nwm-region-mgr/regionalization.py [COFIG_DIR]
 ```
-Where: 
+Where:
 - [NGEN_REG_ROOT] refers to the directory where nwm-region-mgr is installed
 - [COFIG_DIR] refers to the directory containing the three config files as noted in 1), e.g.,
 
@@ -114,35 +126,35 @@ python regionalization.py sample_files/configs
 
 ### STEP 2: Run NGEN simulation with regionalized parameters
 
-- #### Run natively in workspace
-    - ##### 1) Install [ngen](https://github.com/NGWPC/ngen) and all submodules in its own venv
-        You may want to follow the following Confluence pages:
-        - [Clone ngen](https://confluence.nextgenwaterprediction.com/display/NGWPC/Clone+NGWPC+GitHub+Code)
-        - [Build ngen](https://confluence.nextgenwaterprediction.com/display/NGWPC/Build+ngen+completely)
+#### Run natively in workspace
+##### 1) Install [ngen](https://github.com/NGWPC/ngen) and all submodules in its own venv
+You may want to follow the following Confluence pages:
+- [Clone ngen](https://confluence.nextgenwaterprediction.com/display/NGWPC/Clone+NGWPC+GitHub+Code)
+- [Build ngen](https://confluence.nextgenwaterprediction.com/display/NGWPC/Build+ngen+completely)
 
-    - ##### 2) Install [mswm](https://github.com/NGWPC/nwm-msw-mgr) in its own venv
-
-    - ##### 3) Activate MSWM venv, e.g.
-        ```bash
-        source ~/repos/nwm-msw-mgr/venv/bin/activate
-        ```
-    - ##### 4) Set up MSWM configuration as shown in [run_ngen_vpu.sh](https://github.com/NGWPC/nwm-region-mgr/blob/development/run_ngen_vpu.sh)
-
-    - ##### 5) Run MSWM and ngen simulation
-        ```bash
-        cd ~/repos/nwm-region-mgr
-        ./run_ngen_vpu.sh
-        ```
-    - ##### 6) Check inputs, outputs and logs
-    All input, output and log files from running MSWM and NGEN can be found in *[work_dir]/regionalization/[run_name]/[vpu]* 
+##### 2) Install [mswm](https://github.com/NGWPC/nwm-msw-mgr) in its own venv
+
+##### 3) Activate MSWM venv, e.g.
+```bash
+source ~/repos/nwm-msw-mgr/venv/bin/activate
+```
+##### 4) Set up MSWM configuration as shown in [run_ngen_vpu.sh](https://github.com/NGWPC/nwm-region-mgr/blob/development/run_ngen_vpu.sh)
+
+##### 5) Run MSWM and ngen simulation
+```bash
+cd ~/repos/nwm-region-mgr
+./run_ngen_vpu.sh
+```
+##### 6) Check inputs, outputs and logs
+All input, output and log files from running MSWM and NGEN can be found in *[work_dir]/regionalization/[run_name]/[vpu]* 
 (as defined in **run_ngen_vpu.sh**)
 
-    - ##### 7) If ngen fails at t-route
-    Check if all NGEN cat-*.csv and nex-*.csv output files are generated; if yes,
+##### 7) If ngen fails at t-route
+Check if all NGEN cat-*.csv and nex-*.csv output files are generated; if yes,
     run t-route separately from the Output directory where ngen outputs are located, e.g.,
-        ```bash
-        python -m nwm_routing -f -V4 ../Input/vpu_09_troute_config_region.yaml
-        ```
+```bash
+python -m nwm_routing -f -V4 ../Input/vpu_09_troute_config_region.yaml
+```
 
 ### STEP 3: Evaluate NGEN simulation with nwm.verf
 

docs/makefile

@@ -0,0 +1,12 @@
+# Define variables for Sphinx
+SPHINXAPIDOC = sphinx-apidoc
+SPHINXBUILD = sphinx-build
+SOURCEDIR = source
+BUILDDIR = build
+
+clean:
+	rm -rf $(BUILDDIR)/*
+	rm -rf $(SOURCEDIR)/API/_autosummary
+
+html: clean
+	$(SPHINXBUILD) -b html $(SOURCEDIR) $(BUILDDIR)