NGWPC NWM PI-3 Formulation Selection Delivery 2026-02-25

[cmaynard-ngwpc]

This pull request delivers significant robustness, correctness, and scalability improvements to ngen-forcing, with a strong focus on MPI-safe error handling, retry logic, forcing data reliability, and containerized execution.

It expands forcing support across multiple domains (including oCONUS), improves regridding and weight-file workflows, strengthens I/O and race-condition handling, and integrates the EWTS logging framework with MPI-aware diagnostics.

The PR also modernizes path handling, improves CI/CD coverage, refactors large execution paths for clarity and timing analysis, and resolves several long-standing edge cases in GFS, AORC, NBM, and NWM forcing pipelines.

---

Additions

Forcing & Domain Support

• Added NBM ANA supplemental precipitation support for Puerto Rico.
• Added historical forcing data loaders.
• Added SFINCS and SCHISM domain handling, including:
  • Correct file expectations
  • Hawaii domain updates
  • Fixes for SCHISM multi-day simulations
• Added oCONUS parameterization for NWM domains.
• Added ZARR as a supported file type.
• Added Model_tpxo10_atlas support.
• Added automatic fallback logic for NWM data downloads using NCEI and NODD AWS sources.

Reliability, Error Handling & MPI

• Introduced a fully MPI-aware retry decorator:
  • Supports abort vs retry semantics
  • Avoids MPI deadlocks
  • Handles partial-rank failures safely
• Expanded FileNotFoundError handling across all temporary-file cleanup paths.
• Added full MPI barriers and reduced error aggregation logic.
• Added program-status checks at key execution points.
• Added hash- and random-ID-based uniqueness for output and weight files.
• Added cache management and garbage collection improvements.

Logging, Debugging & Observability

• Integrated the EWTS package (including new performance log level).
• Added timing blocks and refactored execution into discrete timed phases.
• Added support for:
  • MPI debug environment variables
  • Debug-wait environment flags
• Improved log formatting, verbosity control, and message consistency.

---

Changes

Data Processing & Regridding

• Refactored regridding workflows:

  • Retry-decorated ESMF calls
  • Safer execution ordering
  • Isolated regrid object execution

• Refactored weight-file lifecycle:

  • Unique naming per realization
  • Safer temp-file writes
  • Improved loading, writing, and caching logic

• Improved AORC, GFS, and NBM processing efficiency.

• Extended time slices to handle forecast boundary edge cases.

Containerization & Path Handling

• Converted all critical paths to absolute path handling for container safety.

• Fixed configuration, domain, and model file resolution inside containers.

• Updated Dockerfiles:

  • Added missing dependencies (e.g., wget, netcdf4)
  • Added domain-specific images (SFINCS, Delft dashboard)

• Removed binary and data files from container images.

Performance & Scalability

• Improved MPI data loading strategy.

• Reduced redundant cache file opens and I/O operations.

• Improved handling of:

  • NFS cleanup
  • EBUSY errors
  • Race conditions during downloads and writes

• Kept full simulation time ranges in memory where appropriate to reduce reloads.

Code Quality & Refactoring

• Large-scale refactors for:

  • Retry utilities
  • Error handlers
  • Regridding and execution orchestration

• Improved type hints, formatting, naming consistency, and docstrings.

• Split large methods into clearer properties and helpers.

• Removed deprecated, unused, or commented-out logic.

CI/CD & Tooling

• Updated CI/CD workflows, including ngwpc-candidate and ngwpc-release branches.
• Synced scripts with latest data-processing expectations.
• Improved unit-test and package installation behavior.

---

Removals

• Removed deprecated diagnostics and stray debug code.
• Removed lumped forcing from the build.
• Removed obsolete abort paths in favor of MPI-safe error handling.
• Removed assumptions about script-relative paths.
• Removed unused properties, imports, and legacy warnings.