zarr in forcingprocessor #62
JordanLaserGit
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Forcingprocessor was originally designed to serve as the forcings generation for the NRDS system. This motivated the design choice to optimize for large domain (>10,000 catchments), short time (<1000 time steps) processing of NWM forcing files. Forcingprocessor's internal algorithm performs a multi processed read and processing of each netcdf required. This is performant at scale, but would be considered a network heavy solution in the context of small domain (<100 catchments) simulations. Also, forcingprocessor is not currently capable of processing anything other than the NWM forcing files provided by nwmurl.
From here, we can implement algorithms (that are already in related tooling) that will generalize forcingprocessor to use the most efficient internal algorithm based on the domain size and time step count. For example, if forcingprocessor starts up with a large geopackage and a few number of nwm files listed in filenamelist.txt, then the original algorithm is used. However if a small geopackage and a large number of nwm files, forcingprocessor could then switch to a zarr based algorithm that is more suitable in that case.
Action items:
If these two are implemented, forcingprocessor could serve as a general purpose forcings tool that can be integrated in any workflow requiring forcings generation.
Beta Was this translation helpful? Give feedback.
All reactions