You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CREDITS.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
The following software libraries and other references are utilized in the creation this repository.
2
2
3
-
[hfsubset](https://github.com/LynkerIntel/hfsubset/tree/main) - distributed by LynkerIntel under [GPL-3.0](https://github.com/CIROH-UA/ngen-datastream/blob/main/LICENSE.md)
3
+
[hfsubset](https://github.com/LynkerIntel/hfsubset/tree/main) - distributed by LynkerIntel under [GPL-3.0](https://github.com/CIROH-UA/forcingprocessor/blob/main/LICENSE.md)
4
4
5
5
[Lynker Spatial](https://www.lynker-spatial.com/) - distributed under the Open Database License, [copyright](https://lynker-spatial.s3.amazonaws.com/copyright.html)
Copy file name to clipboardExpand all lines: README.md
+15-12Lines changed: 15 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,18 +1,21 @@
1
1
# Forcing Processor
2
-
Forcingprocessor converts National Water Model (NWM) forcing data into Next Generation National Water Model (NextGen) forcing data. The motivation for this tool is NWM data is gridded and stored within netCDFs for each forecast hour. Ngen inputs this same forcing data, but in the format of per-catchment csv files that hold time series data. Forcingprocessor is driven by a configuration file that is explained, with an example, in detail below. The config argument accepts an s3 URL.
2
+
Forcingprocessor converts National Water Model (NWM) forcing data into Next Generation National Water Model (NextGen) forcing data. This tool provides the forcing pre-processing for the [NextGen Research DataStream](https://github.com/CIROH-UA/ngen-datastream).
The motivation for this tool is NWM data is gridded and stored within netCDFs for each forecast hour. Ngen inputs this same forcing data, but in the format of catchment averaged data time series data.
Prior to executing the processor, the user will need to obtain a geopackage file to define the spatial domain. [hfsubset](https://github.com/lynker-spatial/hfsubsetCLI) will provide a geopackage which contains a necessary layer, `forcing-weights`, for `processor.py`. The user will define the time domain by generating the forcing filenames for `processor.py` via `nwm_filenames_generator.py`, which is explained [here](#nwm_file).
18
+
Prior to executing the processor, the user will need to obtain a geopackage file to define the spatial domain. The user will define the time domain by generating the forcing filenames for `processor.py` via `nwm_filenames_generator.py`, which is explained [here](#nwm_file). Note that `forcingprocessor` will calcuate weights if not found within the geopackage file.
16
19
17
20
## Example `conf.json`
18
21
```
@@ -50,14 +53,14 @@ Prior to executing the processor, the user will need to obtain a geopackage file
| nwm_file | Path to a text file containing nwm file names. One filename per line. [Tool](#nwm_file) to create this file |:white_check_mark:|
53
-
| gpkg_file | Geopackage file to define spatial domain. Use [hfsubset](https://github.com/lynker-spatial/hfsubsetCLI) to generate a geopackage with a `forcing-weights` layer. Accepts local absolute path, s3 URI or URL. Also acceptable is a weights parquet generated with [weights_hf2ds.py](https://github.com/CIROH-UA/ngen-datastream/blob/main/forcingprocessor/src/forcingprocessor/weights_hf2ds.py), though the plotting option will no longer be available. |:white_check_mark:|
56
+
| gpkg_file | Geopackage file to define spatial domain. Use [hfsubset](https://github.com/lynker-spatial/hfsubsetCLI) to generate a geopackage with a `forcing-weights` layer. Accepts local absolute path, s3 URI or URL. Also acceptable is a weights parquet generated with [weights_hf2ds.py](https://github.com/CIROH-UA/forcingprocessor/blob/main/src/forcingprocessor/weights_hf2ds.py), though the plotting option will no longer be available. |:white_check_mark:|
| storage_type | Type of storage (local or s3) |:white_check_mark:|
60
-
| output_path | Path to write data to. Accepts local path or s3 |:white_check_mark:|
62
+
| storage_type | Type of storage (local or s3 URI) |:white_check_mark:|
63
+
| output_path | Path to write data to. Accepts local path or s3 URI |:white_check_mark:|
61
64
| output_file_type | List of output file types, e.g. ["tar","parquet","csv","netcdf"]|:white_check_mark:|
62
65
63
66
### 3. Run
@@ -88,7 +91,7 @@ ngen_variables = [
88
91
```
89
92
90
93
## nwm_file
91
-
A text file given to forcingprocessor that contains each nwm forcing file name. These can be URLs or local paths. This file can be generated with the [nwmurl tool](https://github.com/CIROH-UA/nwmurl) and a [generator script](https://github.com/CIROH-UA/ngen-datastream/blob/main/forcingprocessor/src/forcingprocessor/nwm_filenames_generator.py) has been provided within this repo. The config argument accepts an s3 URL.
94
+
A text file given to forcingprocessor that contains each nwm forcing file name. These can be URLs or local paths. This file can be generated with the [nwmurl tool](https://github.com/CIROH-UA/nwmurl) and a [generator script](https://github.com/CIROH-UA/forcingprocessor/blob/main/src/forcingprocessor/nwm_filenames_generator.py) has been provided within this repo. The config argument accepts an s3 URL.
@@ -109,13 +112,13 @@ A text file given to forcingprocessor that contains each nwm forcing file name.
109
112
```
110
113
111
114
## Weights
112
-
To calculate NextGen forcings, "weights" must be calculated to extract polygon averaged data from gridded data. The weights are made up of two parts, the `cell_id` and `coverage`. These are calculated via [exactextract](https://github.com/isciences/exactextract) within [weights_hf2ds.py](https://github.com/CIROH-UA/ngen-datastream/blob/main/forcingprocessor/src/forcingprocessor/weights_hf2ds.py), which is optionally called from forcingprocessor.
115
+
To calculate NextGen forcings, "weights" must be calculated to extract polygon averaged data from gridded data. The weights are made up of two parts, the `cell_id` and `coverage`. These are calculated via [exactextract](https://github.com/isciences/exactextract) within [weights_hf2ds.py](https://github.com/CIROH-UA/forcingprocessor/blob/main/src/forcingprocessor/weights_hf2ds.py), which is optionally called from forcingprocessor.
113
116
114
-
If a geopackage is supplied to forcingprocessor, it will be searched for the layer `forcings-weights`. If this layer is found, these weights are used during processing. If not, forcingprocessor will call [weights_hf2ds.py](https://github.com/CIROH-UA/ngen-datastream/blob/main/forcingprocessor/src/forcingprocessor/weights_hf2ds.py) to calculate the weights (cell_id and coverage) for every divide-id in the geopackage. This can take time, so forcingprocessor will write a parquet of weights out in the metadata, that can be reused in future forcingprocessor executions.
117
+
If a geopackage is supplied to forcingprocessor, it will be searched for the layer `forcings-weights`. If this layer is found, these weights are used during processing. If not, forcingprocessor will call [weights_hf2ds.py](https://github.com/CIROH-UA/forcingprocessor/blob/main/src/forcingprocessor/weights_hf2ds.py) to calculate the weights (cell_id and coverage) for every divide-id in the geopackage. This can take time, so forcingprocessor will write a parquet of weights out in the metadata, that can be reused in future forcingprocessor executions.
0 commit comments