Xborder (#185)

* fix #182

* add xborder polys table, load from s3

* remove xborder warning from upstream lookup readme, update cw script and fix issue in measured cw query

* update discharge processing to use shell script (instead of make) and remove xborder caveat

* upgrade gdal, downgrade awscli, document climr download for transborder precip

* document xborder source and load, fix readme path to db create, start precip update

* tidy

* add jq to docker image, do not load nhd or hydrosheds, get precip for wsds not covered by climatebc grid from climr grid

* update fwa_watershedatmeasure to use new xborder data, update docs

* add xborder load query

Author: smnorris

Dockerfile

@@ -1,4 +1,4 @@
-FROM ghcr.io/osgeo/gdal:ubuntu-full-3.10.3
+FROM ghcr.io/osgeo/gdal:ubuntu-full-3.12.1
 
 RUN apt-get update && apt-get --assume-yes upgrade \
     && apt-get -qq install -y --no-install-recommends postgresql-common \
@@ -16,9 +16,10 @@ RUN apt-get update && apt-get --assume-yes upgrade \
     && apt-get -qq install -y --no-install-recommends python3-dev \
     && apt-get -qq install -y --no-install-recommends python3-venv \
     && apt-get -qq install -y --no-install-recommends python3-psycopg2 \
+    && apt-get -qq install -y --no-install-recommends jq \
     && rm -rf /var/lib/apt/lists/*
 
-RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
+RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.22.21.zip" -o "awscliv2.zip" \
     && unzip awscliv2.zip \
     && ./aws/install
 

README.md

@@ -17,12 +17,14 @@ See [documentation](https://smnorris.github.io/fwapg/) for setup and usage detai
 
 ## Quickstart
 
-1. Ensure all requirements/dependencies are met/installed:
+1. Ensure all requirements/dependencies are met/installed. 
     - access to a PostgreSQL (>13) database with the PostGIS extension
     - GDAL >=3.6
     - Python 3
     - [`bcdata`](https://github.com/smnorris/bcdata)
-    - [`make`](https://www.gnu.org/software/make/)/`unzip`/`curl`/`awscli`/ etc (see Dockerfile)
+
+See the Dockerfile for the full list of dependencies.
+    
 
 2. Ensure you have a `DATABASE_URL` environment variable set to point to your database, for example:
 
@@ -32,7 +34,7 @@ See [documentation](https://smnorris.github.io/fwapg/) for setup and usage detai
 
         git clone https://github.com/smnorris/fwapg.git
         cd fwapg
-        cd data && ./create.sh && cd ..  # load extensions, create schemas/tables
+        cd db && ./create.sh && cd ..    # load extensions, create schemas/tables
         ./load.sh                        # load the data
 
 The full load takes some time - but once complete, you can run `fwapg` enabled queries with your favorite sql client. For example:
@@ -67,7 +69,7 @@ The full load takes some time - but once complete, you can run `fwapg` enabled q
         LINESTRINGZM((...
 
 
-See [Usage](https://smnorris.github.io/fwapg/02_usage.html) for more examples.
+See [Usage](https://smnorris.github.io/fwapg/01_usage.html) for more examples.
 
 
 ## Tile and feature services
@@ -84,10 +86,7 @@ See [Usage](https://smnorris.github.io/fwapg/02_usage.html) for more examples.
 
 - BC Freshwater Atlas [documentation](https://www2.gov.bc.ca/gov/content/data/geographic-data-services/topographic-data/freshwater) and [license](https://www2.gov.bc.ca/gov/content/data/open-data/open-government-licence-bc)
 
-- USGS Watershed Boundary Dataset (WBD) [Metadata](https://prd-tnm.s3.amazonaws.com/StagedProducts/Hydrography/WBD/National/GDB/WBD_National_GDB.xml)
-
-- Hydrosheds [License and citation](https://www.hydrosheds.org/page/license)
-
+- BC stream contributing areas (watershed boundaries) outside of BC were provided by Dan Weller, DFO 
 
 ## Development and testing
 
@@ -101,7 +100,7 @@ All results should be true.
 ## Documentation
 
 Documentation is built from the markdown files in `/docs`.
-Info in the table reference page (`03_tables.md`) can be autogenerated from comments in the database. To dump the text to stdout:
+Info in the table reference page (`02_tables_views.md`) can be autogenerated from comments in the database. To dump the text to stdout:
 ```
 cd docs
 ./table_reference.sh 

db/sql/FWA_WatershedAtMeasure.sql

@@ -283,22 +283,6 @@ begin
         FROM FWA_SliceWatershedAtPoint(v_blkey, v_measure) slice
         ),
 
-        -- find any upstream contributing area outside of BC (but not including Alaska panhandle)
-        exbc AS
-         (
-          SELECT hydrosheds.hydroshed(h.hybas_id) AS geom
-          FROM ref_point s
-          INNER JOIN hydrosheds.hybas_lev12_v1c h
-          ON ST_Intersects(h.geom, s.geom_pt)
-          WHERE FWA_UpstreamBorderCrossings(s.blue_line_key, s.measure_pt) IN ('AB_120','YTNWT_60')
-          UNION ALL
-          SELECT FWA_Huc12(h.huc12) AS geom
-          FROM ref_point s
-          INNER JOIN usgs.wbdhu12 h
-          ON ST_intersects(h.geom, s.geom_pt)
-          WHERE FWA_UpstreamBorderCrossings(s.blue_line_key, s.measure_pt) = 'USA_49'
-        ),
-
         -- aggregate the result and dump to singlepart
         agg as
         (

db/sql/schema.sql

@@ -504,6 +504,34 @@ create index fwa_watersheds_geom_idx on whse_basemapping.fwa_watersheds_poly usi
 
 -- ---------------------------------------------------------------------------------------------------------------------
 
+-- cross border watershed polygons, provided by DFO
+create table whse_basemapping.fwa_watersheds_xborder_poly (
+  watershed_feature_id       integer primary key     ,
+  watershed_group_id         integer not null        ,
+  watershed_key              integer not null        ,
+  fwa_watershed_code         character varying(143) not null,
+  local_watershed_code       character varying(143) not null,
+  watershed_group_code       character varying(4) not null,
+  watershed_order            integer                 ,
+  watershed_magnitude        integer                 ,
+  local_watershed_order      integer                 ,
+  local_watershed_magnitude  integer                 ,
+  feature_code               character varying(10)   ,
+  wscode_ltree ltree generated always as (replace(replace(fwa_watershed_code, '-000000', ''), '-', '.')::ltree) stored,
+  localcode_ltree ltree generated always as (replace(replace(local_watershed_code, '-000000', ''), '-', '.')::ltree) stored,
+  geom                       public.geometry(multipolygon,3005)
+ );
+create index fwa_wsd_xborder_watershed_key_idx on whse_basemapping.fwa_watersheds_xborder_poly (watershed_key);
+create index fwa_wsd_xborder_watershed_group_code_idx on whse_basemapping.fwa_watersheds_xborder_poly (watershed_group_code);
+create index fwa_wsd_xborder_watershed_group_id_idx on whse_basemapping.fwa_watersheds_xborder_poly (watershed_group_id);
+create index fwa_wsd_xborder_wsc_gist_idx on whse_basemapping.fwa_watersheds_xborder_poly using gist (wscode_ltree);
+create index fwa_wsd_xborder_wsc_btree_idx on whse_basemapping.fwa_watersheds_xborder_poly using btree (wscode_ltree);
+create index fwa_wsd_xborder_lc_gist_idx on whse_basemapping.fwa_watersheds_xborder_poly using gist (localcode_ltree);
+create index fwa_wsd_xborder_lc_btree_idx on whse_basemapping.fwa_watersheds_xborder_poly using btree (localcode_ltree);
+create index fwa_wsd_xborder_geom_idx on whse_basemapping.fwa_watersheds_xborder_poly using gist (geom);
+
+-- ---------------------------------------------------------------------------------------------------------------------
+
 create table whse_basemapping.fwa_wetlands_poly (
     waterbody_poly_id integer primary key,
     watershed_group_id integer,

docker-compose.yml

@@ -29,30 +29,6 @@ services:
       - PGUSER=postgres
       - PGPASSWORD=postgres
       - DATABASE_URL=postgresql://postgres:postgres@db:5432/fwapg
-    depends_on:
-      db:
-        condition: service_healthy
-  featureserv:
-    image: pramsey/pg_featureserv:latest
-    container_name: fwapg-fs
-    ports:
-      - ${FS_PORT}:9000
-    volumes:
-      - "./:/home/fwapg"
-    environment:
-      - DATABASE_URL=postgresql://postgres:postgres@db:5432/fwapg
-    depends_on:
-      db:
-        condition: service_healthy
-  tileserv:
-    image: pramsey/pg_tileserv:latest
-    container_name: fwapg-ts
-    ports:
-      - ${TS_PORT}:7800
-    volumes:
-      - "./:/home/fwapg"
-    environment:
-      - DATABASE_URL=postgresql://postgres:postgres@db:5432/fwapg
     depends_on:
       db:
         condition: service_healthy

docs/01_setup.md

@@ -2,6 +2,9 @@
 
 This document presumes a working familiarity with FWA data and spatial SQL queries with PostgreSQL/PostGIS.
 
+## Installation / setup
+
+See the README 
 
 ## Upstream / downstream analysis
 

docs/02_usage.md docs/01_usage.mddocs/02_usage.md renamed to docs/01_usage.md

@@ -910,10 +910,7 @@ watershed in which it falls, then that fundamental watershed is not included in
 4. If the point is < 50 m upstream from the bottom of the fundamental
 watershed in which it falls, then that watershed is included in the aggregation
 
-For cross-boundary watersheds, the function returns non-BC areas using these data sources:
-
-- USGS [huc12 watersheds](https://www.usgs.gov/core-science-systems/ngp/national-hydrography/watershed-boundary-dataset?qt-science_support_page_related_con=4#qt-science_support_page_related_con) for USA (WA, ID, MT only)
-- [hydrosheds](https://www.hydrosheds.org) watersheds for other neighbouring jurisdictions
+For trans-boundary watersheds, the results include non-BC areas (not present in the FWA) thanks to data [provided by DFO](https://github.com/smnorris/fwapg/tree/main/extras/xborder).
 
 The table returned includes these columns:
 
@@ -981,7 +978,7 @@ The `refine_method` field in the output table has several possible values:
      100.064535.057628 | 100.064535.057628.634957 | 33478.28 | LAKE           |
     ```
 
-    Mapped, the geometry looks like this - FWA heights of land in BC, HUC12 heights of land in the USA:
+    Mapped, the geometry looks like this - the USA portion of the watershed is included:
 
     ![watershed](images/watershed6.png)
 

docs/03_tables_views.md docs/02_tables_views.mddocs/03_tables_views.md renamed to docs/02_tables_views.md

@@ -23,7 +23,6 @@ fwapg
    :maxdepth: 2
    :caption: Contents:
 
-   01_setup
-   02_usage
-   03_tables_views
-   04_functions
+   01_usage
+   02_tables_views
+   03_functions