Merge branch 'develop' into main

Author: davidfrantz

.github/workflows/docker-publish-debug.yml

@@ -2,7 +2,7 @@ name: Docker-debug
 
 on:
   push:
-    branches: [master, develop]
+    branches: [main, develop]
   release:
     types: [published]
   repository_dispatch:
@@ -29,7 +29,7 @@ jobs:
         run: |
           VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
           [[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')
-          [ "$VERSION" == "master" ] && VERSION=latest
+          [ "$VERSION" == "main" ] && VERSION=latest
           [ "$VERSION" == "develop" ] && VERSION=dev
           echo VERSION=$VERSION
           echo DOCKER=$DOCKER_USER/$IMAGE_NAME:$VERSION-debug

.github/workflows/docker-publish.yml

@@ -2,7 +2,7 @@ name: Docker
 
 on:
   push:
-    branches: [master, develop]
+    branches: [main, develop]
   release:
     types: [published]
   repository_dispatch:
@@ -29,7 +29,7 @@ jobs:
         run: |
           VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
           [[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')
-          [ "$VERSION" == "master" ] && VERSION=latest
+          [ "$VERSION" == "main" ] && VERSION=latest
           [ "$VERSION" == "develop" ] && VERSION=dev
           echo VERSION=$VERSION
           echo DOCKER=$DOCKER_USER/$IMAGE_NAME:$VERSION

ADD_INDEX.md

@@ -0,0 +1,47 @@
+# How to add a new spectral index to the TSA submodule
+
+This page documents the steps that need to be taken to implement a new spectral index in force-tsa.
+Give it a try!
+
+General remarks: please stick to the syntax and indendation etc. of the existing code.
+
+1) create a new branch from develop
+
+2) add a new enum
+
+    In src/cross-level/enum-cl.h: add a new enum before(!) _IDX_LENGTH_.
+    Use a 3-digit ID that has not been used before, e.g. _IDX_XXX_ (replace XXX).
+
+    In src/cross-level/enum-cl.c: provide a dictionary item that can translate between the Index in the parameter file and the enum, e.g. { _IDX_XXX_, "MyNewIndex" }
+
+    Make sure that the added entries in both files are in the same order.
+
+3) define band dependencies
+
+    In src/higher-level/param-hl.c, function check_bandlist():
+
+    - Add a new case clause by referencing the enum
+    - enable the verification vector v[] for required wavelengths - this is needed to pre-check whether the index can be computed with a given set of sensors
+    - choose a 3-digit ID for the filename - this should be the same ID as used for the enum
+
+4) provide citation
+
+    In src/cross-level/cite-cl.h: add a new citation enum before(!) _CITE_LENGTH_.
+    Use the same 3-digit ID as for the enum (if not already taken)
+
+    In src/cross-level/cite-cl.c: provide a dictionary item that holds the citation and the enum, as well as a bool variable that is initialized to false, e.g. { _IDX_XXX_, "Author, A. (2042). Fancy New Index. Remote Sensing of Environment", false }. Stick to the syntax of the other entries. Make sure that the added entries in both files are in the same order.
+
+4) implement index
+
+    In src/higher-level/index-hl.c, function tsa_spectral_index():
+
+    - Add a new case clause by referencing the enum
+    - cite the paper by using the cite enum
+    - use one of the provided function templates if your new index matches one of those (e.g. use index_differenced() for an NDVI-like index). If this is the case, you are done with coding and can skip the next step.
+    - provide a new function. Try to make it as generic as possible to enable other indices that use the same formulation (but just different bands or factors). Try to understand how the other functions work before doing this.
+
+5) test whether the code compiles
+
+6) make a test run (check that scaling and quantization is OK!)
+
+7) submit a pull request on Github

CONTRIBUTING.md

@@ -20,7 +20,7 @@ If asking a question, please tell us who you are, and what your background is.
 And super-important, please report back whether the answer was helpful, and make sure to mark useful responses as ``answered``.
 
 We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
-For more information, see our [code of conduct](https://github.com/davidfrantz/force/blob/master/CODE_OF_CONDUCT.md).
+For more information, see our [code of conduct](https://github.com/davidfrantz/force/blob/main/CODE_OF_CONDUCT.md).
 
 Contributors should ideally subscribe to these channels and follow them in order to keep up to date on what's happening in FORCE.
 Answering questions is an excellent and visible way to help our community, and also demonstrates your expertise.
@@ -115,5 +115,5 @@ if (a > 0){
 New files must include the appropriate license header boilerplate and the author name(s) and contact email(s).
 
 Please pay special attention to citing research and properly acknowledging code that you may adapt for inclusion in FORCE.
-See this [example](https://github.com/davidfrantz/force/blob/master/src/lower-level/equi7-ll.c).
-If applicable, make use of the [citeme](https://github.com/davidfrantz/force/blob/master/src/cross-level/cite-cl.c) functionality, e.g. ``cite_me(_CITE_EQUI7_);``
+See this [example](https://github.com/davidfrantz/force/blob/main/src/lower-level/equi7-ll.c).
+If applicable, make use of the [citeme](https://github.com/davidfrantz/force/blob/main/src/cross-level/cite-cl.c) functionality, e.g. ``cite_me(_CITE_EQUI7_);``

Makefile

@@ -108,7 +108,7 @@ TA=temp-aux
 ### TARGETS
 
 all: temp cross lower higher aux exe
-cross: string_cl enum_cl cite_cl utils_cl alloc_cl brick_cl imagefuns_cl param_cl date_cl datesys_cl lock_cl cube_cl dir_cl stats_cl pca_cl tile_cl queue_cl warp_cl sun_cl quality_cl sys_cl konami_cl download_cl read_cl
+cross: string_cl enum_cl cite_cl utils_cl alloc_cl brick_cl imagefuns_cl param_cl date_cl datesys_cl lock_cl cube_cl dir_cl stats_cl pca_cl tile_cl queue_cl warp_cl sun_cl quality_cl sys_cl konami_cl download_cl read_cl gdalopt_cl
 lower: table_ll param_ll meta_ll cube_ll equi7_ll glance7_ll atc_ll sunview_ll read_ll radtran_ll topo_ll cloud_ll gas_ll brdf_ll atmo_ll aod_ll resmerge_ll coreg_ll coregfuns_ll acix_ll modwvp_ll
 higher: param_hl progress_hl tasks_hl read-aux_hl read-ard_hl quality_hl bap_hl level3_hl cso_hl tsa_hl index_hl interpolate_hl stm_hl fold_hl standardize_hl pheno_hl polar_hl trend_hl ml_hl texture_hl lsm_hl lib_hl sample_hl imp_hl cfimp_hl l2imp_hl spec-adjust_hl pyp_hl udf_hl
 aux: param_aux param_train_aux train_aux
@@ -195,6 +195,9 @@ download_cl: temp $(DC)/download-cl.c
 read_cl: temp $(DC)/read-cl.c
 	$(GCC) $(CFLAGS) -c $(DC)/read-cl.c -o $(TC)/read_cl.o
 
+gdalopt_cl: temp $(DC)/gdalopt-cl.c
+	$(GCC) $(CFLAGS) -c $(DC)/gdalopt-cl.c -o $(TC)/gdalopt_cl.o
+
 
 ### LOWER LEVEL COMPILE UNITS
 

README.md

@@ -2,7 +2,7 @@
 
 **Framework for Operational Radiometric Correction for Environmental monitoring**
 
-**Version 3.7.7**
+**Version 3.7.8**
 
 ![FORCE Logo](/images/force.png)
 

bash/force-cube.sh

@@ -64,7 +64,7 @@ Usage: $PROG [-hvirsantobj] input-file(s)
   -v = show version
   -i = show program's purpose
   -r = resampling method
-       any GDAL resampling method for raster data, e.g. cubic (default)
+       any GDAL resampling method for raster data, e.g. near (default)
        is ignored for vector data
   -s = pixel resolution of cubed data, defaults to 10
   -a = optional attribute name for vector data. $PROG will burn these values 
@@ -148,12 +148,15 @@ function cubethis(){
       echoerr "rasterizing failed."; exit 1
     fi
 
-    MAX=$($RASTER_INFO_EXE -stats "$FOUT" | grep Maximum | head -n 1 | sed 's/[=,]/ /g' | tr -s ' ' | cut -d ' ' -f 5 | sed 's/\..*//' )
+    VALID=$($RASTER_INFO_EXE -stats "$FOUT" 2>/dev/null | grep STATISTICS_VALID_PERCENT | sed 's/ //g; s/[=,]/ /g' | cut -d ' ' -f2 | awk '{sum +=$1} END {print sum != 0}' )
     rm "$FOUT.aux.xml"
-    debug "max: $MAX"
+    debug "valid: $VALID"
 
-    if [ $MAX -lt 1 ]; then
+    if [ $VALID -eq 0 ]; then
       rm "$FOUT"
+      if [ -z "$(ls -A "$DOUT/$TILE")" ]; then
+        rmdir "$DOUT/$TILE"
+      fi
       exit 1
     fi
 
@@ -175,6 +178,19 @@ function cubethis(){
       echoerr "warping failed."; exit 1
     fi
 
+    VALID=$($RASTER_INFO_EXE -stats "$FOUT" 2>/dev/null | grep STATISTICS_VALID_PERCENT | sed 's/ //g; s/[=,]/ /g' | cut -d ' ' -f2 | awk '{sum +=$1} END {print sum != 0}' )
+    rm "$FOUT.aux.xml"
+    debug "valid: $VALID"
+
+    if [ $VALID -eq 0 ]; then
+      rm "$FOUT"
+      if [ -z "$(ls -A "$DOUT/$TILE")" ]; then
+        rmdir "$DOUT/$TILE"
+      fi
+      exit 1
+    fi
+
+
     if [ "$EXIST" == "true" ]; then
       debug "building mosaic"
       mv "$DOUT/$TILE/$COUT.tif" "$DOUT/$TILE/$COUT'_TEMP1.tif'"
@@ -200,7 +216,7 @@ ARGS=`getopt -o hvir:s:o:b:j:a:n:t:l: --long help,version,info,resample:,resolut
 if [ $? != 0 ] ; then help; fi
 eval set -- "$ARGS"
 
-RESAMPLE="cubic"
+RESAMPLE="near"
 RES=10
 DOUT=$PWD
 BASE="DEFAULT"

bash/force-level1-csd.sh

@@ -223,6 +223,9 @@ if [ $# -ne 4 ]; then
 fi
 
 which_satellite
+if [ $LANDSAT -eq 1 ]; then
+  printf "%s\n" "" "Warning: Google Cloud Storage only hosts Landsat Collection 1 data." "It is strongly recommended to use Collection 2 data: https://www.usgs.gov/landsat-missions/landsat-collection-2" "Collection 2 data is provided by the USGS, you may use https://github.com/ernstste/landsatlinks to query the catalogue and create download links."
+fi
 
 # ============================================================
 # Check user input and set up variables
@@ -350,7 +353,7 @@ if [ -z $AOITYPE ]; then
       if $(echo $ENTRY | grep -q -E "[0-2][0-9]{2}[0-2][0-9]{2}"); then
         LSPATH="${ENTRY:0:3}"
         LSROW="${ENTRY:3:6}"
-        if [ $(is_in_range $LSPATH 1 233) -eq 0 ] || [ $(is_in_range $LSPATH 1 248) -eq 0 ]; then
+        if [ $(is_in_range $LSPATH 1 233) -eq 0 ] || [ $(is_in_range $LSROW 1 248) -eq 0 ]; then
           show_help "$(printf "%s\n       " "Landsat PATH / ROW out of range. PATH not in range 1 to 233 or ROW not in range 1 to 248." "PATH / ROW received: $ENTRY")"
         fi
         continue

docs/requirements.txt

@@ -1,2 +1,3 @@
 sphinx==4.2.0
 sphinx_rtd_theme==0.5.1
+docutils==0.16

docs/source/components/auxilliary/index.rst

@@ -5,27 +5,36 @@ Auxiliary
 
 **Table 1.** Components.
 
-+--------+------------------------+-------+---------------------------------------------------------+
-| Module | Program                | Level | Short description                                       |
-+========+========================+=======+=========================================================+
-| AUX    | force                  | /     | Print version, short disclaimer, available modules etc. |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-parameter        | /     | Generation of parameter files                           |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-train            | /     | Training (and validation) of Machine Learning models    |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-qai-inflate      | /     | Inflate QAI bit layers                                  |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-tile-finder      | /     | Find the tile, pixel, and chunk of a given coordinate   |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-tabulate-grid    | /     | Extract the processing grid as shapefile                |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-cube             | /     | Ingestion of auxiliary data into datacube format        |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-pyramid          | /     | Generation of image pyramids                            |
-+        +------------------------+-------+---------------------------------------------------------+
-|        | force-mosaic           | /     | Mosaicking of image chips                               |
-+--------+------------------------+-------+---------------------------------------------------------+
++--------+---------------------+-------+---------------------------------------------------------+
+| Module | Program             | Level | Short description                                       |
++========+=====================+=======+=========================================================+
+| AUX    | force               | /     | Print version, short disclaimer, available modules etc. |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-parameter     | /     | Generation of parameter files                           |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-train         | /     | Training (and validation) of Machine Learning models    |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-qai-inflate   | /     | Inflate QAI bit layers                                  |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-tile-finder   | /     | Find the tile, pixel, and chunk of a given coordinate   |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-tabulate-grid | /     | Extract the processing grid as shapefile                |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-cube          | /     | Ingestion of auxiliary data into datacube format        |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-procmask      | /     | Generate a processing mask from continuous data         |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-pyramid       | /     | Generation of image pyramids                            |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-mosaic        | /     | Mosaicking of image chips                               |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-synthmix      | /     | Synthetic mixing of training data                       |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-stack         |       | Stack images, works with 4D data model                  |
++--------+---------------------+-------+---------------------------------------------------------+
+|        | force-mdcp          |       | Copy FORCE metadata from one file to another            |
++--------+---------------------+-------+---------------------------------------------------------+
+