Merge branch 'develop' of https://github.com/davidfrantz/force into develop

davidfrantz · davidfrantz · commit b3a8efe00ce5 · 2021-10-29T09:01:46.000+02:00
diff --git a/.github/workflows/docker-publish-debug.yml b/.github/workflows/docker-publish-debug.yml
@@ -16,6 +16,7 @@ env:
 jobs:
   push:
     name: Push debug image to Docker Hub
+    if: github.repository == 'davidfrantz/force'
     runs-on: ubuntu-latest
     steps:
       - name: Check out the repo
diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml
@@ -16,6 +16,7 @@ env:
 jobs:
   push:
     name: Push image to Docker Hub
+    if: github.repository == 'davidfrantz/force'
     runs-on: ubuntu-latest
     steps:
       - name: Check out the repo
diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 **Framework for Operational Radiometric Correction for Environmental monitoring**
 
-**Version 3.7.1-develop**
+**Version 3.7.2-develop**
 
 ![FORCE Logo](/images/force.png)
 
@@ -27,7 +27,7 @@ Non-native data sources can also be processed, e.g. Sentinel-1 SAR data or envir
 
 **Learn how to use FORCE**. Have a look at the [Tutorials](https://force-eo.readthedocs.io/en/latest/howto/index.html). Check regularly for new content.
 
-**Get help**, and help others in the FORCE self-help [Google group](https://groups.google.com/d/forum/force_eo)
+**Get help**, and help others in the FORCE [discussion section](https://github.com/davidfrantz/force/discussions)
 
 **Follow** the FORCE project at [ResearchGate](https://www.researchgate.net/project/FORCE-Framework-for-Operational-Radiometric-Correction-for-Environmental-monitoring)
 
diff --git a/docs/source/components/higher-level/index.rst b/docs/source/components/higher-level/index.rst
@@ -40,7 +40,6 @@ Multiple :ref:`hl-submodules` are available, which either process ARD or feature
    :hidden:
    :maxdepth: 3
 
-   higher-level.rst
    hl-input.rst
    hl-compute.rst
    hl-aux.rst
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -77,6 +77,7 @@
 # a list of builtin themes.
 #
 html_theme = 'default'
+#html_theme = 'sphinx_rtd_theme'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
diff --git a/docs/source/history/v3.rst b/docs/source/history/v3.rst
@@ -1,5 +1,28 @@
 .. _v3:
 
+FORCE v. 3.7.2
+--------------
+
+Release: 20.10.2021
+
+* **FORCE L2PS**
+
+  * ESA made major changes in their Sentinel-2 products, which will become effective as of October 26, 2021.
+    Specifically, ESA introduced additive scaling factors to convert the DNs to TOA reflectance.
+    Older FORCE versions will fail!
+    Update to FORCE >= 3.7.2!
+    Thanks to Vincent Schut and Patrick Griffiths for the heads-up!
+
+* **General changes**
+
+  * Docker containers will only be published to Docker Hub, if working in the original repository.
+    This solves failed GitHub Actions for users that have forked the repository.
+    Thanks to Florian Katerndahl for implementing a fix!
+
+* **New Program**
+
+  * There is now a new program ``force-cube-init``, which can generate a ``datacube-definition.prj`` file without needing to process Level 2 data.
+
 
 FORCE v. 3.7.1
 --------------
diff --git a/docs/source/howto/coreg.rst b/docs/source/howto/coreg.rst
@@ -1,11 +1,10 @@
 .. _tut-coreg:
 
-.. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN
-
-
 Coregistration
 ==============
 
+.. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN
+
 **How to coregister Sentinel-2 with Landsat**
 
 This tutorial explains how to generate co-registered Sentinel-2 Analysis Ready Data using the FORCE Level 2 Processing System.
diff --git a/docs/source/howto/l2-ard.rst b/docs/source/howto/l2-ard.rst
@@ -9,7 +9,7 @@ This tutorial explains what Analysis Ready Data are, and how to use the FORCE Le
 
 .. admonition:: Info
 
-   *This tutorial uses FORCE v. 3.2*
+   *This tutorial uses FORCE v. 3.7*
 
 
 What are Levels?
@@ -108,29 +108,26 @@ An empty Level 2 parameter file can be generated with
 
 .. code-block:: bash
 
-   force-parameter /data/force/param LEVEL2 1
+   force-parameter /data/force/param/l2ps.prm LEVEL2
 
-   LEVEL2
    An empty parameter file skeleton was written to
-     /data/force/param/LEVEL2-skeleton.prm
+     /data/force/param/l2ps.prm
    Note that all parameters need to be given, even though some may not be used
    with your specific parameterization.
-   You should rename the file, e.g. my-first-LEVEL2.prm.
    Parameterize according to your needs and run with
-   force-level2 /data/force/param/my-first-LEVEL2.prm
+   force-level2 /data/force/param/l2ps.prm
     or for a single image:
-   force-l2ps image /data/force/param/my-first-LEVEL2.prm
+   force-l2ps image /data/force/param/l2ps.prm
 
 
-The trailing ``1`` means that descriptions for every parameter will be included.
-If you prefer a shorter parameter file, give a ``0`` instead.
+This will generate a verbose parameter file which includes a description for every parameter. If you
+prefer a shorter parameter file, set the optional ``-c`` flag (see :ref:`aux-parameter` for detailed documentation).
 The descriptions can also be found here: :ref:`l2-param`.
 
-Let’s rename the file, and have a look:
+Let’s have a look:
 
 .. code-block:: bash
 
-   mv /data/force/param/LEVEL2-skeleton.prm /data/force/param/l2ps.prm
    head /data/force/param/l2ps.prm
 
    ++PARAM_LEVEL2_START++
@@ -373,6 +370,30 @@ The processing time (real time) is appended at the end.
    S2A_OPER_MSI_L1C_TL_SGS__20160916T135429_A006453_T33JYG_N02.04: dc: 100.00%. wc:   1.78%. sc:   2.85%. cc: 100.00%. Skip. Processing time: 12 mins 17 secs
 
 
+A more convient way to get an overview is to generate an html report:
+
+.. code-block:: bash
+
+   force-level2-report /data/force/log
+
+
+This tool generates some interactive charts that visualize the information available in the logfiles:
+
+.. raw:: html
+   :file: ../html/report_20211011020320.html
+
+
+.. important::
+   The first chart gives an overview about the processing success.
+   ``Success``, ``Too cloudy``, and ``Coregistration failed`` are all OK.
+   The category ``Error`` is bad and means that something went wrong.
+   If so, the names of the failed images will be listed below.
+
+.. note::
+   ``Error`` can also mean that the images are still under processing.
+   Run *force-level2-report* **after** *force-level2*!
+
+
 Output format
 -------------
 
diff --git a/docs/source/howto/stm.rst b/docs/source/howto/stm.rst
@@ -1,11 +1,10 @@
 .. _tut-stm:
 
-.. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN
-
-
 Spectral Temporal Metrics
 =========================
 
+.. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN
+
 **How to aggregate time series**
 
 This tutorial explains what Spectral Temporal Metrics are, and how to generate them using the Time Series Analysis (TSA) submodule of the FORCE Higher Level Processing system (HLPS).
diff --git a/docs/source/howto/tsi.rst b/docs/source/howto/tsi.rst
@@ -1,10 +1,10 @@
 .. _tut-tsi:
 
-.. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN
-
 Time Series Interpolation
 =========================
 
+.. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN
+
 **How to interpolate and animate time series**
 
 This tutorial introduces the Time Series Analysis (TSA) submodule of the FORCE Higher Level Processing system (HLPS), shows how to interpolate time series, and how to visualize and animate them via QGIS plugins.
@@ -14,8 +14,8 @@ This tutorial introduces the Time Series Analysis (TSA) submodule of the FORCE H
    *This tutorial uses FORCE v. 3.5*
 
 
-----------
-Why interpolation?**
+Why interpolation?
+------------------
 
 Despite regular observations, EO time series are generally not equidistant.
 Nominal equidistance is only given for areas that are not covered by orbital overlaps (16 days revisit at nadir for a single Landsat).
@@ -34,13 +34,13 @@ Equidistance is established, which might be needed for following processing step
 
 Time Series interpolation is a basic option within the :ref:`tsa` submodule of the FORCE :ref:`hlps` Processing system (HLPS), and may be performed before using more advanced time series analysis methods:
 
-.. figure:: img/force-tsa-workflow.jpg
+.. figure:: ../components/higher-level/tsa/tsa-workflow.jpg
 
    *Time Series Analysis Workflow*
 
 
-----------
-RBF interpolation**
+RBF interpolation
+-----------------
 
 In FORCE, a couple of interpolation methods are implemented.
 
diff --git a/docs/source/howto/wvdb.rst b/docs/source/howto/wvdb.rst
@@ -129,7 +129,8 @@ The file holds the long-term average, long-term standard deviation, and the numb
    -53.6847 70.9100 0.546611 0.273735 276
 
 
-.. figure:: img/wvdb.gif" width="750
+.. figure:: ../components/lower-level/wvdb/wvdb.gif
+   :width: 750
 
    *Global animation of the climatology (monthly average)*
 
@@ -245,7 +246,7 @@ This directory is the directory, to which DIR_WVPLUT in the FORCE L2PS parameter
 ``DIR_WVPLUT = /data/Earth/global/wvp/wvdb``
 
 
-If you have finished compiling the WVDB, you may delete the MODIS *.hdf files.
+If you have finished compiling the WVDB, you may delete the MODIS \*.hdf files.
 
 Download the entire data record (in one process - this is slow):
 
diff --git a/docs/source/html/report_20211011020320.html b/docs/source/html/report_20211011020320.html
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -5,7 +5,7 @@ FORCE documentation
 
 **FORCE: Framework for Operational Radiometric Correction for Environmental monitoring**
 
-**Version 3.7.1-develop**
+**Version 3.7.2-develop**
 
 `Download from Github <https://github.com/davidfrantz/force>`_.
 
diff --git a/src/cross-level/_version-cl.h b/src/cross-level/_version-cl.h
@@ -32,7 +32,7 @@ Version number
 extern "C" {
 #endif
 
-#define _VERSION_ "3.7.1-develop"
+#define _VERSION_ "3.7.2-develop"
 
 #ifdef __cplusplus
 }
diff --git a/src/cross-level/string-cl.c b/src/cross-level/string-cl.c
@@ -50,3 +50,39 @@ void copy_string(char *dst, size_t size, const char *src){
   return;
 }
 
+
+int char_to_int(const char *src, int *val){
+long int temp_val;
+char *temp;
+errno = 0;
+
+
+  temp_val = strtol(src, &temp, 0);
+
+  if (temp == src || *temp != '\0' || errno == ERANGE){
+    return FAILURE;}
+
+  if (temp_val < INT_MIN ||
+      temp_val > INT_MAX){
+    return FAILURE;}
+
+  *val = (int)temp_val;
+  return SUCCESS;
+}
+
+
+int char_to_float(const char *src, float *val){
+float temp_val;
+char *temp;
+errno = 0;
+
+
+  temp_val = strtof(src, &temp);
+
+  if (temp == src || *temp != '\0' || errno == ERANGE){
+    return FAILURE;}
+
+  *val = (float)temp_val;
+  return SUCCESS;
+}
+
diff --git a/src/cross-level/string-cl.h b/src/cross-level/string-cl.h
@@ -31,13 +31,18 @@ String handling header
 #include <stdio.h>    // core input and output functions
 #include <stdlib.h>   // standard general utilities library
 #include <string.h>   // string handling functions
+#include <limits.h>   // macro constants of the integer types
+#include <errno.h>    // error numbers
 
+#include "../cross-level/const-cl.h"
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
 void copy_string(char *dst, size_t size, const char *src);
+int char_to_int(const char *src, int *val);
+int char_to_float(const char *src, float *val);
 
 #ifdef __cplusplus
 }
diff --git a/src/lower-level/meta-ll.c b/src/lower-level/meta-ll.c
@@ -334,8 +334,6 @@ GDALDatasetH fp_;
 
 
         meta->cal = allocate_calibration(nb);
-        //alloc((void**)&meta->cal, nb, sizeof(cal_t));
-        //for (b=0; b<nb; b++) init_calib(&meta->cal[b]);
 
 
         // set start of Relative Spectral Response Array
@@ -736,8 +734,6 @@ int svgrid = 5000;
         for (b=0; b<nb; b++) set_brick_sensor(DN, b, sensor);
 
         meta->cal = allocate_calibration(nb);
-        //alloc((void**)&meta->cal, nb, sizeof(cal_t));
-        //for (b=0; b<nb; b++) init_calib(&meta->cal[b]);
 
         // set start of Relative Spectral Response Array
         if (strcmp(sensor, "SEN2A") == 0){
@@ -883,6 +879,25 @@ int svgrid = 5000;
           tokenptr = strtok(NULL, separator);
         }
 
+      // additive scaling factor (since baseline 4.0)
+      } else if (strcmp(tag, "RADIO_ADD_OFFSET") == 0){
+        // re-init to 0 to be compatible with older baselines
+        for (b=0; b<nb; b++){
+           if (meta->cal[b].radd == meta->cal[b].fill) meta->cal[b].radd = 0;
+        }
+        b = -1;
+        while (strcmp(tokenptr, "/RADIO_ADD_OFFSET") != 0){
+          if (b < 0){
+            char_to_int(tokenptr, &b);
+          } else if (b >= 0 || b < nb){
+            char_to_float(tokenptr, &meta->cal[b].radd);
+          }
+          tag = tokenptr;
+          tokenptr = strtok(NULL, separator);
+        }
+        #ifdef FORCE_DEBUG
+        if (b >= 0) printf("additive scaling factor for band %d: %f\n", b, meta->cal[b].radd);
+        #endif
       }
 
       // in case tag (key words) is not the first word in a line
diff --git a/src/lower-level/read-ll.c b/src/lower-level/read-ll.c
@@ -395,11 +395,11 @@ float dn_scale, toa_scale;
       dn_scale  = get_brick_scale(DN,  b);
       toa_scale = get_brick_scale(TOA, b);
       
-      #pragma omp parallel shared(b, nc, dn_scale, toa_scale, dn_, toa_) default(none) 
+      #pragma omp parallel shared(b, nc, dn_scale, toa_scale, dn_, toa_, meta) default(none) 
       {
 
         #pragma omp for schedule(static)
-        for (p=0; p<nc; p++) toa_[b][p] = dn_[b][p]/dn_scale*toa_scale;
+        for (p=0; p<nc; p++) toa_[b][p] = (dn_[b][p] + meta->cal[b].radd) / dn_scale*toa_scale;
         
       }
 

Original file line number	Diff line number	Diff line change
`@@ -77,6 +77,7 @@`
`77`	`77`	`# a list of builtin themes.`
`78`	`78`	`#`
`79`	`79`	`html_theme = 'default'`
	`80`	`+#html_theme = 'sphinx_rtd_theme'`
`80`	`81`
`81`	`82`	`# Theme options are theme-specific and customize the look and feel of a theme`
`82`	`83`	`# further. For a list of options available for each theme, see the`