Merge branch 'develop'

Author: telespazio-tim

README.md

@@ -39,11 +39,11 @@ It is therefore possible, to cover large geographic extent with a __seamless ima
 It is worth noting that the overall accuracy of your final ARD product strongly depends on the accuracy of sen2like auxiliary data. Two categories of auxiliary data are important: the raster reference for geometric corrections and the meteorological data for atmospheric corrections. Regarding atmospheric corrections, one possibility is to use data from the Copernicus Atmosphere Monitoring Service [9]. The Sen2Like team prepared a dedicated CAMS monthly dataset for the Year 2020, available from [here](http://185.178.85.51/CAMS/). Please refer to this short [description](http://185.178.85.51/CAMS/Readme_CAMS2020.txt) for additional information.
 
 For further details on the format specification of the harmonized products or the functionalities of the Sen2Like software, please 
-refer to the [Product Format Specification](sen2like/docs/source/S2-PDGS-MPC-L2HF-PFS-v1.1.pdf), and the [User Manual v1.8](sen2like/docs/source/S2-SEN2LIKE-UM-V1.8.pdf).
+refer to the [Product Format Specification](sen2like/docs/source/S2-PDGS-MPC-L2HF-PFS-v1.2.pdf), and the [User Manual v1.8](sen2like/docs/source/S2-SEN2LIKE-UM-V1.9.pdf).
 
 ## Publications and Contacts
 **Yearning to know more ? Check out**
-*	poster [Sen2Like, a tool to generate Sentinel-2 Harmonised Surface Reflectance Products, First Results With Landsat-8, 3rd S2 Validation Team Meeting](https://www.researchgate.net/publication/332428332_Sen2like_a_Tool_to_Generate_Sentinel-2_Harmonised_Surface_Reflectance_Products_-_First_Results_With_Landsat-8)
+*	poster [Sen2like: A solution for Harmonization and Fusion of Sentinel-2 and Landsat 8/9 data, JACIE 2023](https://www.researchgate.net/publication/372831217_Sen2like_A_solution_for_Harmonization_and_Fusion_of_Sentinel-2_and_Landsat_89_data)
 *	A [Sen2Like Relaxing Video](https://youtu.be/KBSYYBShyos) prepared for [ESA EO PHI-WEEK 2020](https://www.youtube.com/playlist?list=PLvT7fd9OiI9XELZXcljYTftUtJ_NFWRrY)
 *	A [Sen2Like Time Lapse including NDVI graphic](https://youtu.be/yEObvI1KQBg) prepared for QWG#12
 
@@ -60,7 +60,7 @@ And the following research papers :
  + [10] Saunier, S.; Pflug, B.; Lobos, I.M.; Franch, B.; Louis, J.; De Los Reyes, R.; Debaecker, V.; Cadau, E.G.; Boccia, V.; Gascon, F.; Kocaman, S. Sen2Like: Paving the Way towards Harmonization and Fusion of Optical Data. Remote Sens. 2022, 14, 3855. (https://doi.org/10.3390/rs14163855) 
 
 
-**Learn how to use Sen2Like**, have a look at the [User Manual](sen2like/docs/source/S2-SEN2LIKE-UM-V1.8.pdf).
+**Learn how to use Sen2Like**, have a look at the [User Manual](sen2like/docs/source/S2-SEN2LIKE-UM-V1.9.pdf).
 
 **Get help**, contact us at sen2like@telespazio.com.
 

sen2like/.dockerignore

@@ -2,4 +2,6 @@
 *
 # but not what we want to send in the build context
 !requirements.txt
+!requirements_dev.txt
 !sen2like
+!aux_data

sen2like/Dockerfile

@@ -14,20 +14,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
+ARG SEN2LIKE_BUILD_IMAGE_TAG
 # docker image baser on miniconda image (debian:latest)
-FROM continuumio/miniconda3:22.11.1 AS build
+FROM ${SEN2LIKE_BUILD_IMAGE_TAG} AS build
 LABEL stage=sen2like_build
 
 # set the working dir to /usr/local/sen2like
 WORKDIR /usr/local/sen2like
 
-# Create the environment:
-# copy requirements.txt from sources to docker image
-COPY ./requirements.txt .
-# create sen2like env from requirement
-RUN conda create -n sen2like --file requirements.txt -c conda-forge
-
 # Install conda-pack:
 RUN conda install -c conda-forge conda-pack
 
@@ -48,11 +42,18 @@ FROM debian:bullseye-slim AS runtime
 COPY --from=build /venv /venv
 
 # install in curent docker image mesa-glx
-RUN apt-get update && apt-get install -y libgl1-mesa-glx
+RUN apt-get update \
+  && apt-get install -y libgl1-mesa-glx \
+  && rm -rf /var/lib/apt/lists/*
 
 # set PATH with venv
 ENV VIRTUAL_ENV=/venv
 ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+ENV PROJ_LIB=/venv/share/proj
+
+# set the working dir to /usr/local/aux_data
+WORKDIR /usr/local/aux_data
+COPY ./aux_data .
 
 # set the working dir to /usr/local/sen2like
 WORKDIR /usr/local/sen2like

sen2like/Dockerfile-base

@@ -0,0 +1,13 @@
+# docker image based on miniconda image (debian:latest)
+# ARG MINICONDA_DOCKER_VERSION
+# FROM continuumio/miniconda3:${MINICONDA_DOCKER_VERSION}
+FROM continuumio/miniconda3:23.3.1-0
+
+# set the working dir to /usr/local/sen2like
+WORKDIR /usr/local/sen2like
+
+# Create the environment:
+# copy requirements.txt from sources to docker image
+COPY ./requirements.txt .
+# create sen2like env from requirement
+RUN conda create -n sen2like --file requirements.txt -c conda-forge

sen2like/Dockerfile-tests

@@ -0,0 +1,18 @@
+# sen2like-base docker image is based on miniconda image (debian:latest)
+ARG SEN2LIKE_BUILD_IMAGE_TAG
+FROM ${SEN2LIKE_BUILD_IMAGE_TAG}
+LABEL stage=sen2like_tests
+
+# Update the environment:
+# copy dev/tests requirements from sources to docker image
+COPY ./requirements_dev.txt .
+
+# update sen2like env
+RUN conda install -n sen2like --file requirements_dev.txt -c conda-forge
+
+# install system dependencies in curent docker image mesa-glx
+RUN apt-get update \
+    && apt-get install -y libgl1-mesa-glx \
+    && rm -rf /var/lib/apt/lists/*
+    
+# image is ready for tests, usage must activate sen2like conda env

sen2like/README.md

@@ -26,8 +26,11 @@ TOC Generated with markdown all in one: https://github.com/yzhang-gh/vscode-mark
     - [Geometry](#geometry)
     - [Atmcor](#atmcor)
     - [Nbar](#nbar)
+    - [Sbaf](#sbaf)
     - [Fusion](#fusion)
     - [Stitching](#stitching)
+    - [TopographicCorrection (Experimental)](#topographiccorrection-experimental)
+    - [DEMRepository](#demrepository)
     - [OutputFormat](#outputformat)
     - [COGoptions](#cogoptions)
     - [JPEG2000options](#jpeg2000options)
@@ -39,6 +42,8 @@ TOC Generated with markdown all in one: https://github.com/yzhang-gh/vscode-mark
     - [Single tile mode](#single-tile-mode)
     - [Multi tile mode](#multi-tile-mode)
     - [ROI based mode](#roi-based-mode)
+- [Auxiliary data tools](#auxiliary-data-tools)
+  - [DEM downloader](#dem-downloader)
 - [Release notes](#release-notes)
 - [License](#license)
 
@@ -100,7 +105,7 @@ conda activate sen2like
 
 ```bash
 python sen2like.py 
-[INFO    ] 2023-01-11 14:50:54 - sen2like             - Run Sen2like 4.3.0
+[INFO    ] 2023-01-11 14:50:54 - sen2like             - Run Sen2like 4.4.0
 usage: sen2like.py [-h] [--version] [--refImage PATH] [--wd PATH]
                    [--conf PATH] [--confParams STRLIST] [--bands STRLIST]
                    [--allow-other-srs] [--no-run] [--intermediate-products]
@@ -124,8 +129,16 @@ Please refer to the docker documentation to install docker on your environnement
 
 From the sen2like root directory (the one containing `Dockerfile`)
 
+First build the base image : 
+
 ```bash
-docker build -t sen2like . && docker image prune --filter label=stage=sen2like_build -f
+docker build -t sen2like --build-arg sen2like_base .
+```
+
+Then the final image : 
+
+```bash
+docker build -t sen2like --build-arg SEN2LIKE_BUILD_IMAGE_TAG=sen2like_base . && docker image prune --filter label=stage=sen2like_build -f
 ```
 
 The result is a docker image with tag `sen2like:latest`
@@ -149,15 +162,15 @@ Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
 Example
 
 ```bash
-docker image tag sen2like my-internal-docker-registry-url/sen2like:4.3
+docker image tag sen2like my-internal-docker-registry-url/sen2like:4.4
 ```
 
 Push the image on a registry with the command `docker push NAME[:TAG]`
 
 Example
 
 ```bash
-docker push my-internal-docker-registry-url/sen2like:4.3
+docker push my-internal-docker-registry-url/sen2like:4.4
 ```
 
 ## Running the tool
@@ -181,15 +194,15 @@ Build sen2like docker image or pull it from a registry with the command `docker
 Example :
 
 ```bash
-docker pull https://my-internal-docker-registry-url/sen2like:4.3
+docker pull https://my-internal-docker-registry-url/sen2like:4.4
 ```
 
 You can run it directly without entering into the container: 
 
 ```bash
-docker run --rm my-internal-docker-registry-url/sen2like/sen2like:4.3
+docker run --rm my-internal-docker-registry-url/sen2like/sen2like:4.4
 
-[INFO    ] 2023-01-11 14:50:54 - sen2like             - Run Sen2like 4.3.0
+[INFO    ] 2023-01-11 14:50:54 - sen2like             - Run Sen2like 4.4.0
 usage: sen2like.py [-h] [--version] [--refImage PATH] [--wd PATH]
                    [--conf PATH] [--confParams STRLIST] [--bands STRLIST]
                    [--allow-other-srs] [--no-run] [--intermediate-products]
@@ -210,7 +223,7 @@ In the following examples **local** folder `/data` is supposed to exist and cont
 ```bash
 docker run --rm \
   --mount type=bind,source="/data",target=/data \
-  my-internal-docker-registry-url/sen2like/sen2like:4.3 \
+  my-internal-docker-registry-url/sen2like/sen2like:4.4 \
   single-tile-mode 31TFJ \
   --conf "/data/config.ini" \
   --start-date 2017-10-30 --end-date 2017-10-31 \
@@ -224,7 +237,7 @@ Python script `sen2like.py` could be accessed from a docker container.
 Launch the docker binding **local** `/data` folder to the container `/data` folder, example: 
 
 ```bash
-docker run --rm -it --mount type=bind,source="/data",target=/data --entrypoint=/bin/bash my-internal-docker-registry-url/sen2like/sen2like:4.3
+docker run --rm -it --mount type=bind,source="/data",target=/data --entrypoint=/bin/bash my-internal-docker-registry-url/sen2like/sen2like:4.4
 
 root@15a2f44ddd70:/usr/local/sen2like
 
@@ -271,9 +284,10 @@ Enable or disable a processing block based on value `(True, False)`:
 * `doGeometryCheck`: Run the geometric assessment using KLT to compute geometry QI
 * `doToa`: Run the TOA correction
 * `doInterCalibration`: Run the Inter Calibration correction (S2B)
-* `doAtmcor`: Run the Atmospheric correction (SMAC or Sen2Cor)
+* `doAtmcor`: Activate the Atmospheric correction (SMAC or with sen2cor)
 * `doNbar`: Run Nbar correction processing
 * `doSbaf`: Run the Sbaf correction processing
+* `doTopographicCorrection`: Activate the Topographic correction (sen2like or with sen2cor)
 * `doFusion`: Run the Fusion processing
 * `doPackagerL2H`: Run the packaging processing for harmonized products
 * `doPackagerL2F`: Run the packaging processing for fused products
@@ -287,7 +301,6 @@ Indicates path for special directories:
 * `cams_daily_dir`: Where the CAMS daily files are located 
 * `cams_hourly_dir`: Where the CAMS hourly files are located 
 * `cams_climatology_dir`: Where the CAMS climatology files are located 
-* `dem_dir`: Where the DEM files are located 
 * `scl_dir`: Where the auxiliary scl maps files are located 
 
 #### Downloader
@@ -349,7 +362,7 @@ url_parameters_pattern_Sentinel2 = /data/PRODUCTS/Sentinel2/31TFJ
 
 Define parameters for geometric correction.
 
-* `reference_band`= The reference band to be used for geometric correction
+* `reference_band`: The reference band to be used for geometric correction
 * `doMatchingCorrection`: Apply the matching correction (`True`, `False`)
 * `doAssessGeometry`: Assess geometry (Band list separated by comma.)
 * `references_map`: Path to the reference json file containing the reference image for each tile
@@ -359,7 +372,7 @@ Define parameters for geometric correction.
 
 Atmospheric correction method to use.
 
-* `use_sen2cor`: Activate sen2cor for Atmospheric correction (SMAC otherwise)
+* `use_sen2cor`: Activate sen2cor for Atmospheric correction (SMAC otherwise), `doAtmcor` MUST be `True`
 * `sen2cor_path`: Path to sen2cor tool command (L2A_Process.py)
 
 #### Nbar
@@ -369,6 +382,23 @@ Define parameters for Nbar processing.
 * `nbar_methode`: Method to get BRDF coefficients. Currently, available methods are : ROY, VJB
 * `vjb_coeff_matrice_dir`: If VJB method is selected, directory path of the BRDF coefficients netcdf file
 
+#### Sbaf
+
+Define parameters for SBAF processing.
+
+* `adaptative`: **Experimental** Use adaptative SBAF or not. Adaptative SBAF is based on NDVI to which is apply a factor and offset depending the processed band.
+* `adaptative_band_candidates`: S2 equivalent band list separated by comma for which to apply adaptative SBAF. For now B04, B11 and B12 are known to be the better candidates.  
+  Band mapping table: 
+  |Landsat|S2|
+  |-|-|
+  |B01|B01|
+  |B02|B02|
+  |B03|B03|
+  |B04|B04|
+  |B05|B8A|
+  |B06|B11|
+  |B07|B12|
+
 #### Fusion
 
 Define parameters for Fusion processing.
@@ -385,6 +415,33 @@ Define parameters for stitching processing.
 * `same_utm_only`: Enable or disable stitching with product on different UTM as current processed product.  
   For Landsat it allows to use previous or next product that can be on another UTM for stitching and have a better tile coverage.
 
+#### TopographicCorrection (Experimental)
+
+Define parameters for topographic correction process. Topographic correction can be done with the sen2like processing block or with sen2cor.
+
+* `sen2cor_topographic_correction`: with topographic correction activated (`doTopographicCorrection=True`), and atmcor with sen2cor enabled (`doAtmcor=True` and `use_sen2cor=True`), the topographic correction is delegated to sen2cor instead of the sen2like TopographicCorrection processing block.
+* `topographic_correction_limiter`: limit to this value the maximum topographic factor to apply (only for sen2like topographic correction, not sen2cor).
+* `apply_valid_pixel_mask`: Use valid pixel mask to select pixel for which the correction is done. Useful to not apply correction on cloudy pixels (only for sen2like topographic correction, not sen2cor).
+
+
+#### DEMRepository
+
+Define parameters for DEMRepository component.
+
+The DEMRepository component is responsible to access DEM files mainly for [`TopographicCorrection`](#topographiccorrection) processing block
+
+* `dem_folder`: Base folder path that host DEM dataset
+* `dem_dataset`: DEM dataset to use
+* `src_dem_resolution`: resolution to use for DEM in DEM dataset
+
+DEM storage tree MUST follow the following structure : 
+
+`{dem_folder}/{dem_dataset}/Copernicus_DSM_{src_dem_resolution}m_{tile_code}.TIF`
+
+Where `tile_code` is the current processed MGRS tile.
+
+See [DEM Downloader chapter](#dem-downloader) to know how to generate MGRS DEM tile.
+
 #### OutputFormat
 
 Define output format, gain and offset for image files.
@@ -393,7 +450,7 @@ Define output format, gain and offset for image files.
 * `offset`: DN Offset to substract from the output image (DN to reflectance conversion)
 * `output_format`: Format of the output image. Supported formats: COG, GTIFF (for geotiff), JPEG2000.
 
-*Note: DN to reflectance conversion: reflectance = (DN - offset) / gain *
+> *Note: DN to reflectance conversion: reflectance = (DN - offset) / gain*
 
 #### COGoptions
 
@@ -714,6 +771,81 @@ Debug arguments:
   --no-log-date         Do no store date in log (default: False)
 ```
 
+## Auxiliary data tools
+
+Auxiliary data tools are located in the folder `aux_data`
+
+For now there is only the DEM downloader tools.
+
+### DEM downloader
+
+This tool allows to create DEM aux data for TopographicCorrection of Sen2like.
+
+It creates a DEM as GeoTiff having the same extend and UTM projection as a MGRS tile.
+
+To do so, from a given MGRS tile code, it retrieves matching Copernicus DEM tiles, build a mosaic, then crop and reproject it.
+
+In order to use this tool, `sen2like` module/folder **MUST** be in `PYTHONPATH`.
+
+Usage example to display the help: 
+
+```bash
+PYTHONPATH=sen2like/ aux_data/dem/dem_downloader.py -h
+```
+
+It should display:
+
+```
+usage: dem_downloader.py [-h] --server_url SERVER_URL [--debug] MGRS_TILE_CODE DEM_DATASET_NAME DEM_LOCAL_URL
+
+    Create DEM for a MGRS Tile.
+    The generated DEM is TIF file in the MGRS extend and its projected in the MGRS tile EPSG (UTM).
+    
+
+positional arguments:
+  MGRS_TILE_CODE        MGRS Tile code for witch to generate DEM, example 31TFJ
+  DEM_DATASET_NAME      DEM dataset name, example COP-DEM_GLO-90-DGED__2022_1
+  DEM_LOCAL_URL         Base output folder for generated DEM, example /data/AUX_DATA/
+                        Generated files are stored as follow :
+                        /data/AUX_DATA/{DEM_DATASET_NAME}/Copernicus_DSM_{resolution}m_{MGRS_TILE_CODE}.TIF
+                                
+
+options:
+  -h, --help            show this help message and exit
+  --server_url SERVER_URL
+                        DEM server base URL (default: https://prism-dem-open.copernicus.eu/pd-desk-open-access/prismDownload/)
+  --debug, -d           Enable Debug mode (default: False)
+```
+
+The `--server_url` options MUST allow to produce URL formatted as follow to download tiles : `<SERVER_URL>/<DEM_DATASET_NAME>/<DEM_GEOCELL_FILE>`
+ 
+With: 
+
+- `<DEM_DATASET_NAME>` is the `DEM_DATASET_NAME` program argument
+- `<DEM_GEOCELL_FILE>` is a file name that match `Copernicus_DSM_VV_WWW_XX_YYY_ZZ.tar`, example: `Copernicus_DSM_30_N42_00_E012_00.tar`  
+See [Product Structure / Naming Convention in this document](https://spacedata.copernicus.eu/documents/20123/122407/GEO1988-CopernicusDEM-SPE-002_ProductHandbook_I5.0+%281%29.pdf)
+
+
+The tools is also available in the Sen2like docker image in `/usr/local/aux_data` folder.  
+`sen2like` path to use for `PYTHONPATH` is `/usr/local/sen2like`.
+
+Example :
+
+```bash
+docker run --rm \
+  -v /data/DEM:/data/DEM \
+  -e PYTHONPATH=/usr/local/sen2like \
+  --entrypoint /usr/local/aux_data/dem/dem_downloader.py \
+  sen2like \
+  33TTG \
+  COP-DEM_GLO-90-DGED__2022_1 \
+  /data/DEM
+```
+
+It results a file `/data/DEM/COP-DEM_GLO-90-DGED__2022_1/Copernicus_DSM_90m_33TTG.TIF` on the docker host.
+
+The tool keep extracted Copernicus geocell DEM TIF files in folder `<DEM_LOCAL_URL>/<DEM_DATASET_NAME>/geocells` to avoid to download them every time it needs them to create a MGRS DEM file.
+
 ## [Release notes](./release-notes.md)
 
 ## License