Merge pull request #50 from ESA-PhiLab/develop

merge develop

Author: BuddyVolly

.gitignore

@@ -6,4 +6,6 @@ __pycache__
 htmlcov
 .empty
 .DS_Store
-ost/.DS_Store
+ost/.DS_Store
+build
+dist

Dockerfile

@@ -1,13 +1,13 @@
-FROM ubuntu:18.04
+FROM ubuntu:20.04
 
-LABEL maintainer="Petr Sevcik, EOX"
-LABEL OpenSARToolkit='0.11.1'
+LABEL maintainer="Andreas Vollrath, FAO"
+LABEL OpenSARToolkit='0.12.3'
 
 # set work directory to home and download snap
 WORKDIR /home/ost
 
 # copy the snap installation config file into the container
-COPY snap8.varfile $HOME
+COPY snap.varfile $HOME
 
 # update variables
 ENV OTB_VERSION="7.3.0" \
@@ -42,9 +42,9 @@ RUN alias python=python3 && \
     mkdir /home/ost/programs && \
     wget $SNAP_URL/$TBX && \    
     chmod +x $TBX && \
-    ./$TBX -q -varfile snap7.varfile && \
+    ./$TBX -q -varfile snap.varfile && \
     rm $TBX && \
-    rm snap7.varfile && \
+    rm snap.varfile && \
     cd /home/ost/programs && \
     wget https://www.orfeo-toolbox.org/packages/${OTB} && \ 
     chmod +x $OTB && \

README.md

@@ -32,19 +32,23 @@ https://github.com/ESA-PhiLab/OST_Notebooks for getting started.
 
 ### Docker 
 
-A docker image is available from docker hub that contains the full package, 
+**Update!**
+
+Dockerhub is not permitting automatic builds. Therefore you need to
+build your own docker image using the DOCKERFIlE https://raw.githubusercontent.com/ESA-PhiLab/OpenSarToolkit/main/Dockerfile
+
+The resulting docker image contains the full package, 
 including ESA's Sentinel-1 Toolbox, Orfeo Toolbox, Jupyter Lab as well
-as the Open SAR Toolkit the tutorial notebooks.
+as the Open SAR Toolkit tutorial notebooks.
 
 Docker installation is possible on various OS. Installation instructions can be 
 found at https://docs.docker.com/install/
 
 After docker is installed and running, launch the container with 
-(adapt the path to the shared host folder):
+(adapt the path to the shared host folder and the name of the docke rimage at the very end):
 
 ```
-docker pull buddyvolly/opensartoolkit
-docker run -it -p 8888:8888 -v /shared/folder/on/host:/home/ost/shared buddyvolly/opensartoolkit
+docker run -it -p 8888:8888 -v /shared/folder/on/host:/home/ost/shared docker/image
 ```
 
 The docker image automatically executes the jupyter lab and runs it on 
@@ -68,6 +72,8 @@ If you install SNAP into the standard directory, OST should have no problems
 to find the SNAP command line executable. Otherwise you need to define the path 
 to the gpt file on your own during processing.
 
+**Make sure to use SNAP 8 with the latest updates installed.**
+
 ##### Orfeo Toolbox
 
 If you want to create mosaics between different swaths, OST will rely on the 
@@ -78,79 +84,25 @@ https://www.orfeo-toolbox.org/download/
 Make sure that the Orfeo bin folder is within your PATH variable to allow 
 execution from command line.
 
-#### OST installation
-
-OST is developed under Ubuntu 18.04 OS in python 3.6. It has not been tested 
-much on other OS and python versions, but should in principle work on any OS 
-and any python version >= 3.5.
-
-##### Ubuntu/Debian Linux (using pip)
-
-Before installation of OST, run the following line on the terminal to 
-install further dependencies:
-
-```
-sudo apt install python3-pip git libgdal-dev python3-gdal libspatialindex-dev nodejs npm
-```
-
-then install OST as a global package (for all users, admin rights needed):
+##### Further dependencies (libs etc)
 
+Ubuntu 18.04 and later:
 ```
-sudo pip3 install git+https://github.com/ESA-PhiLab/OpenSarToolkit.git
+sudo apt install python3-pip git libgdal-dev python3-gdal libspatialindex-dev nodejs npm libgfortran5
 ```
 
-or as local package within your home folder (no admin rights needed):
-
-```
-pip3 install --user git+https://github.com/ESA-PhiLab/OpenSarToolkit.git
-```
+Any Operating system using (mini)conda https://www.anaconda.com/:
 
-run those commands to enable the correct display of progress bars
 ```
-sudo pip3 install jupyterlab
-# this is needed for the progress bar when downloading
-sudo jupyter-labextension install @jupyter-widgets/jupyterlab-manager
-sudo jupyter nbextension enable --py widgetsnbextension
-```
-
-##### Mac OS (using homebrew/pip)
-
-If not already installed, install homebrew as explained on https://brew.sh
-
-After installation of homebrew, open the terminal and install 
-further dependecies:
-
-```
-brew install python3 gdal2 gdal2-python git
-```
-
-then install OST with python pip:
-```
-pip3 install git+https://github.com/ESA-PhiLab/OpenSarToolkit.git
-
-# this is needed for the progress bar when downloading data
-jupyter labextension install @jupyter-widgets/jupyterlab-manager
-jupyter nbextension enable --py widgetsnbextension
+conda install pip gdal jupyter jupyterlab git matplotlib numpy rasterio imageio rtree geopandas fiona shapely matplotlib descartes tqdm scipy joblib retrying pytest pytest-cov nodejs
 ```
 
-##### Conda Installation (Windows, Mac, Linux)
-
-Follow the installation instructions for conda (Miniconda is sufficient) at:
-https://docs.conda.io/projects/conda/en/latest/user-guide/install/
-
-Then run the conda command to install OST's dependencies:
-```
-conda install pip gdal jupyter jupyterlab git matplotlib numpy rasterio imageio rtree geopandas fiona shapely matplotlib descartes tqdm scipy joblib retrying pytest pytest-cov nodejs
+#### OST installation
 
-# this is needed for the progress bar when downloading
-jupyter labextension install @jupyter-widgets/jupyterlab-manager
-jupyter nbextension enable --py widgetsnbextension
-```
+You can use then use pip to install Open SAR Toolkit: 
 
-Finally get the OST by using pip 
-(we will work in future on a dedicated conda package for OST).
 ```
-pip install git+https://github.com/ESA-PhiLab/OpenSarToolkit.git
+pip install opensartoolkit
 ```
 
 
@@ -206,5 +158,6 @@ that are developed in parallel to this core package and should help to get start
 
 ## Author
 
-* Andreas Vollrath, ESA
+* Andreas Vollrath, FAO (ex-ESA philab)
 * Petr Sevcik, EOX
+* James Wheeler, ESA philab

ost/Project.py

@@ -3,6 +3,7 @@
 
 # ------ bug of rasterio --------
 import os
+
 if "GDAL_DATA" in list(os.environ.keys()):
     del os.environ["GDAL_DATA"]
 if "PROJ_LIB" in list(os.environ.keys()):
@@ -69,7 +70,7 @@ def __init__(
             logger.info('Project directory already exists. '
                         'No data has been deleted at this point but '
                         'make sure you really want to use this folder.')
-            
+
         # define project sub-directories if not set, and create folders
         self.download_dir = self.project_dir.joinpath('download')
         self.download_dir.mkdir(parents=True, exist_ok=True)
@@ -366,7 +367,7 @@ def download(self, inventory_df, mirror=None, concurrent=2,
         download_df = inventory_df[inventory_df.download_path == 'None']
 
         # to download or not ot download - that is here the question
-        if not download_df.any().any():
+        if download_df.empty:
             logger.info('All scenes are ready for being processed.')
         else:
             logger.info('One or more scene(s) need(s) to be downloaded.')
@@ -617,7 +618,6 @@ def pre_download_srtm(self):
         logger.info('Pre-downloading SRTM tiles')
         srtm.download_srtm(self.aoi)
 
-
     def bursts_to_ards(
             self,
             timeseries=False,
@@ -712,13 +712,13 @@ def bursts_to_ards(
         if overwrite:
             logger.info('Deleting processing folder to start from scratch')
             h.remove_folder_content(self.config_dict['processing_dir'])
-            
+
         # --------------------------------------------
         # 5 set resolution to degree
         # self.ard_parameters['resolution'] = h.resolution_in_degree(
         #    self.center_lat, self.ard_parameters['resolution'])
 
-        if self.config_dict['max_workers'] > 1:
+        if self.config_dict['max_workers'] > 1 and self.ard_parameters['single_ARD']['dem']['dem_name'] == 'SRTM 1Sec HGT':
             self.pre_download_srtm()
 
         # --------------------------------------------

ost/generic/common_wrappers.py

@@ -192,7 +192,7 @@ def terrain_correction(infile, outfile, logfile, config_dict):
         projection = f"AUTO:{dem_dict['out_projection']}"
     # epsg codes
     elif int(dem_dict['out_projection']) == 4326:
-        projection = CRS.from_epsg(4326).to_wkt()
+        projection = 'WGS84(DD)'
     else:
         projection = f"EPSG:{dem_dict['out_projection']}"
 
@@ -277,7 +277,7 @@ def ls_mask(infile, outfile, logfile, config_dict):
         projection = f"AUTO:{dem_dict['out_projection']}"
     # epsg codes
     elif int(dem_dict['out_projection']) == 4326:
-        projection = CRS.from_epsg(4326).to_wkt()
+        projection = 'WGS84(DD)'
     else:
         projection = f"EPSG:{dem_dict['out_projection']}"
 

ost/helpers/db.py

@@ -151,7 +151,7 @@ def shpGeom2pg(self, aoi, tablename):
             feature = layer.GetFeature(i)
             wkt = feature.GetGeometryRef().ExportToWkt()
 
-            if inProj4 is not '+proj=longlat +datum=WGS84 +no_defs':
+            if inProj4 != '+proj=longlat +datum=WGS84 +no_defs':
                 wkt = reproject_geometry(wkt, inProj4, 4326)
 
             wkt = 'St_GeomFromText(\'{}\', 4326)'.format(wkt)

ost/helpers/scihub.py

@@ -198,8 +198,8 @@ def check_connection(uname, pword,
 
     # we use some random url for checking (also for czech mirror)
     url = (
-        f'{base_url}/odata/v1/Products?'
-        '$select=Id&$filter=substringof(%27_20200714T165921_%27,Name)'
+        f'{base_url}/odata/v1/Products('
+        '\'8f30a536-c01c-4ef4-ac74-be3378dc44c4\')/$value'
     )
 
     response = requests.get(url, auth=(uname, pword), stream=True)

ost/helpers/srtm.py

@@ -95,13 +95,13 @@ def download_srtm(aoi):
     warnings.filterwarnings(
         'ignore', 'Geometry is in a geographic CRS', UserWarning
     )
-    srtm = gpd.read_file(
-        OST_ROOT.joinpath('aux/srtm1sectiles.gpkg')
-    )
+
+    srtm = gpd.read_file(OST_ROOT.joinpath('aux/srtm1sectiles.gpkg'))
 
     aoi_gdf = vec.wkt_to_gdf(aoi)
     aoi_gdf['geometry'] = aoi_gdf.geometry.buffer(1)
     overlap_df = gpd.overlay(srtm, aoi_gdf, how='intersection')
+
     iter_list = []
     for file in overlap_df.url.values:
         iter_list.append(file)

ost/helpers/vector.py

@@ -1,9 +1,10 @@
 import os
 import json
-from functools import partial
 from pathlib import Path
 
 import pyproj
+from pyproj.crs import ProjectedCRS
+from pyproj.crs.coordinate_operation import AzumuthalEquidistantConversion
 import geopandas as gpd
 import logging
 
@@ -171,7 +172,7 @@ def reproject_geometry(geom, inproj4, out_epsg):
     return geom
 
 
-def geodesic_point_buffer(lat, lon, meters, envelope=False):
+def geodesic_point_buffer(lon, lat, meters, envelope=False):
     """
 
     :param lat:
@@ -181,23 +182,24 @@ def geodesic_point_buffer(lat, lon, meters, envelope=False):
     :return:
     """
 
-    # get WGS 84 proj
-    proj_wgs84 = pyproj.Proj('epsg:4326')
-
-    # Azimuthal equidistant projection
-    aeqd_proj = '+proj=aeqd +lat_0={lat} +lon_0={lon} +x_0=0 +y_0=0'
-    project = partial(
-        pyproj.transform,
-        pyproj.Proj(aeqd_proj.format(lat=lat, lon=lon)),
-        proj_wgs84
+    proj_crs = ProjectedCRS(
+        conversion=AzumuthalEquidistantConversion(float(lat), float(lon))
     )
 
+    proj_wgs84 = pyproj.Proj('EPSG:4326')
+
+    Trans = pyproj.Transformer.from_proj(
+        proj_crs,
+        proj_wgs84,
+        always_xy=True
+    ).transform
+
     buf = Point(0, 0).buffer(meters)  # distance in metres
 
     if envelope is True:
-        geom = Polygon(transform(project, buf).exterior.coords[:]).envelope
+        geom = Polygon(transform(Trans, buf).exterior.coords[:]).envelope
     else:
-        geom = Polygon(transform(project, buf).exterior.coords[:])
+        geom = Polygon(transform(Trans, buf).exterior.coords[:])
 
     return geom.wkt
 
@@ -240,7 +242,7 @@ def latlon_to_wkt(
         aoi_wkt = aoi_geom.to_wkt()
 
     elif buffer_meter:
-        aoi_wkt = geodesic_point_buffer(lat, lon, buffer_meter, envelope)
+        aoi_wkt = geodesic_point_buffer(lon, lat, buffer_meter, envelope)
 
     return aoi_wkt
 

ost/s1/burst_batch.py

@@ -309,7 +309,7 @@ def _create_timeseries(burst_gdf, config_file):
             product, product_name = list(pr)
 
             # take care of H-A-Alpha naming for file search
-            if pol in ['Alpha', 'Entropy', 'Anisotropy'] and product is 'pol':
+            if pol in ['Alpha', 'Entropy', 'Anisotropy'] and product == 'pol':
                 list_of_files = sorted(
                     list(burst_dir.glob(f'20*/*data*/*{pol}*img')))
             else:
@@ -654,7 +654,7 @@ def mosaic_timescan(burst_inventory, config_file):
 
     if 'harmonics' in metrics:
         metrics.remove('harmonics')
-        metrics.extend(['amplitude', 'phase', 'residuals'])
+        metrics.extend(['amplitude', 'phase', 'residuals', 'model_mean'])
 
     if 'percentiles' in metrics:
         metrics.remove('percentiles')