Working with non cloud-optimized datasets (e.g. 3DEP 1m DEMS, NEON)

[scottyhq]

Just documenting something to be aware of:

our loading functions are designed to get subsets from cloud-hosted files, unfortuntaley these are not always COGs (https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/validate_cloud_optimized_geotiff.py)

wget https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/Projects/CO_WestCentral_2019_A19/TIFF/USGS_1M_12_x76y427_CO_WestCentral_2019_A19.tif

python validate_cloud_optimized_geotiff.py USGS_1M_12_x76y427_CO_WestCentral_2019_A19.tif

USGS_1M_12_x76y427_CO_WestCentral_2019_A19.tif is NOT a valid cloud optimized GeoTIFF.
The following errors were found:
 - The offset of the main IFD should be 8. It is 143211950 instead
 - The offset of the IFD for overview of index 0 is 13246, whereas it should be greater than the one of the main image, which is at byte 143211950

They are at least tiled with overviews, but there may be a lot of network requests / slow performance, select metadata from gdalinfo below:

Driver: GTiff/GeoTIFF
Files: USGS_1M_12_x76y427_CO_WestCentral_2019_A19.tif
Size is 10012, 10012
Coordinate System is:
PROJCRS["NAD83 / UTM zone 12N",

Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
  PREDICTOR=3

Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
  Min=1678.988 Max=1893.657 
  Minimum=1678.988, Maximum=1893.657, Mean=1765.101, StdDev=41.513
  NoData Value=-999999
  Overviews: 5006x5006, 2503x2503, 1252x1252, 626x626, 313x313

[scottyhq]

The same is true for the OPR DEMs (although these seem to be delivered in much smaller tiles to begin with:

wget https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/OPR/Projects/CO_WestCentral_2019_A19/CO_WestCentral_2019/TIFF/USGS_OPR_CO_WestCentral_2019_A19_be_w1032n1780.tif

USGS_OPR_CO_WestCentral_2019_A19_be_w1032n1780.tif is NOT a valid cloud optimized GeoTIFF.
The following errors were found:
 - The offset of the main IFD should be 8. It is 2042532 instead
 - The offset of the IFD for overview of index 0 is 690, whereas it should be greater than the one of the main image, which is at byte 2042532

Files: USGS_OPR_CO_WestCentral_2019_A19_be_w1032n1780.tif
Size is 1000, 1000
Coordinate System is:
PROJCRS["NAD83(2011) / Conus Albers",

Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
  PREDICTOR=3

Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
  Min=1790.870 Max=1842.960 
  Minimum=1790.870, Maximum=1842.960, Mean=1815.515, StdDev=10.708
  NoData Value=-999999
  Overviews: 500x500, 250x250, 125x125, 63x63, 32x32

[scottyhq]

Also NEON DEMs which are written by line rather than tiles. Also they non-compliant GeoTiffs (but it's a minor tag sorting issue)!

gdalinfo "/vsicurl?empty_dir=yes&url=https://storage.googleapis.com/neon-aop-products/2020/FullSite/D10/2020_ARIK_2/L3/DiscreteLidar/DSMGtif/NEON_D10_ARIK_DP3_713000_4394000_DSM.tif"

Warning 1: NEON_D10_ARIK_DP3_713000_4394000_DSM.tif: TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not sorted in ascending order
Warning 1: TIFFReadDirectoryCheckOrder:Invalid TIFF directory; tags are not sorted in ascending order
Driver: GTiff/GeoTIFF
Files: /tmp/NEON_D10_ARIK_DP3_713000_4394000_DSM.tif
Size is 1000, 1000
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 13N",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            MEMBER["World Geodetic System 1984 (G2139)"],
            MEMBER["World Geodetic System 1984 (G2296)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["UTM zone 13N",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",-105,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Navigation and medium accuracy spatial referencing."],
        AREA["Between 108°W and 102°W, northern hemisphere between equator and 84°N, onshore and offshore. Canada - Northwest Territories (NWT); Nunavut; Saskatchewan. Mexico. United States (USA)."],
        BBOX[0,-108,84,-102]],
    ID["EPSG",32613]]
Data axis to CRS axis mapping: 1,2
Origin = (713000.000000000000000,4395000.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  TIFFTAG_IMAGEDESCRIPTION=Elevation LiDAR - NEON.DP3.30024 acquired at ARIK by RIEGL LASER MEASUREMENT SYSTEMS Q780 2220855 as part of 2020-P3C1
  TIFFTAG_SOFTWARE=Tif file created with a Matlab script (write_gtiff.m) written by Tristan Goulden (tgoulden@battelleecology.org) with data processed from the following scripts: create_tiles_from_mosaic.m, combine_DTM_DSM_gtif.m, LasTools_workflow.csh which implemented LAStools version 200728.
  TIFFTAG_DATETIME=Flown on 2020061714, 2020062014
  TIFFTAG_ARTIST=Created by the National Ecological Observatory Network (NEON)
  TIFFTAG_COPYRIGHT=The National Ecological Observatory Network is a project sponsored by the National Science Foundation and managed under cooperative agreement by Battelle. This material is based in part upon work supported by the National Science Foundation under Grant No. DBI-0752017.
  TIFFTAG_XRESOLUTION=1
  TIFFTAG_YRESOLUTION=1
  TIFFTAG_MINSAMPLEVALUE=1255
  TIFFTAG_MAXSAMPLEVALUE=1275
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  713000.000, 4395000.000) (102d30'59.18"W, 39d40'41.84"N)
Lower Left  (  713000.000, 4394000.000) (102d31' 0.34"W, 39d40' 9.44"N)
Upper Right (  714000.000, 4395000.000) (102d30'17.24"W, 39d40'40.94"N)
Lower Right (  714000.000, 4394000.000) (102d30'18.41"W, 39d40' 8.54"N)
Center      (  713500.000, 4394500.000) (102d30'38.79"W, 39d40'25.19"N)
Band 1 Block=1000x1 Type=Float32, ColorInterp=Gray
  NoData Value=-9999