feat: pixel based query rects for raster requests (#854)

* remove test ouptuts

* add optional bounds points mock source, adapt rasterization tests

* remove prints

* adapt to changes

* building again

* building operators

* remove debug output

* more tests

* adapt to changes

* remove debug output

* remove dbg! calls

* more tests

* clippy fix datatypes

* fix clippy things in services

* clippy lints and reduce complexity

* more lints and renames

* adapt cache text

* more lints

* use correct type ein api

* interpolation: reuse geotransform and use fail safe lookup

* remove unused methods

* update tests

* togeotiff: don't set tiled for COG

* use type from api model in dataset handlers

* reuse pixel to tiling logic

* remove resolution from VectorStreamQuery

* RasterWebsocketStream: log why stream was closed

* use SpatialReference to lookup specs

* remove unused code

* add MlModelResource conversion

* adapt dataset

* streamline tiling

* tiling cleanup and adapt s2 stac provider

* change how raster bounds are reprojected: use actual coords and discard invalid ones.

* prepare up side down raster reading

* update test images

* workflow optimization (wip)

* add downscaled ndvi test data set

* optimization for interpolation

* downsampling optimization

* projection optimization

* optimization in wms and wcs

* test optimization in wms

* optimization for rasterization

* optimization of raster vector join

* test optimization on finer resolution

* lintst & SpatialReference by value

* implement optimization for more operators

* solve Fixme on partitions_extent

* fmt

* test optimization of complex workflow

* optimization for plots

* clean up code

* bring back flip-y and add overview level loading

* cleenup

* adapt tests

* fix gdal source initialization

* add test that optimization removes redundant upsampling

* remove enum from RasterResultDescriptor

* lints

* migration of RasterResultDescriptor

* remove sentinel-2 zones and bands from provider

* adapt to main changes

* fix tests

* move s2 provider zones

* lints

* fix test

* use TimeInterval in netcdf tests

* fix gbif test

* bring back soem benches + linter

* fmt

* reduce wms cache test area

* update ndvi_list dataset

* sql lints

* sql lints

* sql lints

* sql lints

* update dataset defs

* measurement serialization and use btreemap for classes

* adapt datasets api type

* don't fail on resolution=None

* update openapi

* lints

* update RasterDatasetFromWorkflow schema

* update openapi

* clippy :)

* clippy

* clippy

* copy gdal source

* load single tile from multiple files (wip)

* load tiles from db (wip)

* rename type SpatialQuery to SpatialBounds

* rename field spatial_query -> spatial_bounds

* fmt

* remove RasterSpatialQueryRectangle and use GRidBoundingBox2D direct

* fix and test (wip)

* clean up (wip)

* more tests

* add test with reversed z index

* clean up

* migration and tests

* VisualPointClusteringParams

* LineSimplification only use epsilon

* adapt to main chantes

* fmt

* reorder migrations

* remove pub in QueryRectangle

* add text to reproject_spatial_grid_bounds

* cleanup QueryRectangle

* make MockQueryContext creation more private

* chores

* trailing whitespace

* bump version

* rebase current_schema

* reorder attributes make the test happy

* remove fixmes

* overview resolution for test pngs

* update openapi

* update migration name

* fmt

* move more to_dataset logic into task

* bring back density rasterization

* more density rastizion

* re-activate and fix tests

* bring back tests for density rasterizer

* remove todo from ArunaDataProvider

* update sentinel providers

* add multi tiles tests to operators crate

* add benchmark new vs old gdal source

* fix bench and buffer new gdal source

* clean up code

* clippy

* example for importing tiles from a directory

* fix edge case where no tile is inside query rectangle. fix layers with datasets using new gdal source. add force example http calls

* fix test and clippy

* clippy

* fix tests

* clippy

* make pixel edge check more robust with respect to floating point imprecisions

* cargo fmt

* optimize new gdal source

* fix(operators)!: resolution in visual clustering

* flip QueryProcessor impl for RasterQueryProcessor to RasterQueryProcessor: QueryProcessor

* time filler part 1

* reading overview levels from multi tiles

* clean up code

* refactor read advise calculation

* finalize TimeDescriptor and time_query

* fmt

* fluff1

* fluff2

* fix time dimension of test dataset

* vector reprojection optimization

* remove skipping no-data tiles in GdalSource, add wms return empty tiles if query out of data bounds

* reinitialize workflow to fix operator path

* some cleanup

* validate tile compatibility

* produce warning in rasterisation operator

* check tile validity on insert and perform batches

* clippy

* use time gap filler and fix gap filling for regular time series in database

* revert default buffer size

* implement time_query for new gdal source

* cargo fmt

* add default time_query impl for regular time

* force example

* update clippy fix

* check TiimeStep to be > 0

* update dataset extend when inserting new tiles

* check tile alignment before insertion

* refactor clippy

* clippy

* fix errors introduced while fixing lints

* sqlfluff

* layer collection for tile import

* add id to tiles
fix time query edge case
fix time in tile import example

* make file paths relative to volume

* turn example into cli

* clippy

* add remote data dir to cli params

* fix test

* adjust openapi

* remove empty lines

* update openapi.json

* change time dimension repr in openapi

* fix time descriptor

* fix wfs open api spec

* fix wms/wcs

* clippy

* change deserialization of ogc queries

* remove query resolution form wfs

* fix ogc openapi schema

* fix schemas

* cargo fmt

* remove redundant type

* fix import lost in merge

* more helpfull debug output

* more helpful debug output

* fix: properly initialize wrapped operator

* fix: remove workflow benchmarks using gdal becaue benchmark logic no longer works now that resolutions are no longer part of the query rectangle

* remove all workflow benches

* fix: shift overview grid by a pixel fraction if needed

* fix: unify geotransform in api

* reduce no-data  tiles produced in empty tile-intervals with regular steps

* make GeoTransform Copy

* check WMS request size

* check WMS image size

* remove unused TimeStep::new_unchecked

* delete empty file

* more detailed method name

* renumber migrations

* remove obsolete test

---------

Co-authored-by: Michael Mattig <michael.m@ttig.de>
Co-authored-by: Christian Beilschmidt <christian.beilschmidt@geoengine.de>

Author: 3 people

Cargo.lock

@@ -8,7 +8,7 @@ exclude = [
 resolver = "2"
 
 [workspace.package]
-version = "0.8.0"
+version = "0.9.0"
 authors = [
     "Christian Beilschmidt <beilschmidt@mathematik.uni-marburg.de>",
     "Johannes Drönner <droenner@mathematik.uni-marburg.de>",
@@ -156,7 +156,8 @@ pwhash = "1.0"
 quote = "1.0"
 rand = "0.9"
 rayon = "1.8"
-reqwest = { version = "0.12", features = [ # must be compatible with `oauth2`
+regex = "1.11"
+reqwest = { version = "0.12", features = [
     "json",
     "multipart",
     "stream",

Cargo.toml

@@ -34,8 +34,6 @@ password = "geoengine"
 clear_database_on_start = false
 
 [raster.tiling_specification]
-origin_coordinate_x = 0.0
-origin_coordinate_y = 0.0
 tile_shape_pixels_x = 512
 tile_shape_pixels_y = 512
 
@@ -84,6 +82,8 @@ tile_limit = 4
 
 [wms]
 # request_timeout_seconds = 3600
+max_image_width = 4096
+max_image_height = 4096
 
 [wfs]
 # request_timeout_seconds = 3600

Settings-default.toml

@@ -10,8 +10,6 @@ password = "geoengine"
 raster_data_root_path = "../test_data/raster" # relative to sub crate directory for tests
 
 [raster.tiling_specification]
-origin_coordinate_x = 0.0
-origin_coordinate_y = 0.0
 tile_shape_pixels_x = 512
 tile_shape_pixels_y = 512
 

Settings-test.toml

@@ -18,6 +18,7 @@ arrow-schema = { workspace = true }
 bytes = { workspace = true }
 chrono = { workspace = true }
 fallible-iterator = { workspace = true }
+futures = { workspace = true }
 float-cmp = { workspace = true }
 gdal = { workspace = true }
 geo = { workspace = true }

datatypes/Cargo.toml

@@ -104,6 +104,11 @@ pub enum Error {
         i2: TimeInterval,
     },
 
+    #[snafu(display("Time step must be greater than zero, got: {}", step))]
+    TimeStepStepMustBeGreaterThanZero {
+        step: u32,
+    },
+
     #[snafu(display(
         "{} must be larger than {} and {} must be smaller than {}",
         start.inner(),
@@ -294,7 +299,7 @@ pub enum Error {
     WrongMetadataType,
 
     #[snafu(display(
-        "The conditions ul.x < lr.x && ul.y < lr.y are not met by ul:{} lr:{}",
+        "The conditions ul.x < lr.x && ul.y > lr.y are not met by ul:{} lr:{}",
         upper_left_coordinate,
         lower_right_coordinate
     ))]
@@ -345,6 +350,8 @@ pub enum Error {
     DuplicateBandInQueryBandSelection,
     QueryBandSelectionMustNotBeEmpty,
 
+    TilingGeoTransformOriginCoordinateMismatch,
+    TilingGeoTransformResolutionMissmatch,
     #[snafu(display("Invalid number of suffixes, expected {} found {}", expected, found))]
     InvalidNumberOfSuffixes {
         expected: usize,
@@ -363,6 +370,11 @@ pub enum Error {
         expected: usize,
         found: usize,
     },
+    NoIntersectionWithTargetProjection {
+        srs_in: SpatialReference,
+        srs_out: SpatialReference,
+        bounds: BoundingBox2D,
+    },
 }
 
 impl From<arrow::error::ArrowError> for Error {

datatypes/src/error.rs

@@ -46,6 +46,16 @@ impl From<Colorizer> for RasterColorizer {
     }
 }
 
+impl RasterColorizer {
+    /// Returns the no data color of this raster colorizer
+    pub fn no_data_color(&self) -> RgbaColor {
+        match self {
+            RasterColorizer::SingleBand { band_colorizer, .. } => band_colorizer.no_data_color(),
+            RasterColorizer::MultiBand { rgb_params, .. } => rgb_params.no_data_color,
+        }
+    }
+}
+
 /// The parameters for the RGBA colorizer
 #[derive(Copy, Clone, Debug, Deserialize, Serialize, PartialEq)]
 #[serde(rename_all = "camelCase")]

datatypes/src/operations/image/colorizer.rs

@@ -8,4 +8,4 @@ pub use colorizer::{
 };
 pub use into_lossy::LossyInto;
 pub use rgba_transmutable::RgbaTransmutable;
-pub use to_png::ToPng;
+pub use to_png::{ToPng, create_empty_no_data_color_png_bytes};

datatypes/src/operations/image/mod.rs

@@ -34,6 +34,15 @@ pub trait ToPng {
     ) -> Result<Vec<u8>>;
 }
 
+pub fn create_empty_no_data_color_png_bytes(
+    width: u32,
+    height: u32,
+    no_data_color: RgbaColor,
+) -> Result<Vec<u8>> {
+    let image_buffer = ImageBuffer::from_pixel(width, height, no_data_color.into());
+    image_buffer_to_png_bytes(image_buffer)
+}
+
 fn image_buffer_to_png_bytes(
     image_buffer: ImageBuffer<image::Rgba<u8>, Vec<u8>>,
 ) -> Result<Vec<u8>> {
@@ -121,9 +130,7 @@ where
     ) -> Result<Vec<u8>> {
         // TODO: use PNG color palette once it is available
 
-        let image_buffer = ImageBuffer::from_pixel(width, height, no_data_color.into());
-
-        image_buffer_to_png_bytes(image_buffer)
+        create_empty_no_data_color_png_bytes(width, height, no_data_color)
     }
 }