icg: partial integration of data provider requirements

Author: JanssenBrm

guides/file_formats.qmd

@@ -33,14 +33,19 @@ challenging to use.
 An exception to this are the 'RGB' style products, where three bands are used to represent a single image. In this case,
 creating a Cloud Optimised GeoTIFF with three bands is an option.
 
-For associating time information, create one GeoTIFF per timestamp, and one STAC item per timestamp. The GeoTIFF format has
-not built-in support for conveying time information, but STAC metadata is supporting this very well.
+For associating time information, create one GeoTIFF per timestamp, and one STAC item per timestamp. The GeoTIFF format
+has not built-in support for conveying time information, but STAC metadata is supporting this very well.
 
 ### Visualisation in APEx Geospatial Explorer
 
-To optimise visualisation in the APEx Geospatial Explorer, additional guidelines have been established. Adhering to these
-guidelines will ensure that the data is effectively optimised for visualisation on a map. Please refer to
-[this page](../interoperability/geospatial_explorer.qmd#cloud-optimized-geotiff-cog) for more information.
+To optimise visualisation in the APEx Geospatial Explorer, it is recommended to use the GoogleMapsCompatible tiling scheme-
+typically 256x256 pixel tiles aligned to a global grid. The default Coordinate Reference System (CRS) used in the Geospatial
+Explorer is Web Mercator projection (EPSG:3857) and therefore all datasets in this projection will be supported. On the
+fly reprojection and / or configuration of a Geospatial Explorer instance to alternative CRS’s is feasible, although we
+advise contact the APEx team for specific advice when using alternative projections. The BitsPerSample field must accurately
+reflect the data format. Overviews are essential for performance and should be generated using downsampling by factors of
+two until the image dimensions are the size of a tile or smaller. These overviews should also be tiled and placed after
+the main image data to conform with the COG specification.
 
 ## (Geo-)Zarr
 
@@ -61,6 +66,44 @@ At the time of writing, there are, however these important caveats:
 
 ## NetCDF
 
-NetCDF is a self-describing format with some properties similar to Zarr, but less optimised for cloud access. It can be useful
-for exchanging data cubes as single files through traditional methods. However, it is less recommended for convenient
+NetCDF is a self-describing format with some properties similar to Zarr, but less optimised for cloud access. It can be
+useful for exchanging data cubes as single files through traditional methods. However, it is less recommended for convenient
 sharing of large datasets, for which either COG or Zarr provide better options.
+
+## Statistical Datasets (FlatGeobuf, GeoJSON)
+
+Statistical datasets can be used to store precomputed statistics for dataset variables based on spatial units, such as
+administrative areas. An example is to collect land cover statistics on using boundaries from nomenclature of territorial
+units for statistics (NUTS), as shown in the [APEx Geospatial Explorer](https://explorer.apex.esa.int/) (Statistics). The
+guidelines in this section are focused on supporting the integration of statistical data for visualisation in the APEx
+Geospatial Explorer.
+
+The statistical datasets are expected to be vector layers that are provided in a format that can be parsed to a feature
+collection following the GeoJSON [@geojson] specification. Currently tested and supported formats are GeoJSON [@geojson]
+and FlatGeobuf [@flatgeobuf]. FlatGeobuf should be used where the statistical data is a large size as this allows for
+streaming of the relevant features without having to download the full dataset, increasing performance.
+
+The metadata header of the file should contain the following properties to define which fields on the features in the
+dataset should be used for the following purposes.
+
+- identifierKey: The name of the field that stores the unique identifier for each feature.
+- nameKey: The name of the field that stores the human-readable name for display.
+- levelKey: The name of the field that stores the administrative level number.
+- childrenKey: The name of the field that has a comma-separated list of child feature IDs as declared in identifierKey.
+Can be the empty string if this is the bottom level.
+- attributeKeys: A comma-separated list of field numbers that store the statistical data.
+- units: The units as displayed in the UI. This is for UI purposes only and has no effect on the data.
+- visualization_hint: A string of histogram, categorised, or continuous used as a hint to the UI to choose a suitable
+presentation for the data.
+
+For example, properties in the file metadata that is defined as follows:
+
+- identifierKey: NUTS_ID
+- nameKey: NUTS_NAME
+- levelKey: LEVL_CODE
+- childrenKey: children
+- attributeKeys: Trees, Shrubland, Grassland
+- visualization_hint: categorised
+
+would use the fields NUTS_ID, NUTS_NAME, … in the data to determine the navigation and display of statistics in the
+Geospatial Explorer.  For further guidance, please contact the APEx team through the [APEx User Forum](http://forum.apex.esa.int/).

interoperability/datahosting.md

@@ -44,7 +44,7 @@ fostering wider adoption and enabling advanced use cases in downstream applicati
     <tr>
       <td>DATA-REQ-03</td>
       <td>EO project results should be accompanied with metadata in a STAC [@stac] format, including applicable STAC extensions.</td>
-      <td>The specific STAC profiles will align with the recommendations provided by the <a href="https://eoresults.esa.int/reg-api/docs#/Implemented%20tranaction%20operations%3A/collection_items_post_request_collections__collectionId__items_post">ESA Project Results Repository (PRR)</a>. More details regarding which profiles to apply will be added as the project progresses.</td>
+      <td>The specific STAC profiles will align with the recommendations will align with the recommendations provided in the [Metadata Recommendations](#metadata-recommendations) section.</td>
     </tr>
   </tbody>
 </table>
@@ -56,3 +56,42 @@ Table: Interoperability requirements for data providers
 For more details regarding the recommended file formats and their usage within APEx, please refer to the
 [APEx File Format Recommendations](../guides/file_formats.qmd).
 :::
+
+
+## Metadata Recommendations
+
+### Format Specific Recommendations
+
+When sharing geospatial datasets in cloud-optimised formats, such as Cloud Optimised GeoTIFF (COG), NetCDF, and Zarr, it
+is essential to embed as much relevant metadata as possible directly within the files. Although these formats are designed
+for efficient cloud access, their interoperability potential is enhanced when the files carry rich, standardised metadata
+aligned with their respective specifications. Doing so not only improves data reuse by third-party tools but also enables
+more reliable automatic inference of STAC metadata during cataloguing or dataset publication.
+
+APEx recommends that the following details be incorporated into the file metadata:
+
+- The projection system used to present the data within the file
+- he Nodata value applied
+- The unit of measurement for values represented in the dataset
+- A definition of the colour map or legend utilised for the dataset visualisation in case of categorical data.
+- Band or variable names and descriptions
+
+For more details and examples on adding this additional metadata to your results, please consult the specific tools
+(e.g. gdal, rasterio, …) for generating the results.
+
+### STAC Metadata Recommendations
+
+The STAC specification provides a comprehensive and interoperable framework for describing geospatial datasets. Within
+APEx, STAC serves as the foundation to enhance the discoverability, interoperability, and integration of data across a
+range of platforms, data catalogues, including the ESA Project Results Repository, and tools such as the APEx Geospatial
+Explorer.
+
+To enhance interoperability, data providers are advised to consistently use a recommended set of STAC-related extensions
+and best practices. These recommendations come from community input and collaboration with other initiatives, like
+EarthCODE and EOEPCA, to ensure consistency across projects and promote the adoption of best practices.
+
+@tbl-metadata offers a summary of the suggested metadata. For further details, please refer to the resources listed below.
+
+- [STAC Best Practices](https://github.com/radiantearth/stac-best-practices/blob/main/README.md)
+- [EOEPCA+ Datacube Access Best Practices](https://github.com/EOEPCA/datacube-access/blob/main/best_practices/stac_best_practices.md)
+- [ESA PRR Collection Specifications](https://eoresults.esa.int/prr_collection_specifications.html)