Update Sentinel-5P

datasets/io-land-cover/dataset.yaml

@@ -2,7 +2,7 @@ id: io_lulc
 image: ${{ args.registry }}/pctasks-task-base:latest
 
 args:
-- registry
+  - registry
 
 code:
   src: ${{ local.path(./io_lulc.py) }}

datasets/sentinel-5p/README.md

@@ -1 +1 @@
-# Work in progress - in a broken state
+# Sentinel-5 Precursor

datasets/sentinel-5p/collection/description.md

@@ -0,0 +1,19 @@
+The Copernicus [Sentinel-5 Precursor](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-5p) mission provides high spatio-temporal resolution measurements of the Earth's atmosphere. The mission consists of one satellite carrying the [TROPOspheric Monitoring Instrument](http://www.tropomi.eu/) (TROPOMI). The satellite flies in loose formation with NASA's [Suomi NPP](https://www.nasa.gov/mission_pages/NPP/main/index.html) spacecraft, allowing utilization of co-located cloud mask data provided by the [Visible Infrared Imaging Radiometer Suite](https://www.nesdis.noaa.gov/current-satellite-missions/currently-flying/joint-polar-satellite-system/visible-infrared-imaging) (VIIRS) instrument onboard Suomi NPP during processing of the TROPOMI methane product.
+
+The Sentinel-5 Precursor mission aims to reduce the global atmospheric data gap between the retired [ENVISAT](https://earth.esa.int/eogateway/missions/envisat) and [AURA](https://www.nasa.gov/mission_pages/aura/main/index.html) missions and the future [Sentinel-5](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-5) mission. Sentinel-5 Precursor [Level 2 data](http://www.tropomi.eu/data-products/level-2-products) provide total columns of ozone, sulfur dioxide, nitrogen dioxide, carbon monoxide and formaldehyde, tropospheric columns of ozone, vertical profiles of ozone and cloud & aerosol information. These measurements are used for improving air quality forecasts and monitoring the concentrations of atmospheric constituents.
+
+This STAC Collection provides Sentinel-5 Precursor Level 2 data, in NetCDF format, since April 2018 for the following products:
+
+* [`L2__AER_AI`](http://www.tropomi.eu/data-products/uv-aerosol-index): Ultraviolet aerosol index
+* [`L2__AER_LH`](http://www.tropomi.eu/data-products/aerosol-layer-height): Aerosol layer height
+* [`L2__CH4___`](http://www.tropomi.eu/data-products/methane): Methane (CH<sub>4</sub>) total column
+* [`L2__CLOUD_`](http://www.tropomi.eu/data-products/cloud): Cloud fraction, albedo, and top pressure
+* [`L2__CO____`](http://www.tropomi.eu/data-products/carbon-monoxide): Carbon monoxide (CO) total column
+* [`L2__HCHO__`](http://www.tropomi.eu/data-products/formaldehyde): Formaldehyde (HCHO) total column
+* [`L2__NO2___`](http://www.tropomi.eu/data-products/nitrogen-dioxide): Nitrogen dioxide (NO<sub>2</sub>) total column
+* [`L2__O3____`](http://www.tropomi.eu/data-products/total-ozone-column): Ozone (O<sub>3</sub>) total column
+* [`L2__O3_TCL`](http://www.tropomi.eu/data-products/tropospheric-ozone-column): Ozone (O<sub>3</sub>) tropospheric column
+* [`L2__SO2___`](http://www.tropomi.eu/data-products/sulphur-dioxide): Sulfur dioxide (SO<sub>2</sub>) total column
+* [`L2__NP_BD3`](http://www.tropomi.eu/data-products/auxiliary): Cloud from the Suomi NPP mission, band 3
+* [`L2__NP_BD6`](http://www.tropomi.eu/data-products/auxiliary): Cloud from the Suomi NPP mission, band 6
+* [`L2__NP_BD7`](http://www.tropomi.eu/data-products/auxiliary): Cloud from the Suomi NPP mission, band 7

datasets/sentinel-5p/collection/template.json

@@ -0,0 +1,237 @@
+{
+    "stac_version": "1.0.0",
+    "type": "Collection",
+    "id": "sentinel-5p-l2-netcdf",
+    "title": "Sentinel-5P Level-2",
+    "description": "{{ collection.description }}",
+    "license": "proprietary",
+    "links": [
+        {
+            "rel": "license",
+            "href": "https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice",
+            "type": "application/pdf",
+            "title": "Sentinel Data License"
+        },
+        {
+            "rel": "about",
+            "href": "https://sentinel.esa.int/web/sentinel/missions/sentinel-5p",
+            "type": "text/html",
+            "title": "Sentinel-5 Precursor Mission"
+        }
+    ],
+    "stac_extensions": [
+        "https://stac-extensions.github.io/sat/v1.0.0/schema.json",
+        "https://stac-extensions.github.io/table/v1.2.0/schema.json"
+    ],
+    "keywords": [
+        "ESA",
+        "Copernicus",
+        "Sentinel",
+        "Air Quality",
+        "Climate Change",
+        "Forecasting"
+    ],
+    "msft:short_description": "Sentinel-5P Level 2 atmospheric monitoring products in NetCDF format",
+    "msft:storage_account": "sentinel5euwest",
+    "msft:container": "sentinel-5p",
+    "msft:region": "westeurope",
+    "providers": [
+        {
+            "name": "ESA",
+            "roles": [
+                "producer",
+                "processor",
+                "licensor"
+            ],
+            "url": "https://earth.esa.int/web/guest/home"
+        },
+        {
+            "name": "Microsoft",
+            "roles": [
+                "host"
+            ],
+            "url": "https://planetarycomputer.microsoft.com"
+        }
+    ],
+    "assets": {
+        "thumbnail": {
+            "title": "Sentinel-5P Level-2 NetCDF",
+            "href": "https://ai4edatasetspublicassets.blob.core.windows.net/assets/pc_thumbnails/sentinel-5p-l2-netcdf-thumb.png",
+            "media_type": "image/png"
+        },
+        "geoparquet-items": {
+            "href": "abfs://items/io-biodiversity.parquet",
+            "title": "GeoParquet STAC items",
+            "description": "Snapshot of the collection's STAC items exported to GeoParquet format.",
+            "type": "application/x-parquet",
+            "roles": [
+                "stac-items"
+            ],
+            "table:storage_options": {
+                "account_name": "pcstacitems"
+            },
+            "msft:partition_info": {
+                "is_partitioned": true,
+                "partition_frequency": "MS"
+            }
+        }
+    },
+    "summaries": {
+        "constellation": [
+            "Sentinel-5P"
+        ],
+        "platform": [
+            "Sentinel 5 Precursor"
+        ],
+        "instruments": [
+            "TROPOMI"
+        ],
+        "sat:platform_international_designator": [
+            "2017-064A"
+        ],
+        "s5p:collection_identifier": [
+            "01",
+            "02",
+            "03"
+        ],
+        "s5p:processing_mode": [
+            "NRTI",
+            "OFFL",
+            "RPRO"
+        ],
+        "s5p:product_type": [
+            "L2__AER_AI",
+            "L2__AER_LH",
+            "L2__CH4___",
+            "L2__CLOUD_",
+            "L2__CO____",
+            "L2__HCHO__",
+            "L2__NO2___",
+            "L2__NP_BD3",
+            "L2__NP_BD6",
+            "L2__NP_BD7",
+            "L2__O3_TCL",
+            "L2__O3____",
+            "L2__SO2___"
+        ],
+        "s5p:spatial_resolution": [
+            "5.5x3.5km",
+            "5.5x7.0km",
+            "7.0x3.5km",
+            "7.0x7.0km"
+        ]
+    },
+    "extent": {
+        "spatial": {
+            "bbox": [
+                [
+                    -180,
+                    -90,
+                    180,
+                    90
+                ]
+            ]
+        },
+        "temporal": {
+            "interval": [
+                [
+                    "2018-04-30T00:18:50Z",
+                    null
+                ]
+            ]
+        }
+    },
+    "item_assets": {
+        "aer_ai": {
+            "title": "TROPOMI/S5P L2 Ultraviolet Aerosol Index",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "aer_lh": {
+            "title": "TROPOMI/S5P L2 Aerosol Layer Height",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "ch4": {
+            "title": "TROPOMI/S5P L2 Methane Total Column",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "cloud": {
+            "title": "TROPOMI/S5P L2 Cloud Fraction, Albedo, and Top Pressure",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "co": {
+            "title": "TROPOMI/S5P L2 Carbon Monoxide Total Column",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "hcho": {
+            "title": "TROPOMI/S5P L2 Formaldehyde Total Column",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "no2": {
+            "title": "TROPOMI/S5P L2 Nitrogen Dioxide Total Column",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "o3": {
+            "title": "TROPOMI/S5P L2 Ozone Total Column",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "o3_tcl": {
+            "title": "TROPOMI/S5P L2 Ozone Tropospheric Column",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "so2": {
+            "title": "TROPOMI/S5P L2 Sulphur Dioxide Total Column",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "np_bd3": {
+            "title": "TROPOMI/S5P VIIRS/NPP Band 3 Cloud Mask",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "np_bd6": {
+            "title": "TROPOMI/S5P VIIRS/NPP Band 6 Cloud Mask",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        },
+        "np_bd7": {
+            "title": "TROPOMI/S5P VIIRS/NPP Band 7 Cloud Maskt",
+            "type": "application/x-netcdf",
+            "roles": [
+                "data"
+            ]
+        }
+    }
+}

datasets/sentinel-5p/dataset.yaml

@@ -1,12 +1,27 @@
-id: sentinel-1-grd
+id: sentinel_5p
+image: ${{ args.registry }}/pctasks-task-base:latest
+
+args:
+  - registry
+
+code:
+  src: ${{ local.path(./sentinel_5p.py) }}
+
+environment:
+  AZURE_TENANT_ID: ${{ secrets.task-tenant-id }}
+  AZURE_CLIENT_ID: ${{ secrets.task-client-id }}
+  AZURE_CLIENT_SECRET: ${{ secrets.task-client-secret }}
+
 collections:
-  id: sentinel-1-grd
-  containers:
-    assets: sentinel1euwest/s1-grd
-    stac: sentinel1euwest/s1-grd-stac
-    etl-data: sentinel1euwest/s1-grd-etl-info
-triggers:
-  storage:
-    account: sentinel1euwest
-    container: s1-grd-stac
-    filter: "*.json"
+  - id: sentinel-5p-l2-netcdf
+    template: ${{ local.path(./collection) }}
+    class: sentinel_5p:Sentinel5pNetCDFCollection
+    asset_storage:
+      - uri: blob://sentinel5euwest/sentinel-5p-stac/TROPOMI/
+        token: ${{ pc.get_token(sentinel5euwest, sentinel-5p-stac) }}
+        chunks:
+          options:
+            extensions: [.json]
+            chunk_length: 5000 # Larger? There are 1.33 million blobs in storage
+    chunk_storage:
+      uri: blob://sentinel5euwest/sentinel-5p-etl-data/pctasks-chunks/