-
Notifications
You must be signed in to change notification settings - Fork 17
Add Met Office Global Deterministic height level dataset #346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 28 commits
Commits
Show all changes
53 commits
Select commit
Hold shift + click to select a range
a96726d
add global height level dataset
mo-jeff 5588033
add remaining dataset documentation for global and uk collections
mo-jeff acba9ac
Update datasets/met-office/collection/met-office-global-deterministic…
mo-jeff 21b9fbb
Update datasets/met-office/collection/met-office-uk-deterministic-hei…
mo-jeff 851c4ba
add template docs and description markdown for UKV Near Surface Colle…
mo-jeff aa54014
Merge branch 'met-office-datasets' of https://github.com/mo-jeff/plan…
mo-jeff dbb235a
Update datasets/met-office/collection/met-office-uk-deterministic-nea…
mo-jeff ebe9b91
Update datasets/met-office/collection/met-office-uk-deterministic-nea…
mo-jeff 6de0113
finish UKV and add Global Near Surface
mo-jeff 53300c6
Merge branch 'met-office-datasets' of https://github.com/mo-jeff/plan…
mo-jeff 325513f
refactor UK Near Surface to use datacube STAC extension & update mode…
mo-jeff 20691cf
update Global Near Surface with datacube extension & update description
mo-jeff c458572
tidy up Update Frequency docs for UK Collection
mo-jeff fa8446c
add Global collection documentation markdown
mo-jeff 0fd17ad
convert all Global collections to use datacube extension and add Glob…
mo-jeff 39ceaa9
update global height level to refactored STAC spec
mo-jeff d913837
update keyword for Global Height collection
mo-jeff 814bb2e
feat: add ingest code and workflow
gadomski 23b6c1b
Merge branch 'main' into met-office-datasets
gadomski cbcb376
refactor: remove `-level` from folders
gadomski 6e705af
updated uk and global height collections
mo-jeff fc12847
typo in uk height collection
mo-jeff 2040533
add CF standard name to UK Height collection assets
mo-jeff cbf7d30
refactor global pressure collection
mo-jeff 2176cbd
refactor UK Pressure Collection
mo-jeff 949116d
refactor UK and Gloal Pressure and Whole Atmosphere collections
mo-jeff 611347a
refactor near surface collections for UK and Global
mo-jeff aefb3f8
fix: move datacube to the collection level
gadomski f5b47bb
chore: add uk met office test
gadomski ef9ce87
Merge branch 'main' into met-office-datasets
gadomski 1168a40
fix: tests now work on real files
gadomski 4ca2177
fix: update citation
gadomski 557b5ac
docs: add comment the cleanup script
gadomski 57a1615
fix: update collection extents
gadomski fbfb03f
Ingestion fixes for production release
ghidalgo3 aa91a3d
deps: bump stactools to v0.4.0
gadomski e772067
Merge branch 'main' into met-office-datasets
gadomski 7bcaf7c
fix: splits
gadomski 492cd03
feat: add data access section
gadomski fd160fc
fix: urls
gadomski 555ff68
fix: pressure
gadomski 326083c
fix: add accidentally-removed description
gadomski e524c3b
fix: add note about data availability
gadomski 3cb1fec
fix: remove notice and data access header
gadomski 5c4b93f
script fixes
ghidalgo3 3e86de9
Upgrade all ingestion images
ghidalgo3 863f573
Merge branch 'met-office-datasets' of https://github.com/mo-jeff/plan…
ghidalgo3 b9b8230
update link to MPC Met Office collections in descriptions
mo-jeff 9e6a1be
Merge branch 'main' into met-office-datasets
ghidalgo3 e87b37e
add parquet to ukmo collections
ghidalgo3 1004c70
separate chunk storage
ghidalgo3 e212f93
chore: update stactools package version
gadomski b041354
fix ingestion workflows
ghidalgo3 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
53 changes: 53 additions & 0 deletions
53
...ets/met-office/collection/met-office-global-deterministic-height/description.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| This collection offers 1 parameter at 33 available height levels (5m to 6000m) from the Met Office global deterministic 10km forecast. This is a numerical weather prediction forecast for the whole globe, with a resolution of approximately 0.09 degrees i.e. 10km (2,560 x 1,920 grid points). | ||
|
|
||
| The data is available as NetCDF files. It's offered on a free, unsupported basis, so we don’t recommend using it for any critical business purposes. | ||
|
|
||
| ## Height Levels | ||
| Available height levels, in metres (m) above ground, are: | ||
| * 5.0 10.0 20.0 30.0 50.0 75.0 100.0 150.0 200.0 250.0 300.0 400.0 500.0 600.0 700.0 800.0 1000.0 1250.0 1500.0 1750.0 2000.0 2250.0 2500.0 2750.0 3000.0 3250.0 3500.0 3750.0 4000.0 4500.0 5000.0 5500.0 6000.0 | ||
|
|
||
| ## Timesteps | ||
| The following timesteps are available: | ||
| * every hour from 0 to 54 hours | ||
| * every 3 hours from 57 to 144 hours | ||
| * every 6 hours from 150 to 168 hours | ||
|
|
||
| ## Update Frequency | ||
| The model is run four times each day, with forecast reference times of 00:00, 06:00, 12:00 and 18:00 (UTC). | ||
|
|
||
| The runs at 00:00 and 12:00 provide data for the next 168 hours. The runs at 06:00 and 18:00 provide data for the next 67 hours. | ||
|
|
||
| The forecast reference time represents the nominal data time or start time of a model forecast run, rather than the time when the data is available. | ||
|
|
||
| ## Archive length and latency | ||
| As of December 2025, the archive contains data from December 2023 onwards. Forecasts will continue to be available for at least two years from their data date. | ||
|
|
||
| The data is typically available 6 hours after the model run time. | ||
|
|
||
| ## Technical specs | ||
| The data is available as NetCDF files. NetCDF (Network Common Data Form) is an interface for array-orientated data access and a library that supports the interface. It is composed of 3 components: | ||
| * Variables store the data | ||
| * Dimensions give relevant dimension information for the variables | ||
| * Attributes provide auxiliary information about the variables or dataset itself | ||
|
|
||
| NetCDF is used within the atmospheric and oceanic science communities and is network transparent, allowing for it to be accessed by computers that store integers, characters and floating-point numbers. | ||
|
|
||
| Iris supports NetCDF files through reading, writing and handling. Iris implements a model based on the CF conventions, giving a format-agnostic interface for working with data. | ||
|
|
||
| [Find further support on using Iris with NetCDF files.](https://scitools-iris.readthedocs.io/en/stable/) | ||
|
|
||
| ## Help us improve the data services we offer | ||
| [Join the Met Office research panel](https://forms.office.com/Pages/ResponsePage.aspx?id=YYHxF9cgRkeH_VD-PjtmGdxioYGoFbFIkZuB_q8Fb3VUQkoxRVQzTFdUMzNMVzczWVM5VTc3QTY3MC4u) to help us understand how people interact with weather and climate data, uncover challenges and explore opportunities. | ||
|
|
||
| ## How to cite | ||
| Met Office global deterministic 10km forecast was accessed on DATE from _insert Planetary Computer link_. | ||
|
|
||
| ## License | ||
| British Crown copyright 2023-2025, the Met Office, is licensed under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0/deed.en). | ||
|
|
||
| ## Providers | ||
| [Met Office](https://www.metoffice.gov.uk/) | ||
| See all datasets managed by [Met Office.](https://registry.opendata.aws/?search=managedBy:met%20office) | ||
|
|
||
| ## Contact | ||
| [servicedesk@metoffice.gov.uk](mailto:servicedesk@metoffice.gov.uk). Service desk is only available Mon – Fri, 09:00 until 17:00 UTC (-1 hour during BST). As a non-operational service we aim to respond to any service support enquiries within 3-5 business days. | ||
200 changes: 200 additions & 0 deletions
200
datasets/met-office/collection/met-office-global-deterministic-height/template.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,200 @@ | ||
| { | ||
| "id": "met-office-global-deterministic-height", | ||
| "type": "Collection", | ||
| "links": [ | ||
| { | ||
| "rel": "license", | ||
| "href": "https://creativecommons.org/licenses/by-sa/4.0/deed.en", | ||
| "type": "text/html", | ||
| "title": "Creative Commons Attribution-ShareAlike 4.0" | ||
| }, | ||
| { | ||
| "rel": "describedBy", | ||
| "href": "https://www.metoffice.gov.uk/services/data/external-data-channels", | ||
| "title": "Met Office Dataset Documentation" | ||
| } | ||
| ], | ||
| "title": "Height levels collection Met Office Global 10km deterministic weather forecast", | ||
| "assets": { | ||
| "thumbnail": { | ||
| "href": "https://ai4edatasetspublicassets.blob.core.windows.net/assets/pc_thumbnails/met-office-global-deterministic-height.jpg", | ||
| "type": "image/jpeg", | ||
| "roles": ["thumbnail"], | ||
| "title": "Height levels collection Met Office Global 10km deterministic weather forecast thumbnail" | ||
| } | ||
| }, | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [[-180, -90, 180, 90]] | ||
| }, | ||
| "temporal": { | ||
| "interval": [["2023-12-15T00:00:00Z", null]] | ||
| } | ||
| }, | ||
| "license": "proprietary", | ||
| "keywords": ["Met Office", "Weather", "Forecast", "Global", "Cloud"], | ||
| "providers": [ | ||
| { | ||
| "url": "https://www.metoffice.gov.uk/", | ||
| "name": "Met Office", | ||
| "roles": ["producer", "licensor", "processor"] | ||
| }, | ||
| { | ||
| "url": "https://planetarycomputer.microsoft.com", | ||
| "name": "Microsoft", | ||
| "roles": ["host", "processor"] | ||
| } | ||
| ], | ||
| "stac_version": "1.0.0", | ||
| "msft:group_id": "metoffice", | ||
| "msft:container": "staging", | ||
| "stac_extensions": [ | ||
| "https://stac-extensions.github.io/forecast/v0.2.0/schema.json", | ||
| "https://stac-extensions.github.io/datacube/v2.0.0/schema.json" | ||
| ], | ||
| "msft:storage_account": "ukmoeuwest", | ||
| "msft:short_description": "Met Office Global Deterministic Height Level", | ||
| "msft:region": "westeurope", | ||
| "item_assets": { | ||
| "cloud_amount_on_height_levels": { | ||
| "type": "application/netcdf", | ||
| "title": "Cloud amount on height levels", | ||
| "roles": ["data"], | ||
| "forecast:variable": "cloud_volume_fraction_in_atmosphere_layer" | ||
| } | ||
| }, | ||
| "description": "{{ collection.description }}", | ||
| "cube:dimensions": { | ||
| "latitude": { | ||
| "type": "spatial", | ||
| "axis": "y", | ||
| "extent": [-90, 90] | ||
| }, | ||
| "longitude": { | ||
| "type": "spatial", | ||
| "axis": "x", | ||
| "extent": [-180, 180] | ||
| }, | ||
| "height": { | ||
| "type": "spatial", | ||
| "values": [ | ||
| 5, 10, 20, 30, 50, 75, 100, 150, 200, 250, 300, 400, 500, 600, 700, 800, | ||
| 1000, 1250, 1500, 1750, 2000, 2250, 2500, 2750, 3000, 3250, 3500, 3750, | ||
| 4000, 4500, 5000, 5500, 6000 | ||
| ] | ||
| }, | ||
| "forecast:reference_datetime": { | ||
| "type": "temporal", | ||
| "extent": ["2023-12-15T00:00:00Z", null] | ||
| }, | ||
| "forecast:horizon": { | ||
| "type": "temporal", | ||
| "extent": [null, null], | ||
| "values": [ | ||
| "PT0H", | ||
| "PT1H", | ||
| "PT2H", | ||
| "PT3H", | ||
| "PT4H", | ||
| "PT5H", | ||
| "PT6H", | ||
| "PT7H", | ||
| "PT8H", | ||
| "PT9H", | ||
| "PT10H", | ||
| "PT11H", | ||
| "PT12H", | ||
| "PT13H", | ||
| "PT14H", | ||
| "PT15H", | ||
| "PT16H", | ||
| "PT17H", | ||
| "PT18H", | ||
| "PT19H", | ||
| "PT20H", | ||
| "PT21H", | ||
| "PT22H", | ||
| "PT23H", | ||
| "PT24H", | ||
| "PT25H", | ||
| "PT26H", | ||
| "PT27H", | ||
| "PT28H", | ||
| "PT29H", | ||
| "PT30H", | ||
| "PT31H", | ||
| "PT32H", | ||
| "PT33H", | ||
| "PT34H", | ||
| "PT35H", | ||
| "PT36H", | ||
| "PT37H", | ||
| "PT38H", | ||
| "PT39H", | ||
| "PT40H", | ||
| "PT41H", | ||
| "PT42H", | ||
| "PT43H", | ||
| "PT44H", | ||
| "PT45H", | ||
| "PT46H", | ||
| "PT47H", | ||
| "PT48H", | ||
| "PT49H", | ||
| "PT50H", | ||
| "PT51H", | ||
| "PT52H", | ||
| "PT53H", | ||
| "PT54H", | ||
| "PT57H", | ||
| "PT60H", | ||
| "PT63H", | ||
| "PT66H", | ||
| "PT69H", | ||
| "PT72H", | ||
| "PT75H", | ||
| "PT78H", | ||
| "PT81H", | ||
| "PT84H", | ||
| "PT87H", | ||
| "PT90H", | ||
| "PT93H", | ||
| "PT96H", | ||
| "PT99H", | ||
| "PT102H", | ||
| "PT105H", | ||
| "PT108H", | ||
| "PT111H", | ||
| "PT114H", | ||
| "PT117H", | ||
| "PT120H", | ||
| "PT123H", | ||
| "PT126H", | ||
| "PT129H", | ||
| "PT132H", | ||
| "PT135H", | ||
| "PT138H", | ||
| "PT141H", | ||
| "PT144H", | ||
| "PT150H", | ||
| "PT156H", | ||
| "PT162H", | ||
| "PT168H" | ||
| ] | ||
| } | ||
| }, | ||
| "cube:variables": { | ||
| "cloud_volume_fraction_in_atmosphere_layer": { | ||
| "description": "Fraction of horizontal grid square occupied by cloud in layers centred on height levels. The levels are height above ground.", | ||
| "dimensions": [ | ||
| "latitude", | ||
| "longitude", | ||
| "height", | ||
| "forecast:reference_datetime", | ||
| "forecast:horizon" | ||
| ], | ||
| "type": "data", | ||
| "unit": "1" | ||
| } | ||
| } | ||
| } |
86 changes: 86 additions & 0 deletions
86
...t-office/collection/met-office-global-deterministic-near-surface/description.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| This collection offers 48 parameters at near-surface level from the Met Office global deterministic 10km forecast. This is a numerical weather prediction forecast for the whole globe, with a resolution of approximately 0.09 degrees i.e. 10km (2,560 x 1,920 grid points). | ||
|
|
||
| The data is available as NetCDF files. It's offered on a free, unsupported basis, so we don’t recommend using it for any critical business purposes. | ||
|
|
||
| ## Data collection height | ||
| There are 3 forecast heights used within the near-surface this collection: | ||
| * Surface: the default collection height | ||
| * Screen level: 1.5m above the surface | ||
| * Wind parameters: 10m above the surface | ||
|
|
||
| ## Timesteps | ||
| For most parameters, the following time steps are available, see exceptions below: | ||
| * every hour from 0 to 54 hours | ||
| * every 3 hours from 57 to 144 hours | ||
| * every 6 hours from 150 to 168 hours | ||
|
|
||
| Exceptions (for `accumulation`, `min`, `max` and `mean` parameters): | ||
| * Height of orography (height_of_orography) is only available at 0H | ||
| * Hourly latent heat flux at surface mean (latent_heat_flux_at_surface_mean-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H latent heat flux at surface mean (latent_heat_flux_at_surface_mean-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H latent heat flux at surface mean (latent_heat_flux_at_surface_mean-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Hourly precipitation accumulation (precipitation_accumulation-PT01H) is only available every hour from 1 to 48 hours | ||
| * 3H precipitation accumulation (precipitation_accumulation-PT03H) is only available every three hours from 51 to 144 hours | ||
| * 6H precipitation accumulation (precipitation_accumulation-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Radiation flux in uv downward at surface (radiation_flux_in_uv_downward_at_surface) has three hourly timesteps from 0 to 144 hours | ||
| * Hourly rainfall accumulation (rainfall_accumulation-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H rainfall accumulation (rainfall_accumulation-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H rainfall accumulation (rainfall_accumulation-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Hourly rainfall rate from convection max (rainfall_rate_from_convection_max-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H rainfall rate from convection max (rainfall_rate_from_convection_max-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H rainfall rate from convection max (rainfall_rate_from_convection_max-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Hourly snowfall rate from convection max (snowfall_rate_from_convection_max-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H snowfall rate from convection max (snowfall_rate_from_convection_max-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H snowfall rate from convection max (snowfall_rate_from_convection_max-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Hourly snowfall rate from convection mean (snowfall_rate_from_convection_mean-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H snowfall rate from convection mean (snowfall_rate_from_convection_mean-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H snowfall rate from convection mean (snowfall_rate_from_convection_mean-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Hourly temperature at screen level max (temperature_at_screen_level_max-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H temperature at screen level max (temperature_at_screen_level_max-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H temperature at screen level max (temperature_at_screen_level_max-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Hourly temperature at screen level min (temperature_at_screen_level_min-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H remperature at screen level min (temperature_at_screen_level_min-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H temperature at screen level min (temperature_at_screen_level_min-PT06H) is only available every six hours from 150 to 168 hours | ||
| * Hourly wind gust at 10m max (wind_gust_at_10m_max-PT01H) is only available every hour from 1 to 54 hours | ||
| * 3H wind gust at 10m max (wind_gust_at_10m_max-PT03H) is only available every three hours from 57 to 144 hours | ||
| * 6H wind gust at 10m max (wind_gust_at_10m_max-PT06H) is only available every six hours from 150 to 168 hours | ||
|
|
||
| ## Update Frequency | ||
| The model is run four times each day, with forecast reference times of 00:00, 06:00, 12:00 and 18:00 (UTC). | ||
|
|
||
| The runs at 00:00 and 12:00 provide data for the next 168 hours. The runs at 06:00 and 18:00 provide data for the next 67 hours. | ||
|
|
||
| The forecast reference time represents the nominal data time or start time of a model forecast run, rather than the time when the data is available. | ||
|
|
||
| ## Archive length and latency | ||
| As of December 2025, the archive contains data from December 2023 onwards. Forecasts will continue to be available for at least two years from their data date. | ||
|
|
||
| The data is typically available 6 hours after the model run time. | ||
|
|
||
| ## Technical specs | ||
| The data is available as NetCDF files. NetCDF (Network Common Data Form) is an interface for array-orientated data access and a library that supports the interface. It is composed of 3 components: | ||
| * Variables store the data | ||
| * Dimensions give relevant dimension information for the variables | ||
| * Attributes provide auxiliary information about the variables or dataset itself | ||
|
|
||
| NetCDF is used within the atmospheric and oceanic science communities and is network transparent, allowing for it to be accessed by computers that store integers, characters and floating-point numbers. | ||
|
|
||
| Iris supports NetCDF files through reading, writing and handling. Iris implements a model based on the CF conventions, giving a format-agnostic interface for working with data. | ||
|
|
||
| [Find further support on using Iris with NetCDF files.](https://scitools-iris.readthedocs.io/en/stable/) | ||
|
|
||
| ## Help us improve the data services we offer | ||
| [Join the Met Office research panel](https://forms.office.com/Pages/ResponsePage.aspx?id=YYHxF9cgRkeH_VD-PjtmGdxioYGoFbFIkZuB_q8Fb3VUQkoxRVQzTFdUMzNMVzczWVM5VTc3QTY3MC4u) to help us understand how people interact with weather and climate data, uncover challenges and explore opportunities. | ||
|
|
||
| ## How to cite | ||
| Met Office global deterministic 10km forecast was accessed on DATE from _insert Planetary Computer link_. | ||
|
|
||
| ## License | ||
| British Crown copyright 2023-2025, the Met Office, is licensed under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0/deed.en). | ||
|
|
||
| ## Providers | ||
| [Met Office](https://www.metoffice.gov.uk/) | ||
| See all datasets managed by [Met Office.](https://registry.opendata.aws/?search=managedBy:met%20office) | ||
|
|
||
| ## Contact | ||
| [servicedesk@metoffice.gov.uk](mailto:servicedesk@metoffice.gov.uk). Service desk is only available Mon – Fri, 09:00 until 17:00 UTC (-1 hour during BST). As a non-operational service we aim to respond to any service support enquiries within 3-5 business days. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.