-
Notifications
You must be signed in to change notification settings - Fork 0
2. E‐SOH Ingest
Until further notice all data that goes through E-SOH should be uploaded by a National Met Service (NMS).
A WIGOS-ID is a mandatory property of station data within E-SOH. If a WIGOS-ID is not provided by the data supplier, E-SOH will give an unofficial WIGOS-ID. This unofficial WIGOS-ID will be built by the system as 0-[nationalcode]-0-[nationalid]. If a valid WIGOS-ID is found, this will be used in the downstream process. Even if it is verified and present in E-SOH, and the public MQTT stream, stations without a valid WIGOS-ID, that is registered in the OSCAR database, will not be validated by WIS 2.0.
All parameters need to be written as element names following the CF convention. WMO compliant Meteorological parameters in BUFR files will be converted by the system before ingested to the database.
Access control is achieved by whitelisting the ingest API endpoint. This is done by the E-SOH management team. For your NMS to be whitelisted, please send your public IP (preferably a test server, and one where your system operationally be running), the name of your NMS and the name of your technical contact point to support-eumetnet-esoh@met.no. You will be alerted when whitelisting has been implmented.
https://observations.meteogate.eu/ingest/docs is the API reference documentation for uploading observations.
E-SOH ingest supports two different formats.
- BUFR: The widely supported BUFR format and
- JSON: A newly developed JSON format that is aligned with the JSON formatting used in WIS2.0, for publishing messages via MQTT. You should choose the format that best fits your needs and current standards.
The JSON formatting used in this step is reusing the information sent to WIS2.0. It contains relevant metadata for making efficient requests through the API. The work required is a bit more complex.
-
Build and validate your ingest string against the JSON schemas using https://observations.meteogate.eu/ingest/docs#/default/post_json_json_post
-
Build your system for uploading data.
-
Use a JSON validator to ensure that the json is valid.
-
Upload Use any standard tool for uploading your JSON structure. In the example below we use curl.
Both hourly observations as sent to the GTS and additional sub-hourly data can be uploaded using the same method. Expand the BUFR field in the API reference documentation, and press try it out.
If you want your message to be distributed to WIS 2.0. Confirm this by setting publishWIS2=true. Otherwise your data will only be visible through the open E-SOH OGC-API EDR api, and the public mqtt queue. Browse for a BUFR file on your on your computer and press "Execute"
- Upload example Use any standard tool for uploading your JSON structure. In this examples we use curl. This works.
curl -X 'POST' 'https://observations.meteogate.eu/ingest/bufr?publishWIS2=true' -H 'accept: application/json' -H 'Content-Type: multipart/form-data' -F 'files=@TEST.bufr'
The BUFR file will be uploaded, all the needed metadata will be extracted and stored in the storage back-end of E-SOH. This step is automatic in your production line. We do not recommend to upload all your files manually.
Your data will be published to WIS2.0, both as hourly CORE data and as single observations through RECOMMENDED
Centre id: eu-eumetnet-surface-observations
Topic: origin/a/wis2/eu-eumetnet-surface-observations/data/recommended/weather/surface-based-observations
Your ingested data is now ready to be requested by the OGC-API EDR https://observations.meteogate.eu/docs
Some countries (NMS) have already made data available on WIS 2.0. E-SOH has functionality that collects bufr files from WIS2.0 and ingests them to E-SOH. By using the flag publishWIS2=false, stops the redistribution of data back into WIS2.0 under the eu-eumentnet-surface-observations center-id.
Send your center-id and topic for the file you want to share, if this is the way a country wants to make observations available through E-SOH. Core data will be fetched from WIS2.0 or GTS (GTS redist), depending on the status of each country. As long as it is BUFR both core and recommended (sub-hourly observations) can be distributed by E-SOH.
By using this functionality all countries that share core data to GTS in Europe today will find their data in E-SOH.
Your data will be uploaded, all required metadata will be extracted and stored in the storage back-end of E-SOH. Ready to be requested by the OGC-API EDR. https://observations.meteogate.eu/docs
All questions concerning E-soh could be pointed to support-eumetnet-esoh@met.no