Invalid Context in JSON-LD and GeoJSON Outputs

[ozmorph]

Describe the Bug

1. The @context for a subset of API endpoints return 404/Invalid URLs in application/geo+json and application/json+ld outputs.

404/Invalid URLs in application/geo+json output:

• https://raw.githubusercontent.com/geojson/geojson-ld/master/contexts/geojson-base.jsonld
• https://api.weather.gov/ontology#

404/Invalid URLs in application/json+ld output:

• https://api.weather.gov/ontology#

2. For some cases where the JSON-LD @context can be successfully processed, many of the definitions are using non GeoJSON/JSON-LD vocab/definition files.

To Reproduce

The following endpoints display both URLs in application/geo+json output. Correlation IDs added for each:

https://api.weather.gov/alerts | cf06b385-566b-41a1-8fcb-cbb55faf843a
https://api.weather.gov/alerts/active | d5847b86-12cf-4752-b5a3-94454ccf7274
https://api.weather.gov/alerts/{id} | d09dea20-5ece-4a81-b94d-b19dfb0a5214
https://api.weather.gov/alerts/active/zone/{zoneId} | d7c80873-a338-4f4c-94f0-489567162179
https://api.weather.gov/alerts/active/area/{area} | 9e65d559-2d54-4fb4-8e6d-773c61f93b08
https://api.weather.gov/gridpoints/{wfo}/{x},{y} | 7a54f51f-7e53-4162-a0cc-3513e1e922da
https://api.weather.gov/gridpoints/{wfo}/{x},{y}/forecast | 7cee1501-ea66-4116-b1e7-210271081d76
https://api.weather.gov/gridpoints/{wfo}/{x},{y}/forecast/hourly | a70cc4ad-0863-4fa9-b373-818b960f99da
https://api.weather.gov/gridpoints/{wfo}/{x},{y}/stations | 53866285-aeb1-4973-9efa-7b364ffd63d9
https://api.weather.gov/stations/{stationId}/observations | 02bcc98c-253e-4116-bbfe-acb0961c3a5d
https://api.weather.gov/stations/{stationId}/observations/latest | b3fb5ebb-5cfb-4268-ae00-6f27dbf7ccd7
https://api.weather.gov/stations/{stationId}/observations/{time} | e623c45f-ffcc-4aaa-b399-deae4667e854
https://api.weather.gov/stations/{stationId} | e6a3f7ea-c164-4b55-be41-71947708c0ce
https://api.weather.gov/points/{point} | ceb3dea0-fbef-4be6-9a29-c1c7eed6b63a
https://api.weather.gov/points/{point}/stations | 53866285-aeb1-4973-9efa-7b364ffd63d9
https://api.weather.gov/zones | 503ac21b-3fde-4988-a14d-390366b826ef
https://api.weather.gov/zones/{type} | 96cc5ac3-0aef-44aa-974b-ea1c47ffe4f6
https://api.weather.gov/zones/{type}/{zoneId} | ecda9657-c986-4e14-afad-572be1d75434
https://api.weather.gov/zones/forecast/{zoneId} | ed954421-973c-446d-99a2-a23499e257cb
https://api.weather.gov/zones/forecast/{zoneId}/observations | 2f16be4d-a8d4-428c-8fe3-9ea727f28a6e
https://api.weather.gov/zones/forecast/{zoneId}/stations | 09f9c9d7-ebf0-4cd0-be2f-6671c97fa275

The following endpoints display only https://api.weather.gov/ontology# in application/geo+json output. Correlation IDs added for each:

https://api.weather.gov/products | f4efe766-8e7c-4d01-80f7-09c5a9c06520
https://api.weather.gov/products/types | 7bac6da0-9ff6-4316-a8c8-738b53ac277b
https://api.weather.gov/products/{productId} | e4c60973-eb53-425e-a30c-ed2f66532a71
https://api.weather.gov/products/types/{typeId} | f2d6134a-81e8-46dc-a1b2-6fc7042afc8f
https://api.weather.gov/products/locations/{locationId}/types | 7ab92dda-f4df-4777-84c4-0505461beba0
https://api.weather.gov/products/types/{typeId}/locations/{locationId} | 45628f39-5b0f-4df8-ac19-9c857737f04f
https://api.weather.gov/zones/{type}/{zoneId}/forecast | d748002e-75fc-4e75-b10d-e177ab341263

Expected behavior

The URL https://raw.githubusercontent.com/geojson/geojson-ld/master/contexts/geojson-base.jsonld no longer exists. This causes JSON-LD processor libraries like PyLD to ignore GeoJSON definitions for specific elements such as "Polygon, Point, coordinates, geometry, etc." except in cases where a manual association has been made in the @context. For example from an API call to https://api.weather.gov/stations/{stationId} for application/geo+json:

{
"@context": [
    {
        ...
        "s": "https://schema.org/",
        "geo": "http://www.opengis.net/ont/geosparql#',
        ...
        "geometry": {
            "@id": "s:GeoCoordinates",
            "@type": "geo:wktLiteral"
        },
        ...
    }
],
...
}

which parses into

https://schema.org/GeoCoordinates
http://www.opengis.net/ont/geosparql#wktLiteral

These conflict with GeoJSON's definitions. When making the API call to the same endpoint for application/ld+json these manual schema definitions make sense but they should not be there in GeoJSON output.

The authoritative JSON-LD context for GeoJSON is now located at https://geojson.org/geojson-ld/geojson-context.jsonld according to https://geojson.org/geojson-ld .

Therefore all references to https://raw.githubusercontent.com/geojson/geojson-ld/master/contexts/geojson-base.jsonld should be rewritten to https://geojson.org/geojson-ld/geojson-context.jsonld

In addition, it appears that https://api.weather.gov/ontology# endpoint does not exist. I have found no references to this endpoint on Google. Is this intended or has this endpoint not been created/exposed yet to the public?

According to the Overview on https://www.weather.gov/documentation/services-web-api# the choice to move to JSON-LD was to promote machine data discovery, however I have noticed somewhat large gaps in the support each API endpoint has with regard to JSON-LD (I can make this a separate issue if needed/wanted). Is the intention to slowly add more context and better definitions over time or is this feature being put on the backlog? Without the availability of the https://api.weather.gov/ontology# endpoint or a public JSONSchema that we can verify/test data with, I feel that most of the data exposed from the API as a whole is useless for automatic Search Engine Optimization.

Environment
I'm using a Firefox 74.0.1 browser and make requests by manually typing in the API call into the URL. I use PyLD to test whether the JSON-LD or GeoJSON output can be processed.

[StephenClouse]

We are aware of the current issues with the contexts being incorrect.

Our main internal debate right now is whether it makes sense to continue supporting JSON-LD, since as you noted, we have not gotten the cycles to make it right, and almost everything has priority over this at the moment.

Lately we have been focused more on the OpenAPI definition for API and making it complete, including full schemas and examples for responses. There is a lot of overlap between these and once that is done we're not sure JSON-LD will still be adding any real value.

We also don't feel like JSON-LD has really caught on broadly like we expected it to when we started the API project several years ago. It has certainly found uses, but not in the scientific data realm.

We certainly welcome your feedback on this. But as far as fixing what's there now, based on our backlog this is unlikely to happen in 2020.

[ozmorph]

Thanks for the quick response! I'm glad I asked before spending too much time with the JSON-LD specifics in my own code.

The internal debate makes sense, something has to fall off the wagon and I personally think that OpenAPI is a much better priority/choice over JSON-LD. I personally hadn't given much thought to JSON-LD until I came across the weather.gov API.

Thank you all for the terrific API. Ever since DarkSky announced it was shutting down its API, I've enjoyed exploring the data that weather.gov provides. If you would like a beta tester for any additions to the current OpenAPI definition you linked, I would be happy to throw some calls against it.

Cheers!

[ozmorph]

Last question, even if the team chooses to drop JSON-LD support do you have a feeling whether GeoJSON will still be the preferred output format (without the @context specifics of course)?

Thanks.

[StephenClouse]

Yes, GeoJSON is not going anywhere :)

[ozmorph]

Wonderful!

I looked at the openapi.json file you linked and thought it would be cool to use it to programmatically generate API clients for my project.

As I was looking through the specification, I noticed a lot of duplication of parameters and schemas and broke those out into respective component fields. In addition, I took the liberty to also include a few more regex format codes for the fields that are pretty standardized. I'm fairly certain that this file still has the same functionality to the one you linked.

https://gist.github.com/ozmorph/35b5b73551565d6cad5dc0b749d41531

Please take and use it if you want, there's no license attached to it whatsoever.

In the following days, I plan to start filling out the Response objects with schemas for the API endpoints that I'm pulling from to help validate the data I'm getting back from the API. I would be more than happy to share back my work if there's a good place I can share it. If the team wants to put the production openapi.json file in this project on GitHub, I could fork and then issue pull requests for you all to review.

Thanks for the responses!

[StephenClouse]

Part of the reason for all the duplication is because the OpenAPI doc is not actually a static file; it is generated by the application from its routing and validation metadata. So what you see there is exactly what the application itself is checking for. But the system is not all that great and it doesn't attempt to de-duplicate anything.

Part of the work we're doing with fleshing out the OpenAPI doc is flipping that script, where the OpenAPI is static and the application generates its routing and validation from it. But we really want them to stay coupled because the only thing worse than no documentation is wrong documentation.

We have a lot of this done already but it's just waiting on the next release, so don't feel the need to expend personal effort on it.

[ozmorph]

Makes complete sense. I figured the file you linked must've been generated from the server code. I spent this time mostly to get reacquainted with OpenAPI 3.0, so no worries there.

I'm glad to hear the server routing and validation code will be generated from a static doc in the future! It will certainly allow client code that uses a generated API client to become more stable as well.

Do you have a rough estimate of when the new spec would release (weeks/months/year)?

Also, would there potentially be any room / a mechanism in the future for the community to contribute to the openapi.json spec (links to external documentation, schema examples, etc.), especially in areas that doesn't not directly affect the server code generation?

Thanks.

[StephenClouse]

Dates are one thing I can't really comment on, and even if I could, COVID-19 has sort of blown up everything schedule-wise. We have an already-finalized release in queue, and this change would be in the release following that. My hope would be in the second half of this year, if everything else cooperates.

Once we finish that flip I mentioned we could probably be a lot more open to accepting contributions to the spec from the user community. We may even just throw it on GitHub, who knows?

[ozmorph]

I understand completely. Thanks for the feedback, I look forward to seeing the new changes in the future.

I will close the issue since the original Bug Report will likely no longer be relevant within the next couple of releases.