Properties at top level vs. nested

[jcondit]

Hi!

I'm an engineer at Google working on Earth Engine and BigQuery. We're publishing some tables to BigQuery that contain raster metadata with a STAC-inspired schema. So far we've based the table schema on STAC itself, but we were considering adopting stac-geoparquet in order to be more consistent with external formats aimed at analytical databases. In particular, it'd be nice to be able to import/export easily between BigQuery image tables and stac-geoparquet files.

My main concern is that stac-geoparquet puts the properties at top level, which creates some awkward conflicts; for example, we've already encountered an asset with an "id" property that would be disallowed (at least with its current name) due to a conflict with the main "id" field. And for BigQuery at least, there's no benefit in putting the properties at top level vs. nested within the "properties" field -- they're easily queryable in both schema locations.

Can you tell me a bit more about why you chose to move the properties at top level? Is it necessary for some query engines? Is there any possibility that a future version of stac-geoparquet might move them back inside a "properties" object, or is stac-geoparquet fully committed to keeping them at top level?

Thanks!
Jeremy

[TomAugspurger]

Hi Jeremy, thanks for raising this.

Do you have a sample item / items that's potentially causing issues, just to make sure we're on the same page? In particular,

My main concern is that stac-geoparquet puts the properties at top level, which creates some awkward conflicts

We do pull the item properties up to the top level, but not the properties / extra fields of each asset. So it's fine to have an id property on an asset, but not an id property on an item. I don't recall seeing any STAC items in the wild with an id field under properties, but I don't think the spec explicitly disallows it.

I'm noticing now that the stac-geoparquet docs don't have a simple example that shows a basic JSON item and the equivalent stac-geoparquet, just this more thorough and complex NAIP example. I'll put up a PR to fix that.

Can you tell me a bit more about why you chose to move the properties at top level?

I thought this was discussed somewhere, but I wasn't able to find references to it in the repo. Perhaps it was somewhere else. My hazy recollection is that I was following geopandas' GeoDataFrame.from_features, which does pull the properties up to the top level. I'm not sure if that follows an interface or (maybe more likely) was a workaround for pandas' lack of support for nested / struct dtypes.

Is there any possibility that a future version of stac-geoparquet might move them back inside a "properties" object, or is stac-geoparquet fully committed to keeping them at top level?

I wouldn't rule it out entirely if we discovered a meaningful class of items that can't be represented as stac-geoparquet. I'm hopeful this isn't an issue though.

[gadomski]

@TomAugspurger as an aside, I've been using this slide as a visual example of the difference between the two, in case that's of any use for ya.

[jcondit]

Thanks for the quick reply, Tom!

The specific dataset I'm looking at is this one:
https://developers.google.com/earth-engine/datasets/catalog/JRC_CEMS_GLOFAS_FloodHazard_v1#image-properties

That's an Earth Engine ImageCollection, and we're trying to store the metadata for this collection as a stac-geoparquet table, with one row per image in the collection. However, each image in the collection has an "id" property (unrelated to the STAC "id" field), so the names collide when the "id" property is pulled to top level. (I'm definitely not a STAC expert, so if I'm doing this wrong, please let me know!)

That name conflict is not an insurmountable problem, since we can always rename the "id" property to something like "property_id" for the purpose of avoiding the conflict at top level, but now the stac-geoparquet schema is slightly different from the source schema, and the properties themselves have inconsistent names (some with prefix and some without). This issue made me wonder whether it would be possible to reinstate the "properties" struct to avoid such naming conflicts altogether.

Regarding the properties at top level, the official stac-geoparquet documentation only says that "We do pull the properties to the top level, so that it is easier to query and use them", but I didn't see any additional explanation, and to be honest it wasn't completely clear how pulling them to top level made them "easier to query". I was wondering whether there was a specific technical requirement behind the decision or just a matter of preference -- and if the latter, my own preference would be to preserve the nested "properties" field to (1) keep the schema tidy, (2) avoid name conflicts, and (3) avoid unnecessary divergence from STAC.

I understand the ship may have already sailed here, but if there's a chance that the standard could restore the "properties" struct at some point in the future, that would be interesting info for us when designing our own schemas.

Thanks again!
Jeremy

[TomAugspurger]

I'd be curious to hear others' thoughts, but IMO consistency with geopandas' behavior here is more desirable.

I'd also be curious whether there's appetite for a flag in the GeoParquet metadata that indicates whether properties are nested under a properties field or not. I suspect it's not worth it for users, but maybe others will disagree.

As a middle ground, we could try to specify the behavior writers should take when encountering a clash between a property key and one of the top-level keys (perhaps recording which keys clashed, what they were renamed to, and maybe even an algorithm to use for determining what new name to use).

[kylebarron]

I think that was part of his original question: Are we using/supporting tools that only support top-level column queries?

I.e. most Parquet-enabled databases would be able to push down queries into the properties struct column, so that if you only needed geometry, properties.id, and properties.datetime, that would be equivalent to how it works at the top level, and you wouldn't have to load and decode the entire properties column.

[gadomski]

Ah ok, thanks. I wasn't aware that you can push down into structs.