Decide + document how layercake handles invalid geometries

[ianthetechie]

Relations are broken sometimes; it's just a fact of life in this data model (. We should decide what we want to do about (if anything) that and document it.

I don't know precisely what constraints shapely, WKB encoders, our geoparquet encoder, etc. impose, but I know that the PyOsmium reader will throw an exception in most cases when encountering an invalid geometry. I don't think anyone wants the geoparquet files to contain invalid geometries (e.g. non-closed, self-intersecting, etc.... I think WKB would theoretically allow this), but there are a few ways we could proceed I guess:

• Store null geometries (as in a proper null value; not a null island point), but keep all the other tags?
• Output a second set of files for rejects (e.g. boundaries-rejected.parquet) which has the same schema but doesn't have a geometry column (and is thus not actually "geo" parquet anymore)? We could optionally include a reason here too.

I think right now we probably just drop them, which is fine for some use cases, but I think other consumers would also like to know from a quality perspective what is currently broken. This could actually have other uses too in the community ;)

I lean toward the "reject file" option but happy to hear other ideas.

[marklit]

The rejects idea sounds good. It would be a shortlist of cleaning things up back upstream potentially or at least educating everyone on what to look out for.

I'm not sure if GEOMETRY and GEOGRAPHY models play into what is invalid as well. I hit errors a lot feeding data into BigQuery a few years back and I didn't have a lot of good answers for how to identify and remove invalid geometry beforehand. Much of it would render just fine in QGIS making things more complicated.

[ianthetechie]

Hey @marklit! Yeah exactly... a lot of software has some level of tolerance for invalid geometries, and some will transparently fix them.

I like the rejects file because, from day to day or whatever your frequency, things often get fixed or reverted. It's nice to know what you're missing as a data consumer so you can decide if you want to pull from a precious / later version, conflate from another source, etc. That's essentially what we do internally for some of our pipelines using Layercake already.

I guess I'll just work up a PR next month that does this and get feedback that way :)

[marklit]

Had a rejects file been available to me during my busiest days on BigQuery, I could have just generated a convex hull around those items and uploaded them with some sort of flag. I'm excited to see the code that can do this filtering.

This got posted to LinkedIn a while back, I'm not sure if it'll be of help when testing things out.