Skip to content

feat: GeoArrow export — GEOGRAPHY/GEOMETRY as geoarrow.wkb#100

Open
jatorre wants to merge 1 commit intoadbc-drivers:mainfrom
jatorre:geoarrow-export
Open

feat: GeoArrow export — GEOGRAPHY/GEOMETRY as geoarrow.wkb#100
jatorre wants to merge 1 commit intoadbc-drivers:mainfrom
jatorre:geoarrow-export

Conversation

@jatorre
Copy link

@jatorre jatorre commented Mar 18, 2026

Summary

Adds geospatial export support: GEOGRAPHY/GEOMETRY columns are automatically tagged with geoarrow.wkb Arrow extension metadata, enabling DuckDB and other Arrow consumers to receive native geometry types.

How it works

  1. WKB output format — Sets GEOGRAPHY_OUTPUT_FORMAT=WKB and GEOMETRY_OUTPUT_FORMAT=WKB at connection time so geo columns arrive as binary WKB (not GeoJSON strings)
  2. Geo column detection — Before executing a query, extracts the table name and runs DESCRIBE TABLE to identify GEOGRAPHY/GEOMETRY columns. Catalog metadata correctly reports the original type regardless of output format.
  3. geoarrow.wkb tagging — Tags identified columns in the Arrow schema with ARROW:extension:name = "geoarrow.wkb". GEOGRAPHY gets CRS EPSG:4326 (always WGS84). GEOMETRY gets no CRS (see TODOs).
  4. Zero conversion overhead — Data arrives as binary WKB and passes through unchanged (identCol).

Schema mapping

Snowflake type Arrow type Extension CRS
GEOGRAPHY Binary geoarrow.wkb EPSG:4326
GEOMETRY Binary geoarrow.wkb (empty — see TODO)

Why DESCRIBE TABLE?

Snowflake's REST API reports geo columns as "type": "binary" in rowtype response metadata when WKB output format is set, losing the original type information. Both rowtype and DESCRIBE RESULT are affected. Only DESCRIBE TABLE (catalog metadata) preserves the original type. We've reported this limitation to Snowflake.

TODOs

  1. GEOMETRY SRID — Requires inspecting data to determine SRID, which needs first-batch buffering. Same cross-driver challenge as adbc-drivers/redshift#2 and adbc-drivers/databricks#350. Fixed CRS for GEOGRAPHY (always 4326), empty CRS for GEOMETRY.

  2. Arbitrary queries — Currently only table scans (SELECT ... FROM table) trigger geo detection. Complex queries (joins, subqueries, CTEs) won't get geoarrow metadata. The data is still correct WKB, just without the Arrow extension type annotation.

Test results

End-to-end against real Snowflake:

  • GEOGRAPHY columns read as DuckDB native GEOMETRY('EPSG:4326')
  • Points, lines, polygons all correct ✓
  • GeoParquet export preserves GEOMETRY type with CRS ✓
  • Non-geo queries unaffected ✓

Related

🤖 Generated with Claude Code

Detect GEOGRAPHY/GEOMETRY columns during query execution and tag them
with geoarrow.wkb Arrow extension metadata, enabling DuckDB and other
Arrow consumers to receive native geometry types with CRS information.

How it works:
1. Set GEOGRAPHY/GEOMETRY_OUTPUT_FORMAT=WKB at connection time so geo
   columns arrive as binary WKB instead of GeoJSON strings
2. Before executing a query, extract the table name and run DESCRIBE
   TABLE to identify GEOGRAPHY/GEOMETRY columns (catalog metadata is
   unaffected by the WKB output format setting)
3. Tag identified columns with geoarrow.wkb extension metadata in the
   Arrow schema — GEOGRAPHY gets CRS "EPSG:4326", GEOMETRY gets no CRS
4. Data flows as binary WKB with zero conversion overhead

Note: Snowflake's REST API reports geo columns as "binary" in rowtype
metadata when WKB output format is set, losing the original type info.
This is why we need the separate DESCRIBE TABLE query. We've reported
this to Snowflake.

Limitations (documented as TODOs):
- GEOMETRY SRID: requires data inspection to determine, same cross-driver
  issue as adbc-drivers/redshift#2 and adbc-drivers/databricks#350
- Arbitrary queries: only table scans (SELECT ... FROM table) get geoarrow
  metadata. Complex queries with joins/subqueries don't trigger geo
  detection. The data is still correct WKB, just without the metadata.

Tested end-to-end: DuckDB reads Snowflake GEOGRAPHY as native GEOMETRY
with CRS EPSG:4326, and GeoParquet export preserves the type.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jatorre
Copy link
Author

jatorre commented Mar 18, 2026

cc @sfc-gh-obielov — We've reported to Snowflake that the REST API rowtype metadata loses the original column type (geography/geometry) when GEOGRAPHY_OUTPUT_FORMAT or GEOMETRY_OUTPUT_FORMAT is set to WKB/EWKB. Both rowtype and DESCRIBE RESULT report "binary" instead of the original type. Only DESCRIBE TABLE (catalog metadata) preserves it.

This is why this PR needs an extra DESCRIBE TABLE query before execution — to identify which columns are actually GEOGRAPHY/GEOMETRY. If Snowflake preserved the original type in rowtype metadata regardless of output format (or added a sourceType/databaseType field), the extra query would be unnecessary and we could support arbitrary queries, not just table scans.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant