Add geoarrow.wkb Arrow extension metadata for geometry columns

[jatorre]

Summary

When Databricks tables contain geometry columns (e.g., geometry(4326)), the ADBC driver returns them as plain binary Arrow arrays without any extension metadata. This means downstream consumers like DuckDB cannot automatically recognize them as geometry.

Proposal: Tag geometry columns with geoarrow.wkb Arrow extension metadata so they flow as native geometry types through the Arrow ecosystem.

Background

GeoArrow defines standard Arrow extension types for geospatial data. The geoarrow.wkb extension type wraps WKB-encoded geometry in a binary Arrow array with two metadata fields:

• ARROW:extension:name = "geoarrow.wkb"
• ARROW:extension:metadata = JSON with CRS info (e.g., {"crs": "OGC:CRS84"})

DuckDB 1.5 has built-in GEOMETRY type support and can consume geoarrow.wkb Arrow arrays natively via register_geoarrow_extensions(). The DuckDB Snowflake extension already uses this pattern for geometry passthrough (iqea-ai/duckdb-snowflake#24).

Current workaround

When using the Databricks ADBC driver with DuckDB's adbc_scanner, geometry requires explicit WKB conversion on both sides:

-- Databricks side: explicitly convert to WKB
SELECT *, ST_AsBinary(geom) as geom_wkb FROM my_table

-- DuckDB side: explicitly convert back from WKB
SELECT ST_GeomFromWKB(geom_wkb) as geom FROM adbc_scan(...)

Proposed behavior

If the driver tagged geometry columns with geoarrow.wkb metadata:

-- Just works — geometry flows as native type
SELECT * FROM adbc_scan(...)

Implementation sketch

In ipc_reader_adapter.go, after obtaining the Arrow schema from the IPC stream:

1. Query Databricks column metadata (from INFORMATION_SCHEMA.COLUMNS or the Thrift response) to identify which columns have DATA_TYPE matching geometry(...)
2. For those columns, modify the Arrow schema field to include:
   • ARROW:extension:name = "geoarrow.wkb"
   • ARROW:extension:metadata = {"crs": {"type": "authority_code", "value": "OGC:CRS84"}} (or extract SRID from the type definition)
3. The underlying binary data is already WKB, so no data transformation is needed — just metadata annotation

Use case

We're benchmarking geospatial data transfer between DuckDB and cloud warehouses (duckdb-warehouse-transfer). The Databricks ADBC export via adbc_scanner already achieves ~24,000 rows/sec (2x faster than the @databricks/sql JSON connector). Adding geoarrow.wkb metadata would eliminate the WKB conversion overhead and enable native geometry passthrough.

References

• GeoArrow extension types spec
• DuckDB spatial GeoArrow support (PR #485)
• DuckDB Arrow extension registration (PR #15285)
• DuckDB Snowflake extension geometry support

[lidavidm]

In general, querying INFORMATION_SCHEMA.COLUMNS will not work (e.g. when the query is not a simple SELECT ... FROM table). Also, it is the Databricks side that generates the Arrow data, so we may need Databricks to assist here.

[jatorre]

Yeah, Just asked some people at Databrickls if they could add the metadata.

Meanwhile maybe it might make sense to add some patching on https://github.com/databricks/databricks-sql-go detecting that a column is geometry and adding the ARROW:extension:name = "geoarrow.wkb"? I think Redshift is the same right?

[lidavidm]

It all comes down to whether we get the metadata in the first place. Redshift gives us the metadata, but I don't know off the top of my head for Databricks. (Also AFAIK the protocol is not documented? It seems Databricks uses an extended version of the Hive Thrift protocol but the Arrow methods are not part of the original protocol and I don't recall seeing the IDL file anywhere, only generated clients. I'd love to be corrected though.)

[jatorre]

So here is my current understanding

In databricks-sql-go, USER_DEFINED_TYPE (geometry) maps to arrow.BinaryTypes.String and arrives as
EWKT text.

We might want Databricks Thrift server send WKB binary instead. Then the Go driver can map it to
arrow.BinaryTypes.Binary with geoarrow.wkb metadata, and the entire Arrow/GeoArrow ecosystem works natively.

[jatorre]

Yep, i have confirmed myself that this is what is happening. Converting from EWKT to WKB is an expensive operation so we really want Databrciks to change that on their side and then we map metadata.