feat(python/sedonadb): Implement GDAL/OGR formats via pyogrio #283

paleolimbot · 2025-11-07T23:33:42Z

This PR implements GDAL/OGR format reading, which also required some infrastructure. I tried to put as much infrastructure as possible in Rust so that we can reuse it when we implement this in Rust for the CLI/maybe other bindings.

There were a few missing pieces I discovered here (extracting a bounding box from a filter expression, which in turn required the ability to intersect intervals/bounding boxes which we hadn't implemented yet. Anybody should feel free to say the word and I'm happy to PR these separately to help with review.

import sedona.db

sd = sedona.db.connect()
sd.options.interactive = True

url = "https://github.com/geoarrow/geoarrow-data/releases/download/v0.2.0/ns-water_water-line.fgb"
sd.read_pyogrio(url).to_view("water_line")

sd.view("water_line").head(5)

Some cool features we support here:

Bounding box pushdown: the (spatial) query filters are pushed into the datasource. We could potentially push some non-spatial filters but it's less clear if that would actually help (even for spatial filters we might be able to evaluate them faster than GDAL if there's no spatial index on the source or if it's a wide window because we can do it in parallel).
Projection pushdown: skip reading unused columns
https:// and/or .zipped sources using /vsicurl/ and /vsizip/, respectively.
Multi file support. You can read_pyogrio("some/dir/*.fgb") and it should work (and is tested).
Infrastructure to wire up other file formats via Python. We probably need to extend the implementation a bit to support other formats but it is mostly there. While we don't really want to implement the actual reader in Python (== holding the GIL while reading stuff), using Python to obtain the C ArrowArrayStream should be OK.

petern48 · 2025-11-09T05:11:46Z

Closes #137 I assume? Just creating a link

Copilot

Pull Request Overview

This PR implements support for reading spatial file formats using GDAL/OGR through pyogrio in the Python sedonadb package. The changes introduce a plugin architecture for external file formats and implement the first concrete format (pyogrio) that enables reading common geospatial formats like FlatGeoBuf, GeoPackage, and GeoJSON.

Key changes:

Added ExternalFormatSpec trait and infrastructure for pluggable file format support
Implemented PyogrioFormatSpec for GDAL/OGR format support via pyogrio
Added spatial filter pushdown capability with bounding box extraction
Implemented interval intersection methods for spatial filtering optimization

Reviewed Changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`rust/sedona/src/context.rs`	Added `read_external_format()` method and tests for external format reading
`rust/sedona-geometry/src/interval.rs`	Added `is_full()` and `intersection()` methods to interval traits
`rust/sedona-geometry/src/bounding_box.rs`	Added `intersection()` method for bounding box operations
`rust/sedona-expr/src/spatial_filter.rs`	Added `filter_bbox()` method and `Clone` trait for spatial filter pushdown
`rust/sedona-datasource/src/utility.rs`	New file implementing `ProjectedRecordBatchReader` for column projection
`rust/sedona-datasource/src/spec.rs`	Enhanced `Object` with URL reconstruction and display, added tests
`rust/sedona-datasource/src/format.rs`	Added schema inference validation for empty object lists
`python/sedonadb/src/datasource.rs`	New file implementing Python bindings for external format specs
`python/sedonadb/src/context.rs`	Added `read_external_format()` method to Python context
`python/sedonadb/python/sedonadb/datasource.py`	New file with `ExternalFormatSpec` and `PyogrioFormatSpec` implementations
`python/sedonadb/python/sedonadb/context.py`	Added `read_pyogrio()` public API method
`python/sedonadb/tests/test_datasource.py`	New test file for pyogrio format functionality
Various `Cargo.toml` files	Added dependencies for new functionality

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rust/sedona-geometry/src/interval.rs

rust/sedona-datasource/src/spec.rs

python/sedonadb/python/sedonadb/datasource.py

Co-authored-by: Copilot <[email protected]>

paleolimbot · 2025-11-19T16:25:17Z

rust/sedona-expr/src/spatial_filter.rs

+    /// Compute the maximum extent of a filter for a specific column index
+    ///
+    /// Some spatial file formats have the ability to push down a bounding box
+    /// into an index. This function allows deriving that bounding box based
+    /// on what DataFusion provides, which is a physical expression.
+    ///
+    /// Note that this always succeeds; however, for a non-spatial expression or
+    /// a non-spatial expression that is unsupported, the full bounding box is
+    /// returned.
+    pub fn filter_bbox(&self, column_index: usize) -> BoundingBox {
+        match self {
+            SpatialFilter::Intersects(column, bounding_box)
+            | SpatialFilter::Covers(column, bounding_box) => {
+                if column.index() == column_index {
+                    return bounding_box.clone();
+                }
+            }
+            SpatialFilter::And(lhs, rhs) => {
+                let lhs_box = lhs.filter_bbox(column_index);
+                let rhs_box = rhs.filter_bbox(column_index);
+                if let Ok(bounds) = lhs_box.intersection(&rhs_box) {
+                    return bounds;
+                }
+            }
+            SpatialFilter::Or(lhs, rhs) => {
+                let mut bounds = lhs.filter_bbox(column_index);
+                bounds.update_box(&rhs.filter_bbox(column_index));
+                return bounds;
+            }
+            SpatialFilter::LiteralFalse => {
+                return BoundingBox::xy(Interval::empty(), Interval::empty())
+            }
+            SpatialFilter::HasZ(_) | SpatialFilter::Unknown => {}
+        }
+
+        BoundingBox::xy(Interval::full(), Interval::full())
+    }


@Kontinuation can you sanity check this?

paleolimbot · 2025-11-19T16:28:47Z

rust/sedona-geometry/src/interval.rs

+    fn intersection(&self, other: &Self) -> Result<Self, SedonaGeometryError> {
+        match (self.is_wraparound(), other.is_wraparound()) {
+            // Neither is wraparound
+            (false, false) => Ok(self.inner.intersection(&other.inner)?.into()),
+            // One is wraparound
+            (true, false) => other.intersection(self),
+            (false, true) => {
+                let inner = self.inner;
+                let (left, right) = other.split();
+                match (inner.intersects_interval(&left), inner.intersects_interval(&right)) {
+                    // Intersects both the left and right intervals
+                    (true, true) => {
+                        Err(SedonaGeometryError::Invalid(format!("Can't represent the intersection of {self:?} and {other:?} as a single WraparoundInterval")))
+                    },
+                    // Intersects only the left interval
+                    (true, false) => Ok(inner.intersection(&left)?.into()),
+                    // Intersects only the right interval
+                    (false, true) => Ok(inner.intersection(&right)?.into()),
+                    (false, false) => Ok(WraparoundInterval::empty()),
+                }
+            }
+            // Both are wraparound
+            (true, true) => {
+                // Both wraparound intervals represent complements of excluded regions
+                // Intersection of complements = complement of union of excluded regions
+                // self excludes (hi, lo), other excludes (other.hi, other.lo)
+                // We need to find the union of these excluded regions
+                let excluded_self = Interval::new(self.inner.hi, self.inner.lo);
+                let excluded_other = Interval::new(other.inner.hi, other.inner.lo);
+
+                // We can't use the excluded union if the excluded region of self and other
+                // are disjoint
+                if excluded_self.intersects_interval(&excluded_other) {
+                    let excluded_union = excluded_self.merge_interval(&excluded_other);
+
+                    // The intersection is the complement of the union of excluded regions
+                    Ok(WraparoundInterval::new(
+                        excluded_union.hi(),
+                        excluded_union.lo(),
+                    ))
+                } else {
+                    Err(SedonaGeometryError::Invalid(format!("Can't represent the intersection of {self:?} and {other:?} as a single WraparoundInterval")))
+                }
+            }
+        }


This is also a bit of non-trivial math that could use a sanity check (the tests below have some ASCII art to help visualize the various cases)

paleolimbot added 5 commits November 7, 2025 17:04

first go

916ec46

python

5ccba21

string together more parts

bb465e8

fix for the example

c940e1a

rat

86bd2fa

paleolimbot added 24 commits November 11, 2025 14:47

first test

eaa09c1

towards projection

9058561

projecting?

deed0a9

format

a8106a0

test multi file

710d3d7

select a bunch of files

e775633

pass batch size through

5d1102d

start on the filter interface

ea97d28

start on filter bbox

ec5d638

implement interval intersection

2ae7c01

bounding box intersection

8673673

implement the intersection

239bbe8

oogh

7e31546

vsizip, identify error with test

824acd7

fix error for bad file

32b5b9a

document Python

41d4dc0

document Python datasource

64766b5

document py datasource

87ad44c

test object

83b3b08

add test

8d81a6d

test local filesystem

524f6c8

test projected reader

484a3d1

test spatial filter

033bfc8

document utility

173c3d3

paleolimbot added 3 commits November 19, 2025 01:12

test wraparound intersection

96fb28b

test bounding box intersection

4bab9ce

fix python tests

7666278

paleolimbot requested a review from Copilot November 19, 2025 15:14

Copilot AI reviewed Nov 19, 2025

View reviewed changes

rust/sedona-geometry/src/interval.rs Outdated Show resolved Hide resolved

rust/sedona-datasource/src/spec.rs Show resolved Hide resolved

python/sedonadb/python/sedonadb/datasource.py Outdated Show resolved Hide resolved

paleolimbot and others added 2 commits November 19, 2025 09:28

Update python/sedonadb/python/sedonadb/datasource.py

6568df3

Co-authored-by: Copilot <[email protected]>

Update rust/sedona-geometry/src/interval.rs

76e88ca

Co-authored-by: Copilot <[email protected]>

paleolimbot changed the title ~~feat(python/sedonadb): Implement GDAL/OGR formats~~ feat(python/sedonadb): Implement GDAL/OGR formats via pyogrio Nov 19, 2025

paleolimbot marked this pull request as ready for review November 19, 2025 16:03

paleolimbot commented Nov 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(python/sedonadb): Implement GDAL/OGR formats via pyogrio #283

feat(python/sedonadb): Implement GDAL/OGR formats via pyogrio #283

Uh oh!

paleolimbot commented Nov 7, 2025 •

edited

Loading

Uh oh!

petern48 commented Nov 9, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paleolimbot Nov 19, 2025

Uh oh!

paleolimbot Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(python/sedonadb): Implement GDAL/OGR formats via pyogrio #283

Are you sure you want to change the base?

feat(python/sedonadb): Implement GDAL/OGR formats via pyogrio #283

Uh oh!

Conversation

paleolimbot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petern48 commented Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paleolimbot Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

paleolimbot Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

paleolimbot commented Nov 7, 2025 •

edited

Loading

petern48 commented Nov 9, 2025 •

edited

Loading