Remove the need for registering an ObjectStore for remote files

[mesejo]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently, we need to register an ObjectStore to register remote files stored in S3, https, etc. This could be more ergonomic from a DX perspective.

Describe the solution you'd like
Automatically detect if an ObjectStore is needed, like in the datafusion-cli; see here.

Describe alternatives you've considered
keep it as it is now

Additional context
We should increase the test coverage for ObjectStore

[ion-elgreco]

Just like delta-rs, polars etc we should just take the objectstore from the uri and then parse the storage options either as parameter or env variables

[timsaucer]

Related, though not yet on crates.io: https://github.com/developmentseed/object-store-rs

[kylebarron]

Related, though not yet on crates.io: developmentseed/object-store-rs

I'd be happy to work with you to use this! Indeed this is the explicit goal of pyo3-object_store, to define the Python builders for ObjectStore once and then reuse them for multiple projects.

[robtandy]

@kylebarron What are your thoughts on how to approach this? I'm happy to try to address it and submit a PR for it.

I'd like the same thing for github.com/apache/datafusion-ray, as after the rewrite, I need to add support for object stores back in and I'd love it to be consistent with how it will work in datafusion-python.

[kylebarron]

I'd suggest to wait until the object_store 0.12 release (and, then, for datafusion to use that) (because I'm pinned to latest main of object_store from pyo3_object-store right now until that release). But you can see how I'm reusing pyo3_object_store here: https://github.com/developmentseed/async-tiff/pull/17/files

In essence, you can just re-export the store builders and then just accept PyObjectStore in whatever function that should interact with a store.

Then copy the type hints for the builders if desired. That exposes the full builder API as documented within obstore.store: https://developmentseed.org/obstore/latest/api/store/aws/ (so your Python API can supply all possible configuration settings to object_store)

[robtandy]

I'm not sure I follow exactly. Do we need to use python bindings to the object_store? If we are going to, in datafusion-ray and datafusion-python automatically register a store based on the url of the data files within register_listing_table won't the interaction with the store be in rust?

[kylebarron]

To answer the original question of this issue: I think it's necessary to export classes to Python to customize object_store instances, because people often have custom config/authentication they need, and putting all config in env variables isn't sometimes possible.

But it's an API decision whether to require that the ObjectStore class is created manually. You can alternatively create one from the environment when one isn't passed in by hand.

But the point of pyo3_object_store is that you still get raw ObjectStore instances under the hood.

[robtandy]

If we accepted PyObjectStore in functions that need to interact with a store, and I agree that is a good idea, won't we have the rust ABI issue outlined here https://github.com/apache/datafusion-python/blob/main/docs/source/contributor-guide/ffi.rst ?

[kylebarron]

No, because the Python classes would be re-exported via datafusion: datafusion.store.

So the end user would use, for example

from datafusion.store import S3Store

store = S3Store("bucket", region="us-west-2", skip_signature=True)
# pass this into a DataFusion API that accepts `PyObjectStore`

So there's no FFI happening here.

[kevinjqliu]

I also ran into this issue integrating iceberg-python with datafusion-python using the FFI Table Provider. The integration goes from iceberg-python -> iceberg-rust -> datafusion (via FFI_TableProvider) -> datafusion-python

iceberg-python configures object store via the FileIO interface. I'm trying to figure out the best path to pass the configs around.

[kylebarron]

Can you say what issue you ran into?

[kevinjqliu]

sure, the main issue is figuring out how to pass the credentials from one system to the other.

I would like to load an iceberg table object using pyiceberg and then register the table as a table provider with datafusion-python. Internally, pyiceberg uses iceberg-rust to interface with the __datafusion_table_provider__ trait. The iceberg table object is "recreated" in iceberg-rust using its metadata.json file.

Heres an example from the WIP PR https://github.com/kevinjqliu/iceberg-rust/pull/2/files#diff-edc1fdc3ace9b88d4f61113ff01e977eb928495adaa25c9e03aa07cfc5d6aceeR64

I want to figure out a way to automatically pass credentials from pyiceberg -> iceberg-rust -> datafusion.

[kylebarron]

Where's the ObjectStore there?

[kylebarron]

I published pyo3-object_store 0.1, which works with pyo3 0.23, and pyo3-object_store 0.2, which works with pyo3 0.24. But both of these require object_store 0.12, so we'll need to wait for a datafusion release that uses that version.