datafusion

[riziles]

I just discovered (accidentally) that DataFusion now provides tight integration via register_table_provider. You should add it to your docs! It's very helpful.

[ion-elgreco]

@riziles you want to make pr for that?

[riziles]

@ion-elgreco , I started working on this, but then I ran into some issues testing. I upgraded to deltalake 0.25.1 (Python), and now this very basic example fails. Will post issue in DataFusion repo as well.

from deltalake import DeltaTable, write_deltalake
import pyarrow as pa
from datafusion import SessionContext

arrow_tbl = pa.Table.from_pydict({"x": [1, 2], "y": [2.0, 4.0]})

write_deltalake("./test/scratch", arrow_tbl, mode="overwrite")

dt = DeltaTable("./test/scratch")

ctx = SessionContext()

ctx.register_table_provider("tbl", dt)

print(ctx.table("tbl"))

[riziles]

issue posted here: https://github.com/apache/datafusion/issues/14842

[riziles]

This issue is resolved by upgrading deltalake to 0.25.4 and datafusion to 45.2.0 . I tried to submit a PR to update the docs, but the commit linter is ruining my Saturday, so I gave up.