Support constructing a LazyFrame from a serializable logical plan

[jonmmease]

This is very likely out of scope of Narwhals, but thought I'd share the shape of a problem I'm thinking about for VegaFusion, that Narwhals could potentially help with.

As described in vega/vegafusion#567, it would be very useful for VegaFusion to support computation against external DataFrame libraries (rather than only the in memory DataFusion engine). The Narwhals API has grown to the point where I think it could support all of the relational operations the VegaFusion relies on. And so, if we could work out a way to take a DataFusion LogicalPlan in Rust, and somehow build a Narwhals LazyFrame corresponding to this plan, then VegaFusion could push computation to Narwhals.

For just VegaFusion, we could serialize the DataFusion logical plan to Python using protobuf and then manually walk it to build up a Narwhals LazyFrame. But I wonder if it would be useful to the larger ecosystem for there to be some serializable representation of a Narwhals logical plan that Narwhals could ingest to build a LazyFrame, and then VegaFusion could target instead.

[MarcoGorelli]

thanks for the request!

i'm not totally sure how in-scope this would be, but i'd really like to prioritise the extensibility mechanism now (after we land stable.v2) so that, if there's community interest, people can experiment with these kinds of plugins themselves. If any plugin was to gain sufficient interest / popularity, we could consider upstreaming it

[dangotbanned]

@jonmmease I've been experimenting with Expr-level plans in (#2572) based on a subset of the polars internals

• https://github.com/pola-rs/polars/blob/53913e0b1de0414b521848597a2c79e167de91a7/crates/polars-plan/src/dsl/expr/mod.rs

IIUC, what you'd be interested in is bigger in scope and corresponds to these parts, for the LazyFrame-level plans:

• https://github.com/pola-rs/polars/blob/53913e0b1de0414b521848597a2c79e167de91a7/crates/polars-plan/src/dsl/plan.rs
• https://github.com/pola-rs/polars/blob/53913e0b1de0414b521848597a2c79e167de91a7/crates/polars-plan/src/dsl/builder_dsl.rs
• https://github.com/pola-rs/polars/blob/53913e0b1de0414b521848597a2c79e167de91a7/crates/polars-plan/src/plans/builder_ir.rs

For now, I'd say it's out of scope - but who knows where this all leads 😅

---

Have you looked into any of these packages?

• https://ibis-project.org/backends/datafusion
• https://sqlframe.readthedocs.io/en/stable/
• https://sqlglot.com/sqlglot/dialects.html

We support ibis and sqlframe, which both use sqlglot internally and that can produce a Plan

I wonder if some combination of those packages may be a better fit?

[jonmmease]

Thanks for getting back to me @MarcoGorelli and @dangotbanned. I am familiar with (and have used!) ibis, sqlglot, and sqlframe.

ibis and sqlframe have the property that you use their imperative DataFrame API to build up a hidden internal query plan, and then execute that plan against a variety of backends. What I've been hoping to find (for a while) is a system that can input a declarative logical plan (substrait would be one example) and then execute it against a variety of backends. Maybe I'll eventually play with writing a adapter that processes a substrait plan and uses the imperative narwhals or ibis APIs to build a corresponding query.