[Enh]: Support Daft DataFrame

[hongbo-miao]

We would like to learn about your use case. For example, if this feature is needed to adopt Narwhals in an open source project, could you please enter the link to it below?

We use Daft as it is a unified engine for Data Analytics, Engineering & ML/AI and fast: https://delta.io/blog/daft-delta-lake-integration/

Performance chart done by Daft team

Please describe the purpose of the new feature or describe the problem to solve.

It would be great to support Daft DataFrame, thanks! ☺️

Upstream Issues

• time_unit and time_zone parameters in Timestamp DataType Eventual-Inc/Daft#4031
• enh: A way to check if a data type is Timestamp Eventual-Inc/Daft#4032
• daft.count('all') for use in group-by without any column names Eventual-Inc/Daft#4033
• dt.millisecond, dt.microsecond, dt.nanosecond, dt.ordinal_day, dt.timestamp, dt.to_string (or dt.strftime) Eventual-Inc/Daft#4094
• duration temporal expressions Eventual-Inc/Daft#4095
• time zone conversion features Eventual-Inc/Daft#4096
• enh: DataFrame.join_asof Eventual-Inc/Daft#4098
• DataFrame.unique(subset=..., keep=...) Eventual-Inc/Daft#4151
• str.to_datetime ignores '%z' Eventual-Inc/Daft#4220

[FBruzzesi]

Hey @hongbo-miao this is definitely in scope. We were waiting for a big refactor to land. Now that it has, we can start working towards supporting Daft, but I have the suspicion that @MarcoGorelli is already been cooking something recently 👀

[dangotbanned]

I've got a local branch added (#2223) with a mostly finished CompliantDataFrame (see for other parts #2202, #2119, #2064).

From a quick look at daft.DataFrame it looks like it would be using CompliantLazyFrame.
Luckily, it'll be able to match (#2211) via daft.DataFrame.explain

It'll probably be easier to scope out the work after spec-ing CompliantLazyFrame - but that shouldn't take too long.

---

The API reference is an interesting read.
It seems like a mix of polars, pyspark, pyarrow but also some Image operations that seem novel

[MarcoGorelli]

I have the suspicion that @MarcoGorelli is already been cooking something recently 👀

😄 indeed, got something cooking, will update when it's in a more complete state

[dangotbanned]

I have the suspicion that @MarcoGorelli is already been cooking something recently 👀

😄 indeed, got something cooking, will update when it's in a more complete state

@MarcoGorelli you should've mentioned you've got a narwhals label in their repo!

I've linked all the issues you've sneakily opened 😉

[MarcoGorelli]

We're getting closer

@dangotbanned 's work made this so much easier - well done Dan, thanks so much, it's hard to overstate how much impact you've had on this project 🙌

[dangotbanned]

We're getting closer

@dangotbanned 's work made this so much easier - well done Dan, thanks so much, it's hard to overstate how much impact you've had on this project 🙌

Thanks @MarcoGorelli, really means a lot!

[NellyWhads]

Hey folks! Few questions here:

1. Would narwhals use the daft executor directly under the hood? Or translate the daft.DataFrame to an internal representation to perform its operations? I'm mostly interested in how this interfaces with daft on ray
2. How is the narwhals typing system mapped to daft's typing system? daft support Tensors (including numpy arrays), for example.
3. Would using narwhals remove daft's ability to attach GPUs to transforms?

[MarcoGorelli]

Hey @NellyWhads !

The idea is that if you have daft.DataFrame then you can pass that to nw.from_native. Then, any operation you perform using the Narwhals API gets mapped to the daft DataFrame API

Any characteristics of the original object (e.g. what it's connected to, where it runs) should remain unchanged