[Enh]: Support `datafusion-python`

[hendrikmakait]

We would like to learn about your use case. For example, if this feature is needed to adopt Narwhals in an open source project, could you please enter the link to it below?

datafusion-python includes the Python bindings for Apache DataFusion, a fast, embeddable, modular analytic query engine. As part of diving deeper into the Python API, I've noticed that one of the biggest gaps in using DataFusion effectively is the lack of support in third-party libraries. This is why supporting it in narwhals would be very useful.

Please describe the purpose of the new feature or describe the problem to solve.

It would be great to support datafusion-python in narwhals.

Suggest a solution if possible.

No response

If you have tried alternatives, please describe them below.

No response

Additional information that may help us understand your needs.

I'd be happy to do the heavy lifting on this enhancement, but I don't know how to start.

From what I can tell, support could be added by implementing the compatibility layer within the narwhals repository, i.e., by adding a _datafusion module that implements all necessary functionality. I also noticed work happening extensibility and plugins for narwhals, .e.g., https://github.com/MarcoGorelli/narwhals-daft or #2978. What is the recommended way of adding support for another dataframe library?

[MarcoGorelli]

thanks for the request - at the moment i think this should be a plugin

[hendrikmakait]

Closing this as not planned as part of this repository and starting the integration in https://github.com/hendrikmakait/narwhals-datafusion

[FBruzzesi]

Hey @hendrikmakait 👋 the last time I checked the datafusion python bindings, the API was still in fast development.
I am not sure where it got, but consider taking a look at how we have a more generic _sql module that is used for all common operations in duckdb, ibis, and pyspark-like. If it fits into that without too much special cases I would personally be ok considering adding it directly in Narwhals. I am not sure about @MarcoGorelli takes though

[MarcoGorelli]

I think we'd need to do a lot more code-sharing in narwhals/_sql before we could add another backend to it, but i'm not completely opposed to the idea

[hendrikmakait]

@MarcoGorelli, do you have anything specific in mind? I'm happy to jump in and help.