Spike: evaluate if cuDF can be used with datafusion-python

[timsaucer]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As other DataFrame libraries start moving to leveraging GPU resources, it would be useful to see if we could leverage the work already done in pandas and polars for interoperating with cuDF to give a similar experience in DataFusion.

Describe the solution you'd like

Evaluate the level of effort and technical limitations to using cuDF to evaluate DataFrames. Also worth evaluating is their c++ interface which we could potentially bring in to DataFusion upstream if we are willing to write the appropriate wrappers.

Describe alternatives you've considered

Leave as is.

Additional context

This task is really just focused on researching what would be required and if there is an opportunity here.

[andygrove]

I have some experience in this area. While at NVIDIA, I created a POC with Rust bindings around cuDF and then provided interoperability with arrow-rs. Unfortunately, that code was internal and not open-source. I used cxx to create the bindings.

This repo (datafusion-python) once contained a prototype of translating DataFusion logical plan to cuDF operations (all in Python). It is still there in the history somewhere.

[andygrove]

I see that there is now one RAPIDS library that provides Rust bindings: https://docs.rapids.ai/api/cuvs/nightly/rust_api/ so it may be interesting to see what approach they took to wrap C++ in this case.

edit: cuvs is using bindgen

[drauschenbach]

This repo (datafusion-python) once contained a prototype of translating DataFusion logical plan to cuDF operations (all in Python). It is still there in the history somewhere.

Possibly #602.

[deanm0000]

polars has a gpu engine mode. Looks like NVIDIA/RAPIDS made it https://docs.rapids.ai/api/cudf/stable/cudf_polars/ and the meat and potatoes aren't in the main polars library. When I click on Show Source from that page it doesn't go to actual source code so maybe not actually helpful so sorry if this is misdirection.

[paleolimbot]

Just two things I was involved in that may be useful here:

• cudf::from_arrow(): https://github.com/rapidsai/cudf/blob/2789fa83d943649b982493d68bbba852f848d82c/cpp/include/cudf/interop.hpp#L677-L698
• https://github.com/apache/arrow-nanoarrow/blob/fe0e3a2745952c7bd415fdf09bda2a869bf1390a/src/nanoarrow/nanoarrow_device.h#L278-L288

Probably integrating at a higher level than those makes sense (e.g., converting expressions), since the trick of optimizing data transfer to/from the GPU is sort of insane.

The nanoarrow version also includes a (probably oudated) Apple Metal backend, which is potentially easier to integrate because you can use the same memory addresses space for the buffers (or I could, the last time I tried this).

[milenkovicm]

polars blog gives quite a lot of details about CUDA integration https://pola.rs/posts/gpu-engine-release/ . Looking the code, looks like they have integrated it in the python layer not in rust.

[milenkovicm]

looks like they have used bindgen https://github.com/rapidsai/cuvs/blob/1155a3a427cd1d1bfaf8fe74a937ed6dfa797ec7/rust/cuvs-sys/build.rs#L75 to generate rust bindings

[JigaoLuo]

polars blog gives quite a lot of details about CUDA integration https://pola.rs/posts/gpu-engine-release/ . Looking the code, looks like they have integrated it in the python layer not in rust.

As an external contributor to cuDF, I ran into this issue. I could confirm that cuDF and Polars interact exclusively at the Python level, as there’s currently no binding between cuDF’s C++ backend and Polars' Rust core.

One point worth highlighting is the role of pylibcudf: https://docs.rapids.ai/api/cudf/stable/pylibcudf/. This Python wrapper aims to expose the entire libcudf API with minimal overhead, enabling high-performance data operations in Python. Its design allows efficient bridging between Python and C++ without incurring the usual latency associated with Python bindings. However, pylibcudf still lacks full coverage of libcudf’s functionality, and its documentation is not complete.