Cross-language FFI

[gatesn]

Apologies if this has been discussed before, I’m not quite sure on the keywords to search for here.

The workflow I’m looking for is:

1. User creates an OpenDAL client in Python using the Python API.
2. User passes this client to some other Python library written in Rust using PyO3.
3. The library written in PyO3/Rust can make use of the client configured using the regular Python API.

In other words, similar to Apache Arrow’s export_to_c / import_from_c where the pointer passed around has a stable ABI.

I think this would really help boost adoption of OpenDAL. It would allow an OpenDAL client configured by some project to be used by plugins / other projects and make OpenDAL a performant but pluggable I/O layer.

---

Mailing list thread: https://lists.apache.org/thread/kso30lyp4sltmm5f2nnhtfyr7tcoqpv3

[Xuanwo]

It seems like a great idea. I can imagine that it could also be quite useful to implement a new storage service as a plugin, which can then be easily passed to an OpenDAL user for use.

[gatesn]

I can be more concrete about my specific use case.

I maintain the https://github.com/vortex-data/vortex file format. We also have bindings in many languages, with a core written in Rust.

I'd like to support all sorts of object stores and other functionality like tracing, retries, caching etc. It seems sensible to delegate that to a project like OpenDAL.

But if I want a user of PyVortex to provide a Python OpenDAL client, I cannot currently (afaict) pass that client to core Vortex Rust. At least not without forcing strict version compatibility.