Rust InterOp Architecture Decision: The role of delta-kernel and/or delta-rs in delta-dotnet

[mdrakiburrahman]

Hey @mightyshazam,

Jotting down some updates, hoping to get your thoughts on some options + questions below.

✏️Updates

I spent a lazy Friday reading through the Bridge implementation. First of all, really great work, I especially love the DataFusion support we inherit from delta-rs, running SQL on top of Delta via C# is something I never thought I'd see 🙂.

The Tokio runtime passing the Cancellation Token from C# to Rust is super smooth, I love how everything is async through and through, and the unsafe code is neatly tucked away in the Bridge.

It's obvious you've put a lot of effort and thoughtfulness into this project, I really appreciate it, it's fantastic for the Dotnet community.

As @MrPowers mentioned earlier, before I knew about delta-dotnet - I've been dabbling in the work the delta-kernel folks have done, and used a very similar approach as you to convert the FFI into C# via ClangSharp. I spent a weekend throwing some working C# code together against the delta-kernel FFI where you can convert Delta into a Arrow Table - but it's not nearly close to the amount of thoughtfulness or effort you've put into the Bridge, Runtime etc - and the underlying Rust code.

I actually ran into this repo when string searching for ClangSharpPInvokeGenerator params :)

My observation is, today - delta-rs is a lot more feature-rich, compared to delta-kernel at this point (e.g. SQL support via DataFusion). I also understand that delta-rs has slowly started converting delta-kernel too.

Given that delta-dotnet already works just fine for read/write, I'm wondering what your vision is for the project.

There's 2 high level options I'm seeing.

---

Option 1: delta-dotnet keeps dep on delta-rs , but no dep on delta-kernel

If it ain't broke, don't fix it.

Keep the existing dep on delta-rs for foreseeable future.

As delta-rs adopts delta-kernel, the Bridge evolves it's FFI in lockstep with delta-rs and enjoys new delta-kernel features anyway.

Use the saved effort to build more handy features on top of the Bridge that's C# native, e.g. expose a DataFrame C# API (works against Arrow) - via DataFrame.FromArrowRecordBatch.

✅Pros:

1. Minimal implementation logic in delta-dotnet, enjoy as delta-rs folks do most of the hard work 🙂
2. Existing delta-dotnet implementation is more or less feature complete for most use cases, just keep in cruise control, new features, etc.
3. delta-rs - being Rust - is performant as is via P/Invoke and hopefully supports multiple writers via partitioning (presumably, I haven't benchmarked yet, didn't see a delta-dotnet unit test.)
4. Azure Blob, GCP Bucket, AWS S3 etc all works as is (I haven't tested yet, but I see in the TOML file you enable that feature, so hopefully there's a method the Bridge exposes that takes in params to connect to each cloud storages)

This would be good to document/add an example for regardless, I imagine most delta-dotnet use-cases needing this

❌ Cons:

1. delta-dotnet architecture tied to delta-rs for good, rather than being tied to delta-kernel (more flexible).
2. If the need comes, and we want to implement a custom Kernel Engine Implementation in C# for some reason, can't do that

---

Option 2: interface out delta-dotnet's dep on delta-rs , and slowly migrate to delta-kernel as it matures

Step 0 - add Interfaces to everything under Table/*.cs, and slowly migrate to delta-kernel, starting with Arrow-based read support which already works. Mark unsupported methods with NotSupportedException or something similar, until Kernel supports it.

Step 1 - When delta-kernel adds write-support, adopt that.

Step 2 - Phase out dependency on delta-rs? Of course, we'd need the same SQL capabilities DataFusion adds to not break users. maybe the Kernel could add support for something similar via scan.

When web-searching, I didn't see any glaring examples of native DataFusion support in C# yet, so removing delta-rs and not having SQL support anymore could be a pretty big limitation.

✅Pros:

1. Tied closer to the "future source of truth of Delta", delta-kernel
2. Since we control more knobs in C#, can get fancy and add Entity Framework support etc. I'm not sure how much value this adds though in real life use cases.

❌ Cons:

1. Not really "gaining much" in business value by doing all this work, the C# code still works against "some other Rust thing that's also fast".
2. C# isn't a Database Engine, so it's not like this migration makes us gain much perf, if we tried to implement our own Kernel Engine. Maybe we could use the native C# Parquet reader to get a bit more perf boost, but perhaps that will be a trivial amount if the delta-rs Parquet -> Arrow reader is rapid already.
3. Bunch more work to add delta-kernel support, csproj wrappers, build pipelines, maintenance etc.
4. No clear path in future for DataFusion AKA SQL support.

---

Would love to hear your thoughts on which Option you prefer, and if there's a 3rd+ logical Option.

I'm personally happy to support implementing both options - specially Option 2, since I have a bunch of context on the Kernel from digging into it - but, only if there's a clear-value add in this new architecture.

Option 1 doesn't really need "new architecture work" (you've done it already) - but we could focus on pushing the limits of delta-dotnet and enabling new use cases, like beefing up the examples folder for Azure Blob etc, using TokenCredential, patterns for N parallel threads writing to same table against N partitions, samples on how to deserialize Event Hubs/Kafka partitions (basically KDI in C#), ensure things work in a container without weird problems, adding a DataFrame API to extend Table, and so on.

[mightyshazam]

Kernel's ffi types are the future. I haven't spent as much time as I should have investigating the kernel code because write support is a current limiter. I've helped with some python adapters, but that's it.
I think my short term vision is option 1. A working implementation based on delta-rs. Long-term, delta kernel is the way. It should allow for more customization without having to radically change delta rs. However, given that there is significant overlap between the kernel and delta-rs contributors, what shakes out will be interesting

[mdrakiburrahman]

I also hope to see the promise of the Kernel pull through.

I suppose for Option 2, similar to delta-rs, there's nothing stopping us here in delta-dotnet to start bringing the Kernel FFIs in then.

My only concern before starting to code was delta-dotnet fully depends on the DataFusion SQL for the read path today, I don't think we can get that from the Kernel anytime soon, unless someone makes DataFusion work with C# (perhaps we can one day).

But, I was able to get C# DataFrames working with the Kernel. Perhaps that's good enough to start.

I'll start working on a PR along the lines of "DataFrame API support backed by Kernel FFIs" as a soft entry to Option 2. You keep on honing Option 1, and we meet halfway and continue on Option 2.

LMK your thoughts.

[MrPowers]

Thanks for typing this up so neatly.

I think Delta Kernel will support all the table features.

I'm not sure if Delta Kernel will ever help out with an operation like Z Ordering:

dt = DeltaTable("tmp")
dt.optimize.z_order([country])

So perhaps Delta Kernel will be useful for all the table features, but delta-rs will still be useful for Z Ordering-type operations. Or perhaps we can just use Delta Kernel and then implement Z Ordering natively in C#.

[mdrakiburrahman]

Sounds good, I think this is making a case to interface delta-dotnet operations such that both delta-rs and delta-kernel can be leveraged via an FFI to give C# a "best of both worlds" experience.

We can:

1. Prioritize delta-kernel usage where possible
2. Fall back to delta-rs for the rest, unless a native C# implementation exists