Main feature differences with the pyspark delta combo?

[khulaifi95]

Hi everyone! I am currently looking into a POC lakehouse solution, processing and managing data for a language model service via stream or batch inference. I have similar experience with the spark ecosystem and rust language.

I find delta-rs has port to python package. How does the current release of delta-rs compare to the delta-spark pip package w.r.t the feature sets? I am in favor of a rust service because spark jobs introduce more complexity. And since models both read and write to the lakehouse, how does its I/O performance compare to the spark on a single-cluster? Thanks.