Streaming analytics without pub-sub

[cloudcompute]

Hi

For each microservice, we store events in delta lake using delta-rs (event sourcing), that is to say, using Delta lake as a database itself for event-sourcing.

Then run a distributed SQL query engine on the Delta lake tables (one or more tables for a single microservice), .

Publishers --> Delta lake tables ---> Consumer doing analytics on delta lake tables

Can this pattern be used without any performance penalty or any other issue, does it even make sense?

Thanks

[houqp]

Yes, this kind of streaming pipeline is a very common pattern for deltalake. Only tradeoff compared to kafka is latency will be a lot higher (single digit seconds v.s. sub second)

[cloudcompute]

Hi @houqp

So this means when Kafka sits between Delta and a publisher, it reduces the latency to such a great extent. Kafka consumes memory, does the IPC, takes time to store an event message and forwards it to sink (deltalake), how can it help reduce latency? Does it write the data to deltalake in micro-batches, is this why Or some other reason?

Looks like deltalake is not suitable to do real-time analytics like analysing click streams, fraud detection whose response time should be less than a second, am I right?