2024-12-19 kernel meeting notes

[zachschuermann]

summary

0.6.0 released!

pull requests/ongoing threads

• @zachschuermann on write FFI, user guides, parquet reader improvement (async), need a way to create engine data
• @OussamaSaoudi-db CDF testing
• @hntd187 has Map PR
• @roeap delta-rs integration!
• @nicklan Exposing expression for physical to logical transformation to engines.

attendees

@nicklan @hntd187 @zachschuermann @OussamaSaoudi-db @roeap @rtyler @dennyglee

leftovers

• zach: define MSRV policy (follow datafusion?)
• zach: define release expectation
• zach: labels discussion
• zach/robert: datafusion integration

notes stream

• nick: There are issues with CI. We didn't check MSRV during tests. A deep dependency bumped its MSRV version, causing us to fail. Running cargo update then running msrv check is a good way to check. Added a test for MSRV during testing, and updated the coverage github job.
• nick: 0.6 released and CDF is shipped! 🎉
• robert: internal snapshot abstraction in delta-rs was changed to match delta-kernel-rs. Since then, there are new extensions in delta-rs that keeps the snapshot state using a caching layer. This allows us to incrementally update a snapshot with new commits. Action item is to re-integrate delta-rs and kernel without exposing kernel internals.
• robert: Moving to kernel brings new features like column mapping and dvs, but delta-rs needs to avoid losing Transaction support. Scans currently drop transaction log actions. Two solutions: 1) integrate only log segment and instead of the scan api. 2) Implement the features in delta-kernel.
• robert: When retrieving scan-data, would delta-rs be able to read transaction metadata and other actions?
• nick: kernel will drop those columns during a scan.
• robert: How will kernel work with transactions, since we need to be able to read these actions?
• nick: We need to discuss further for transaction support
• robert: How is column mapping and physical to logical transformation handled?
• nick: We are planning to hand you the logical schema, read schema, and expression to transform physical to logical.
• robert: Is there a contract that states that the ordinals are the same? How do you correlate physical column to logical column?
• nick: There is a contract that ordinals are preserved. The read schema + the expressions will produce data that ordinally matches the logical schema.
• robert: given an expression, is it in terms of physical or logical schema?
• nick: The input is the physical schema, while the output is the logical schema.
• robert: I'd have to look at the expression to know how physical maps to logical. I can't probe the logical and physical schema to infer.
• robert: This approach makes the user base much narrower. Fewer users know how to handle expressions and schemas. Perhaps this is okay since kernel is meant to be consumed by engines instead of delta users.
• nick: This approach requires sophistication, but a query engine implementers should be able to handle this.
• robert: Some documentation would be good. Got a start, but new changes may deprecate it. It's important to communicate how this should be done.
• nick: There were some options considered, but we stick to the simple abstraction of applying expressions. This requires that the engine can introspect on expressions so that they can figure out how to apply them. However, variant decoding is an example of expressions that engines are unlikely to be able to do on their own. The plan is to have the engine call back to kernel in these trickier cases so that kernel handles it.
• robert: Is it worth modeling all of this? Kernel can't express everthing that substrait expresses.
• nick: What's substrait
• robert: Substrait is a common convention to represent a relational algebra/logical plans in a unified/generic way.
• stephen: Similar to spark-connect, but more abstract. It's meant to be universal among engines.
• nick: This is worth looking into, but perhaps not immediately.