2024-04-11 kernel meeting notes

[zachschuermann]

summary

published on crates.io! https://crates.io/crates/delta_kernel

action items

some PRs in flight:

• [WIP] Implement extern EngineData #152
• Split scan execute #169
• add app txn visitors and struct #170
• start implementing FFI #91

attendees

@nicklan @zachschuermann @hntd187 @rtyler @roeap @ryan-johnson-databricks @tdas

notes stream

• published on crates.io! https://crates.io/crates/delta_kernel
• scan API discussion (from @nicklan)
• @roeap: talking with Tim Dikland about delta-sharing-rs
• need to think about overall story
• nick: python bindings live in delta-rs
• otherwise, you can build own python bindings on top of kernel
• types/expression integration with delta-rs

scan discussion:

• context: old execute() 'crutch' API from the prototype
• the 'real' API should be the pieces in execute() (i.e. scan.files(), etc.)
• nick PR Split scan execute #169
• for metadata associated with a file, how does engine get access
• option (1) custom type (struct) with info inside (needs to be ser/de, etc.)
• option (2) could just be EngineData. we read AddFile row and then transform into another form that has all data (including file-specific). might be trickier since we don't have APIs to create EngineData ourselves.
• robert: think of metadata as data. td: in fact this has been an early decision in Delta's creation!
• rust struct will become public API
• we might not need to create EngineData out of thin air since it comes from AddFile
• add file rows --> transform --> give actual data you care about
• log replay: engine data and selection vector. needs some extractors since kernel doesn't know the format of the data.
• need a public subset of the schema (e.g. add.path we promise will be there)