2024-07-11 kernel meeting notes

[zachschuermann]

summary

• Nick working on large Push parquet select to leaves, add correct reordering #271
• another release soon
• next week in this meeting go over java kernel writes

action items

• Push parquet select to leaves, add correct reordering #271
• First pass on IN/Not In #270

attendees

@rtyler @zachschuermann @nicklan @roeap @hntd187 @vkorukanti

notes stream

• Push parquet select to leaves, add correct reordering #271: you can give schema to parquet, do validation, fill in columns that were not there and are nullable, do reorder (recursively - structs in lists etc. -- doesn't do struct in map yet) been a heavy lift
• another release coming soon!
• @hntd187 want to include IN/NOT IN: First pass on IN/Not In #270
• from nick: splitting types in kernel into a new crate - planning to do soon
• motivation: if we have a slow-moving sub-crate (kernel-types) then it drastically reduces dependency issues for those that want to integrate with kernel. (sub-crate in current kernel repo) - similar to http crate for http types
• benefit to delta-rs? could depend on types crate for lower-speed dep.
• @roeap: might not see benefit?
• @rtyler: benefits: types in isolated spot means stable data model and faster moving API. delta-rs takes dep on kernel-type = 1 (instead of kernel = 0.1.1) etc. main benefit is release management/dependency management.
• @nicklan: put catalog types in the types crate as well (in the future)
• managed commits = need to talk between engine/catalog/kernel
• @nicklan unified code paths for execute() and scan_data()
• looking in write support soon!
• FYI: working on adding the coordinated commits (prev name: managed commits) support in Kernel Java. There will be a design doc soon. It has implications for Engine APIs