2024-06-06 kernel meeting notes

[zachschuermann]

summary

delta-rs integration!! yay!

action items

PRs in flight:

• Support unary NOT and IsNull in data skipping #231 zach to review
• new attribute proc macro that will make handles easy to define (annotate a struct) coming from @scovich
• [WIP] updates for delta-rs #189 WIP from @roeap
• Have an ffi c example that actually reads data #203 (end-to-end read table print schema etc.) some decisions about how to expose engine functionality across FFI.

attendees

@zachschuermann @nicklan @roeap @hntd187

notes stream

• thanks @roeap for migration PR!!
• follow-up on advanced API discussion from last time?
• isNULL pr Support unary NOT and IsNull in data skipping #231. can largely eliminate NOT expr by pushing down. except for NOT NULL. then we can implement NOT NULL as expr and no requirement of NOT.
• robert: want to upstream for convenience: physical names, invariants on fields, can avoid NOT expr but if we have invariants with NOT? constraints on write path. dep on metadata.
• constraints / generated columns requires expressions. anything goes.
• is the simplification of 'NOT pushdown' worth it? could cause confusion.
• duckDB: notnull and null expr types. easier for them to not handle explicit NOT? in duckdb you can do select * from x where NOT ... => transform with de morgan's
• always need users to have NOT available. need to decide on representable vs actually implemented by engine.
• NOT is in SQL spec; obviously can't say that users can't write it. but could let engines not need to evaluate it
• blog post is up! https://delta.io/blog/delta-kernel/
• deciding on canonical names: delta-rs = 'delta-ar-es' not 'delta-rust' :)
• @hntd187 u64 slices instead of bool slices for DVs. u64 of row index. vec of u64: give indexes which are deleted.
• nick is big brain: DVs are opaque type until the last minute.
• new API for getting back the indexes instead of a bunch of bools.
• aside: http client in object store
• robert: other convenience stuff to consider: TableConfig to wrap opaque map of config, include defaults, etc. just need ref to map
• @hntd187 volunteer! https://github.com/delta-io/delta-rs/blob/main/crates/core/src/table/config.rs