2024-08-01 kernel meeting notes

[zachschuermann]

summary

#295 and expr handler discussion. new release soon 0.2.1

action items

• First pass on IN/Not In #270
• feat: better ways to use deletion vectors #215
• Add golden table tests from delta spark/java kernel #295
• feat: typed table features #222

attendees

@zachschuermann @nicklan @roeap @hntd187

notes stream

• discuss Add golden table tests from delta spark/java kernel #295
• found nested col mapping doesn't work
• two kinds of expressions: col(name) - allow col renaming, struct expressions (seq of expressions turn into struct)
• root rename easy (struct)
• nested rename hard (struct embedded in high level struct, but need to be able to rename the struct col itself - need to be able to rename stuct)
• need to do release for feat: better ways to use deletion vectors #215? unblock duckdb
• missing 'give set of expr and put into struct but give the struct a new name'
• robert: concern - names as an array of strings instead of single string
• for reading we pass a schema (fully nested struct what to get)
• for expr eval is the same true? if you want subset of cols in struct it would happen at schema time?
• col only at leaf level
• traverse into nested col (struct) you use struct expr (might rename struct col or do work on nested cols)
• do you need support for nested cols inside expression? though in data skipping you might need it
• input type, output type, expr. only give input schema to expr eval
• when to use input/output in expr eval
• RecordBatch.with_schema(xxx) check if schema contains original, then can swap schema
• expr evaluated over input data type, then spit out output type. check that types all match but then do rename fixup at the end if we need to
• adds more complexity to docs for expression evaluator
• could be supported by acceptance tests for expr handler
• parquet reader: kernel specifies read schema, parquet handler has to give you that (kernel then knows order)
• tentative idea: expr eval should have input schema + expr together, you eval over input schema and expr. then produces some data with same 'shape' as output type but possibly different names. then do renames to match output schema
• in-and-out of arrow schema. would be nice to all share the same data types for schemas
• HTTP crate is a success story. can we do something like this for data
• maybe target this quarter
• start this work with kernel-types crate (slow moving crate for kernel data types - main consumer delta-rs)
• maybe we could use that as success story for motivating a 'schema' crate to share with arrow
• feat: typed table features #222 needs reviews!