2023-12-14 kernel meeting notes

[zachschuermann]

summary

todo

action items

from last time

• Nick design doc
• new hackathon/FFI code review
• kernel write draft review

attendees

@wjones127 @nicklan @roeap @zachschuermann @vkorukanti @ryan-johnson-databricks @tdas

notes stream

• from Will: asking about testing/more DAT
• DAT test is a directory with a subdirectory for Delta table and expected data as raw parquet (reading table should be the same as just reading raw parquet files)
• extend DAT to do more complex testing? e.g. queries (DAT currently doesn't assume SQL support etc.)
• PR to remove the file context-related APIs from Java Kernel which will simplify the Rust APIs and bindings (?). More details are in this doc.
• Nick: rename TableClient to EngineClient as the interface is for dealing with the engine.
• RyanJ: Build a dataframe'sh kind of API for manipulation columnar batch. Currently, the expression handler outputs the ColumnVector, instead we make it return a ColumnarBatch.
• RyanJ: has a PR review. working on: how to support traits across the FFI boundaries. hammer out handling iterator lifetime.
• RyanJ: discussion around passing concrete class of TableClient across the FFI boundaries
• Insert/Append support in Kernel design draft is here. Please take a look at it.