2024-05-23 kernel meeting notes

[zachschuermann]

summary

action items

PRs in flight:

• Harmonize BoxHandle and ArcHandle #182 <- close to merge
• Add support for "name" column mapping #205 <- simple, knocking it out now
• new attribute proc macro that will make handles easy to define (annotate a struct) coming from @scovich
• [WIP] updates for delta-rs #189 WIP from @roeap
• Have an ffi c example that actually reads data #203 (end-to-end read table print schema etc.) some decisions about how to expose engine functionality across FFI.
• don't fully enforce matching schema #210

attendees

@nicklan @zachschuermann @scovich @roeap @vkorukanti @rtyler

notes stream

• spark writes everything as nullable to parquet?
• see don't fully enforce matching schema #210
• nullable struct with non-null fields (can't do it in parquet) so we enforce higher in the stack
• kernel perspective: how complicated is our parquet interface going to be?
• schema transformations difficult; could do something 'close enough' then default to actual on-disk schema (nullability)
• serde: every action has sentinel col (path for adds) we could take nullable schema and enforce nullability application level
• e.g. naming in maps (key type etc.)
• kernel sends fully nullable schema to parquet reader. application level enforce constraints (e.g. if add.path exists then others must be there aka non-nullable)
• unfortunate: if we want to do row group pruning. can still do it via add.path instead of add.
• various parquet readers in the wild
• java side has parquet-mr but trino has custom and some others
• ryan: handles! safe (unsafe rust that is sound) and sane FFI
• danger from engine side is Send and Sync. mutable is Send not Sync, shared is Send + Sync
• shared handle: any number requests from any number threads
• mutable handle: unique pointer semantics. if you send to another thread you have to mutex protect to 'make sync'
• confusing if kernel consumes handles. (when is it usable again)
• if kernel gives engine handle, remain valid until drop
• mutable handles that are references to their own type: we have indirection - handle FooDescriptor instead of just Foo. if it is sized/mutable handle can a type be its own handle? ryan's todo!
• negative testing for handles to verify they disallow bad behaviors
• kernel 0.0.1 into delta-rs
• LLVM update hit recent rust version changes i128 repr
• DAIS in 3 weeks: tyler hoping nick/robert can land PR (link) for delta-rs 0.18 on kernel 0.0.1!