2023-12-07 kernel meeting notes

[zachschuermann]

summary

action items

from last time

• nick design doc
• new hackathon/FFI code review
  new items
• kernel write draft review

attendees

@wjones127 @nicklan @roeap @zachschuermann @ryan-johnson-databricks @tdas @vkorukanti @MrPowers

notes stream

robert data skipping question
do we need contextualization in file reads?
column blob/data chunk/thing interface
we likely do want to be generic over the engine data type

• could use Any (would have to do this in FFI anyways)
• could have type safety with generic type (and only use Any in FFI)

tangent: robert question about expression handler - to update PR
engine has to convert from types requested from caller to types buried in data chunk

nick: two types

1. parent/container type (data chunk)
2. leaf node types (things you get back from engine after extraction)

we can't be type safe on the leaf types, but we can be type safe on the parent types.
FFI will likely have a separate implementation without generics.
Ryan concern: will we actually gain something from this type safety?
Main tradeoff to consider: (good) type safety, (bad) generics proliferation
matthew: clickhouse meeting coming up
polars? use pyarrow for now then upgrade to kernel!
TD: venki is close to first draft of kernel writes.
robert: what is the future of LogStore?
next week no ryan/TD