2024-03-28 kernel meeting notes

[zachschuermann]

summary

lots of PRs in flight, DAT tests working, various features/fixes (listed below). duckDB moving forward and current highest prio is FFI v1 merge (then iterate).

action items

some PRs in flight:

• start implementing FFI #91
• Use a proc macro for being able to derive schemas for our action structs #129 (shouldn't be closed)
• Add backtrace option to errors #151
• [WIP] Implement extern EngineData #152
• Read and verify dat data + initial bit of timestamp_ntz #154
• Proper reading of parquet columns #150
• Arrow engine data cleanup #153
• Fix parition columns + add support for binary and decimal scalars #146 get DAT working!

higest prio right now is FFI PR #91 - split up, test, etc.

attendees

@nicklan @zachschuermann @hntd187 @ryan-johnson-databricks @tdas

notes stream

• duckDB moving forward!
• about to have FFI chat, drive that forward
• ryan: working through FFI stuff, traits, etc. lots of boilerplate: vtable thing and double-trampoline for callback where someone has to pass a callback when calling other system. as this proliferates might want to consider macros (one problem: there is C++ involved)
• in original FFI branch: engine can traverse kernel schema, callback for every node, can build its own schema. when implementing visitor, it has to deal with schema and can reuse the same code. dual recursion: schema and columns all in duckDB now. handles, vtables, contexts = lots of code
• proc macro pr still todo - waiting on github support LOL - losing hope in github support
• nick has PR for reading DAT and verifying. we can do all of them except column mapping, iceberg compat. NOTE: new tables in master branch but not in release. our build pulls release version. then after trying master we don't pass col mapping (and iceberg compat) and multi-partitioned (even after timestamp fix something is wrong). weren't passing timestamp-no-timezone, but as part of PR for reading, added as type we can deserialize. more work todo will follow up. protocol doesn't make difference between timestamp/timestamp-NTZ.
• without table feature there is no timestamp/nothing in parquet. with feature there is timezone and must be set in parquet. and introduces timstatmpNTZ which explicitly says no time zone.
• we have basically implemented timestampNTZ now
• need to go through and fix actual timestamp time with timezones. nick to fix in the future.
• TD. great to explore real way to integrate with duckDB. if we go down rabbit hole too much, we might want to be careful with what we can deliver for DAIS. some point we need to decide what we can deliver by DAIS. (consider using default implementation if it will compress timeline)
• ryan: some shim for arrow passed across language boundary
• duckdb could use default client for internal metadata reads etc.
• post-processing scan for data files is big item
• td: can we write examples soon? C consuming FFI?