2024-05-16 kernel meeting notes

[zachschuermann]

summary

duckdb repo public! lots of PRs. ryan contributing to cbindgen.

action items

PRs in flight:

• Harmonize BoxHandle and ArcHandle #182 <- close to merge
• Add support for "name" column mapping #205 <- simple, knocking it out now
• new attribute proc macro that will make handles easy to define (annotate a struct) coming from @scovich
• [WIP] updates for delta-rs #189 WIP from @roeap
• Have an ffi c example that actually reads data #203 (end-to-end read table print schema etc.) some decisions about how to expose engine functionality across FFI.

attendees

@nicklan @zachschuermann @hntd187 @scovich @roeap

notes stream

• duckdb open sourced their extension! https://github.com/duckdb/duckdb_delta
• ion's ask for type fixes: link?
• robert on delta-rs: normalize types, add struct literals back (lost them at some point). we can create a struct column but if you need to say "here are stats for file" that's struct literal that we don't support. lost this in expressions early on.
• ryan: quesion - what if we want to just unpack as a bunch of literals? is that easier or harder for an engine to use? delta-rs use case is using to create partition values in add actions.
• will iterate on a PR (from @roeap)
• lots of in-flight PRs! (linked above)
• discussion about what to do about cbindgen? use slightly less optimal header for now? yes.
  • context: in new handle PR we have a transparent struct Handle which allows us to attach functionality and prevents raw pointer stuff from being available. (i.e. disallow pointer usage and only our interface) - bindgen makes this typedef correctly.
  • right now you can say const SharedSnapshot * and you will match typedef - TLDR we don't have to use typedef (it's an alias)
  • place it matters: if we want nullable handle.
  • cbindgen breaks down for nullable handle.
  • current state: annoying but harmless
  • add support for transparent typedef in cbindgen: todo link
• making inspect-table work? some copypasta.