2023-09-07 kernel meeting notes

[zachschuermann]

summary

• chat about meet-up and some new threads in the ecosystem (delta-sharing, iceberg-rust, etc.)
• planning meetup
• re-starting delta community talks
• merge flow: target 2 deep reviews
• merging proposal: remove async api, keep async default client #29 soon (after @nicklan rereview)
• few repo admin follow-ups owned by @MrPowers
• discuss how to implement multi-language support: two APIs - rust API and C API. language bindings built on top of one of the two. (and in the immediate future we will get python APIs for free if we lift delta-rs on top of kernel 😄)

action items:

• CI running against forks now :) (@zachschuermann - from last time)
• action item: merge @wjones127 PR? proposal: remove async api, keep async default client #29
• action item: how to request multiple reviewers (@MrPowers)
  • probably update to Github Teams
  • @MrPowers is owner of incubator org now so we can get this stuff sorted

attendees

@tdas @rtyler @wjones127 @nicklan @MrPowers @ryan-johnson-databricks @zachschuermann

notes stream

• discussing meet-up
• to sync with delta-sharing-rs (@ognis1205)
• discussing new delta-sharing protocol
  • plays nice with delta-kernel. there needs to be client-side logic in order to replay actions.
  • server can use kernel to read delta, then send off actions, client can use kernel to read the data
• @MrPowers bringing back delta community meetings: starting back on the 20th.
  • lightning talks, etc.
  • @rtyler to share updates on delta-rs side
• https://www.arroyo.dev/
• note https://github.com/apache/iceberg-rust
• follow up from last time: @wjones127 PR proposal: remove async api, keep async default client #29
  • nick to re-review
  • merge soon then iterate?
  • PR merge flow: 2 deep reviews? some combination of databricks/OSS eyes to avoid isolation
• from @nicklan: if there are changes coming in please include for review (and more time budgeted soon)
• updates from @MrPowers
  • kernel is gaining traction
  • full delta support in pandas hopefully soon and wanting to promote it
  • when can we target rollout of read support for e.g. pandas
  • java side coming in delta 3.0 (sans skipping/stats/etc) that can be used
• target same level of support (read-only) by the end of January 2024
  • with FFI so we don't design without it (and make it harder to implement later)
  • tangent: how to handle FFI - ideally a narrow waist that can be shared between C++/Python/etc.
    • option 1: C FFI that does everything (python/c++/go/etc. on top)
    • option 2: use specific crates (Maturin for python, Cxx for C++, etc.)
  • Tyler: maturin is useful - use if you are making native python lib in rust (target CPython ABI)
    • consumer-side (connector side) can use Maturin but python would not be a 'native connector' for the purpose of kernel design work
  • two APIs? Rust API and C API/ABI? then build wrappers around one of them for different high-level languages?
    • @MrPowers wants python
    • we can support python by lifting delta-rs onto kernel
    • pip install deltalake would pull in delta-rs on kernel?
    • market that can't reach delta right now might be C/C++ guys