2024-12-05 kernel meeting notes

[zachschuermann]

summary

0.5.0 release! write FFI, metrics FFI, CDF ongoing, lots more expression/data skipping/col mapping/nested col support from ryan

pull requests/ongoing threads

• @zachschuermann on write FFI, user guides, etc.
• @scovich had visitor improvements and expression/schema transform landed, data skipping improvement/nested cols going in
• @OussamaSaoudi-db CDF ongoing
• @hntd187 has Map PR (ryan's PRs landed, just last one to do column mapping and nested data skipping)

attendees

@nicklan @hntd187 @zachschuermann @OussamaSaoudi-db @scovich @roeap

notes stream

• (from last time) note arrow needs help! Looking for help maintaining (reviewing PRs) in arrow/parquet/object_store apache/arrow-rs#6418
• robert: checkpoints? want to get rid of that code in delta-rs. writing checkpoints.
• upcoming project: V2checkpoint reads and then writes.
  • arrow used to be optional in delta-rs
  • robert: wants to take kernel's Actions and remove delta-rs' - need to get rid of checkpoints.
• the nullable vs. blank issue in delta-rs: TODO
• tyler: get rid of checkpoints? problems in this area lately.
  • robert: checkpoint writes is old code (original checkpoint 'v1' code)
  • does raw parquet parsing/handling (arrow can do all this for us)
  • tyler: we should use the same action types! robert: checkpointing is blocking this
  • tyler: not the write side - we have FromParquetRead that does manual read of row groups from checkpoint
  • currently with kernel 0.4.1, delta-rs can use kernel for log replay
• need to upgrade delta-rs to 0.5.0 then integration tests will use main
• DAT is inadequate
• kernel bump: chore: bump kernel delta-rs#3036
• list element naming broke everything :') - datafusion/arrow/etc.
• zach to do 0.5.1 after we fix column mapping default.
• zach: ask if delta-rs does specific commit/PR formatting? automated releases?
  • don't do commit-level formatting, do PR requirements
  • delta-rs requires PR formatting
  • check dev/release folder (script) GH changelog generated
• semver didn't catch breaking API change: Scan::execute takes an Arc<dyn EngineData> now #553 (comment)
• robert: PR messages are used to create the changelog
  • merge queue messes this up?
• robert: should enforce semantic commit messages at PR level
  • release on pushing tags
• ryan: squash merge: because github cannot handle force-merge gracefully we have to squash merge.
  • git = great at rebase
  • github = terrible at rebase
  • for now require squash, future if we get better rebase we can open up to non-squash
• nick: would be cool if we could have a PR bot that comments "here is your changelog update for the PR" - derived from description of the PR
• robert: prefer 'conventional commits' with 'semantic release'
• flagging: bad for monorepo
• check out changesets?