2024-11-14 kernel meeting notes

[zachschuermann]

summary

new PR board, lots of progress, 0.5 release?, graduation to delta-io!!

pull requests/ongoing threads

• write path merged, need FFI (from @zachschuermann)
• @hntd187 has Map PR
• lots of PRs from @scovich on visitor/predicate eval
  • visitor API not as dense (interface changes)
  • harmonize predicate eval (this is a big one but has exhaustive unit tests!)
• @OussamaSaoudi-db CDF

attendees

@nicklan @dennyglee @hntd187 @zachschuermann @OussamaSaoudi-db @scovich

notes stream

• graduating to delta-io: Graduate delta-kernel-rs from incubator to main delta-io GitHub repo. delta#3873
• GA prior to 1.0 release?
• flagging need for PR reviews!
• @OussamaSaoudi-db CDF update. E2E prototype working
• PR order
  • LogSegmentBuilder PR Add LogSegmentBuilder to construct LogSegments #457
  • TableChanges constructor PR [WIP] Add constructor for Tablechanges to get a CDF scan #459
  • Read path PR
• when DVs exist for a file is the numRecords stat an upper bound? yes? logic is still correct for isNull checks?
• protocol says numRecords is an upper bound when we have DV
• why does numRecords lag? might be higher since it doesn't reflect deleted rows yet.
• flagging as something to think about. are we correct on this?
• data skipping this is handled (ryan fixed in predicate harmonization) simple way turned out to be the correct way
• generally, as long as we don't return those stats as query results then that's okay
• metadata-only query answers start to get into trouble