2024-08-08 kernel meeting notes

[zachschuermann]

summary

released 0.3.0! PRs coming in, need to review.

pull requests

• First pass on IN/Not In #270 (block on ryan's review)
• feat: add typed table configuration #309
• feat: better ways to use deletion vectors #215
• Add golden table tests from delta spark/java kernel #295
• feat: typed table features #222

attendees

@zachschuermann @roeap @hntd187

notes stream

• some more golden table tests work from nick
• new config PR from robert: feat: add typed table configuration #309 (adapted from delta-rs)
• bug? Error reading from delta table with >= 100 commits #308
• target write review next week (zach)
• robert next items: checkpoints (kernel/delta-rs integration)
• steve to tackle expressions in kernel/delta-rs integration
  • first: partitionValues. using kernel expression system for partition values
  • relies on First pass on IN/Not In #270
  • will unblock delta-rs adoption. take partition value stuff out of delta-rs into kernel (as data skipping)
  • context in First pass on IN/Not In #270
• robert to review Enable some more golden tests #301
• lots of churn on the map/list field names in parquet. (element/key_values/etc.) we need to rely on ordinals instead!
• reviewed PR board