2025-08-21 kernel meeting notes

[zachschuermann]

Summary

two big items

1. 0.15.0 this week/next
2. tracking issue for lots of our work threads: summer 2025 kernel work #1184

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)
• catalogManaged reads
• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

upcoming

• zach: metrics
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory
• Look into exception unwind from foreign code
• do we need to update protocol for resolving field ids? in col mapping id mode?
• @rtyler fixed delta-rs checkpoint issue (follow up to check on our side)

Attendees

@zachschuermann @roeap @OussamaSaoudi

notes stream

• delta-rs perf issues (transforms): two PRs fixing perf!: ExpressionRef instead of owned for transforms #1171 and perf: partition cache for transform expressions #1172
• delta-rs next branch - merged to main! 🎉
  • follow up PR that does some cleanup
  • won't release until kernel stuff makes it into delta-rs (perf improvements)
• FYI: need serializing/deserializing ListedLogFiles (need internal-api again)
  • need Snapshot to be serializable/deserializable
  • basically just need some IO-less way to make these things
  • consumers: Daft (delta-rs) and Ray (deltalake/python)
• zach and robert on 0.15.0 migration for delta-rs

leftovers

• kernel side follow up: to check on checkpoint data iterator shouldn't return batches with different schemas #1099
• follow up on Urls discussion from last time.