2025-09-04 kernel meeting notes

[zachschuermann]

Summary

0.15.2 released! did a couple patch fixes: 0.15.1 for delta-rs visibility, 0.15.2 for MSRV fix and json reader bug.

Attendees

@zachschuermann @roeap @OussamaSaoudi @DrakeLin @MannDP

notes stream

• shipped JSON reader fix in 0.15.2 Json error when reading table through DuckDB #1139
• delta-rs migration check-in @roeap
  • log replay is done, merged to main! rust released with it, waiting on python release.
  • next phase: proper scans. currently do log replay to get file data then delta-rs interprets/reads (blocking col mapping, DVs)
  • pretty small code path that does reads
  • can work against main for now but will release the new iteration of transforms API in 0.16.0
  • zach targeting 0.16.0 next week.
  • next next phase: CDC reads kernelized in delta-rs
  • interesting thing in datafusion: some async/sync (i.e. spots you cannot do async bc you're in a sync spot). want catalog interactions in analysis (logical --> physical plan).
• unity support from robert (datafusion)
• catalog-managed tables support work @zachschuermann
  • reads coming soon, will work on integration in delta-rs (likely with uc-catalog crate?)
  • writes later.
• CDF transforms exploration
• started log compaction writes

leftovers/follow-ups

• kernel side follow up: to check on checkpoint data iterator shouldn't return batches with different schemas #1099
• ask: why does daft/ray need serializing/deserializing ListedLogFiles
  • need Snapshot to be serializable/deserializable
  • basically just need some IO-less way to make these things
  • consumers: Daft (delta-rs) and Ray (deltalake/python)
• follow up on Urls discussion from last time?