2025-07-17 kernel meeting notes

[zachschuermann]

Summary

0.13.0 released!

current PRs

• !!!! opaque expressions @scovich
• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)
• catalogManaged reads

upcoming

• zach: metrics PR!
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory
• Look into exception unwind from foreign code
• do we need to update protocol for resolving field ids? in col mapping id mode?

Attendees

@zachschuermann @nicklan @joonyoo181 @hntd187 @OussamaSaoudi @roeap

notes stream

• @mightyshazam the rustls stuff merged - right after 0.13.0 --> coming in 0.14.0
• zach builidng catalogManaged
• oussama: column mapping APIs? asking engine to read parquet - we maybe want to read with name mode or id mode
  • according to parquet spec: resolve with field_ids if they are present
  • note this is how delta-spark, etc. does it
  • do we need to update protocol?
  • kernel removes field ids from schema unless we are in id mode
  • nested fields tricky (check what spark does? always fall back to name or only fall back to name for nested?)
• robert: delta-rs with kernel log replay
  • delta-rs lets you serialize snapshots
  • do we want Snapshot to be serializable?
  • zach: note this gets harder for catalogManaged - if there is inline EngineData -> need it to be serialiazable
  • ParsedLogPath<Location = FileMeta>