2025-07-24 kernel meeting notes

[zachschuermann]

Summary

0.14.0 inbound :) and @rtyler fixed delta-rs checkpoint issue (follow up to check on our side)

current PRs

• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)
• catalogManaged reads

upcoming

• zach: metrics
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory
• Look into exception unwind from foreign code
• do we need to update protocol for resolving field ids? in col mapping id mode?

Attendees

@zachschuermann @nicklan @joonyoo181 @hntd187 @rtyler

notes stream

• @rtyler finally fixed checkpoint issue in delta-rs!
  • see issue: Checkpoint schema breaking change between 0.25.5 and 1.0.2 delta-rs#3527 (comment)
  • delta-rs fix: delta-io/delta-rs@497a70b
• kernel side: need to check on checkpoint_data returning batches with two different schemas
  • opened: checkpoint data iterator shouldn't return batches with different schemas #1099
• @roeap log replay take 3!
• strings are original sin in delta-rs
• need to figure out LogSegment construction (tables with spaces bad - S3 tests failing, some encoding wrong)
• kernel tests: should do shallow clone table with local files or Url to someone else
  • probably doesn't work in kernel/delta-rs currently
  • @nicklan working on some Url pathfinding :)
  • lots of things here: (1) ADLS Urls are weird and have container name sometimes - related to something @rtyler looked at (object store issue?) - we have kernel issue https:// URLs are broken on Azure #1007, (2) need engine creation separate from table creation, (3) tyler raised test case: shallow clone table with local files + Url to somewhere else
• context: delta-rs does parse_url_opts -> object store impl per table
  • object stores, prefix stores on top of them
  • object store impl = single table
  • DeltaTable { logstore { object_store } } --> kinda hard to share object store, not done frequently?
  • check out reqwest: build a Client -> then you can get a RequestBuilder. we could do EngineClient --> get back engines
    • could reuse heavy stuff (runtime etc) but new object store (lightweight)
  • make sure we retain ability to bring own object store
    • builder works well here! engine for objectstore, engine for url, etc.
• should kernel hand back full Urls to table? and handle the mess of figuring out what was in the log (data file paths)? OR should we just hand back raw paths in log and engine joins with table root that they (ostensibly) know about