2025-08-07 kernel meeting notes

[zachschuermann]

Summary

0.14.0 released!

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)
• catalogManaged reads
• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

upcoming

• zach: metrics
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory
• Look into exception unwind from foreign code
• do we need to update protocol for resolving field ids? in col mapping id mode?
• @rtyler fixed delta-rs checkpoint issue (follow up to check on our side)

Attendees

@zachschuermann @OussamaSaoudi @joonyoo181 @roeap @mightyshazam @scovich

notes stream

• arrow 56 upgrade / MSRV 1.82 -> 1.84? dropping 54? -> Zach to update readme with arrow version expectations. everyone LGTMs
• ObjectStore removal? FFI changes: we only do Url passed to make engine.
  • passing Arc<dyn ObjectStore> over FFI
  • ObjectStore is async: hard across FFI
  • we could define our own FileIO: subset of ObjectStore APIs (sync)
  • clickhouse/kyjah/etc. desire for FileIO layer
  • custom object store over FFI
  • define sub-trait for GET, LIST, HEAD (and get range) --> sync vtable, wrap up sync behind async
  • if backing impl is sync then you use async as glorified thread pool
  • if we have two async runtimes - complex
  • TODO learn asks of different engines - what surface do we want to optimize
  • main ask is permissions/credentials not perf
  • note: tokio is good enough async runtime, just let tokio be thread pool, give us sync methods to do the magic you need - if it becomes problem we can optimize later.
• Remove executor traits etc.?
  • remove and let people make their own DefaultEngine if they really care
  • what about no_std/embedded?
• native-TLS removal?
  • to follow up
  • steve/kyjah advocate for keeping native tls
  • but then maybe advocate for 'rustls' being default choice unless needed
• @roeap log replay take 3! update: PR is up with tests green!! will have follow ups for DV/col mapping
• zach following up on the checkpoint schema read issue!
• kyjah: ask on FFI for custom Engine impl

leftovers

• kernel side follow up: to check on checkpoint data iterator shouldn't return batches with different schemas #1099
• follow up on Urls discussion from last time.