2025-08-14 kernel meeting notes

[zachschuermann]

Summary

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)
• catalogManaged reads
• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

upcoming

• zach: metrics
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory
• Look into exception unwind from foreign code
• do we need to update protocol for resolving field ids? in col mapping id mode?
• @rtyler fixed delta-rs checkpoint issue (follow up to check on our side)

Attendees

@zachschuermann @roeap @mightyshazam @scovich @hntd187 @OussamaSaoudi

notes stream

• delta-rs perf issues (transforms)
  • workload: scan_metadata (just producing scan files, no data reads)
  • table: 300k add files, most are in a checkpoint (28MB), 100 col
  • problem: O(N * M) processing for transforms, N = # add files, M = # columns
  • (1) validate transforms are the issue, (2) come up with solution - strawman: split scan-const transform and file-const transform
  • potential per-row transfrom: parquet row index BUT we don't do those in kernel, reader does then as metadata column
  • is the problem: (1) column rearrange/schema traverse or (2) per-row-per-column overhead - it is during transform creation (per-scan-file)
  • try instruments app
  • tyler: for every row (of scan file) iterate over every transform for every column
  • addremovededup row visitor (is_valid_add)
  • transform code:

    delta-kernel-rs/kernel/src/scan/log_replay.rs

    Lines 242 to 246 in f4c6e54

    	let transform = self
    	.transform
    	.as_ref()
    	.map(|transform| self.get_transform_expr(transform, partition_values))
    	.transpose()?;

  • partition values are per-row value in metadata
  • map to strong type is hard
  • how can we unblock in the short term?
  • identity transform implied instead of stated? then use sparse transforms
  • "jump" table idea
• delta-rs old checkpoint iterator diff schemas
• 0.15.0 next week
• new snapshot builders, catalog-managed experimental support
• robert datafusion engine!

leftovers

• kernel side follow up: to check on checkpoint data iterator shouldn't return batches with different schemas #1099
• follow up on Urls discussion from last time.