2025-07-03

[OussamaSaoudi]

Summary

0.12.0 released, and 0.12.1 (probably) coming very soon! (fixing azure issues)

current PRs

• IN/NOT IN PR to merge @roeap (waiting on opaque expressions. then will go in to default engine)
• !!!! opaque expressions @scovich
  • arrow opaque expressions need another level of indirection
  • need an 'arrow expression'
• #[derive(IntoEngineData)] (zach)
• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)

upcoming

• zach: metrics PR!
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory

Attendees

@nicklan @scovich @mightyshazam @joonyoo181

notes stream

• @mightyshazam Compilation issues with default engine rustls + ffi. This can be fixed with a breaking change.
• @nicklan We're ammenable to breaking changes
• @OussamaSaoudi How do we represent column id parquet reads?
• @nicklan This is part of the parquet reading contract, so we'd have to alter the parquet read APIs. Try to avoid having mistakes on the engine-side.
• @roeap there should not be a "fallback". It's either an id read or a physical name read.
• @roeap We need to track feature support publicly. Possibly a pinned issue. Make it clear the set of features that kernel supports.
• @nicklan Let's not promise timelines, just show the status of what's actively in development.
• @roeap Track features in delta-rs that can be easily offloaded to kernel.
• @nicklan There's an issue with the checkpoint schema. Different versions of delta-lake has different schemas for the checkpoint. Old versions of delta-lake sets everything is nullable. But old versions of delta-lake can't read tables that kernel writes because it has tighter schema nullability.
• @roeap I have a different concern with checkpoints. We don't do stats as struct. There's a delta configuration that lets you do stats as struct where stats are written as a fully-expanded struct instead of a JSON string. Kernel currently doesn't support this. Delta-rs has supported this, but delta-OSS doesn't even support this.
  • Ramification: The stats as struct may be lost by kernel, which could lead to performance regressions after checkpointing.
  • Java kernel may also support parsed stats.
• @roeap casting non-nullable to nullable field is relatively easy, so that may be a quick and easy fix.
• @roeap Does delta-spark have the same issue with reading kernel-written checkpoints?
• @nicklan The issue comes from arrow being inflexible about schema nullability mismatch (even in the safe direction).
• @nicklan How do we address the concern with regressions when incorporating kernel into delta-rs?
• @OussamaSaoudi tests where we read/write from different engines will be valuable.
• @roeap DAT doesn't support write tests.
• @nicklan kernel won't be able to do CHECK constraints directly, but we'd likely expose it through the schema.
• @roeap Just like how parquet has a bunch of non-compliant tables, we may need to make a decision on what where we deviate from the spec.