2023-08-03 kernel meeting notes

[zachschuermann]

summary

Updates on Java side

• Java examples using Kernel APIs: [Kernel] Add examples for Delta Kernel API usage delta#1926
• API user guide PR is in progress, should be merged today: [Kernel] API user guide delta#1927
• Efficient LogReplay PR in progress: [Kernel] Improved LogReplay logic using column pruning + ColumnarBatch-based design delta#1939

Updates on rust side

• couple PRs landed: feat: read _last_checkpoint #19 test: extend DAT tests and sharable test setup #20
• need to derisk FFI: DuckDB prototype. async is hard over FFI

attendees

@tdas @MrPowers @roeap @vkorukanti @nkarpov @schuermannator

notes stream

@vkorukanti - Java updates

• Java examples using Kernel APIs: [Kernel] Add examples for Delta Kernel API usage delta#1926
• API user guide PR is in progress, should be merged today: [Kernel] API user guide delta#1927
• Efficient LogReplay PR in progress: [Kernel] Improved LogReplay logic using column pruning + ColumnarBatch-based design delta#1939

A few design decisions

• exposing the complex type column values from ColumnVector interface
• expression interface
• quirks around timestamp support

Ex: getting scan files from kernel: iterator of ColumnVec but requires kernel rewriting col batch after partition pruning but we will expose col batch and selection vector instead. This exists in two places: (1) data side: apply DVs and (2) metadata side: filtering the actions to produce a list of scan files. If we don't do that we have to rewrite col batches - this is likely simpler to express the scan files as the 'old' checkpoint/commits/etc. with a selection vector. Partition pruning, file skipping, etc updates the selection vector.

Venki will continue to prepare/update JVM side in this meeting.
From @tdas: any non-trivial compute must exist outside the kernel.

@tdas - LogReplay PR

• Scott working on PR now
• Expose things as columnar batches + selection vectors
• We can leverage the JVM side as starting point for rust implementation

@roeap @schuermannator rust side

• feat: read _last_checkpoint #19
• test: extend DAT tests and sharable test setup #20
• DuckDB FFI experiment @schuermannator async difficulty
• different cloud storage
• from @roeap check on PyO3
• from @tdas if we are taking decisions that are blocking us from building certain connectors we are doing it wrong
• from delta-rs side it is better to have async
• push/pull on async: IO is obviously often async in rust but there are implications of requiring a runtime/reactor