2023-08-10 kernel meeting notes

[zachschuermann]

summary

main java threads:

• Scott log replay PR: [Kernel] Improved LogReplay logic using column pruning + ColumnarBatch-based design delta#1939
• timestamp design coming soon (they are a pain)

main rust threads:

• @schuermannator to connect with duckDB folks
• still attempting to derisk design for async FFI
• two PRs in delta-rs that are pertinent
  1. @roeap refactor which will help kernel integration
  2. @fvaleye removal of tokio from python bindings

attendees

@rtyler @tdas @MrPowers @roeap @vkorukanti @nkarpov @schuermannator

notes stream

@vkorukanti - Java updates

• scott working on new log replay PR
• identified bottlenecks in parquet reader
• q from robert: do we have benchmarks to compare e.g. Arrow parquet vs java one
• time is a pain in the ass - spark does INT96
• since we are working with timestamp handling code, useful for rust side is if it is made very explicit. Ask: can we create a mini-spec or DAT tests for timestamp with(out) time zone etc.
• errors experience on delta-rs side: Arrow has one set of defaults for what a numeric value that should be a timestamp should be converted into - this isn't compatible with what Spark/Delta do by default. make explicit how to interpret numerals as timestamp.
• what should be in-memory representation for timestamp with timezone (delta spec says in parquet it must be microseconds in UTC) but what is not defined (TODO) is how do you read from int/numeral in parquet and expose to user in the 'correct' way (in local TZ etc.)
  • two choices (1) connector passes local TZ to kernel and kernel does transformation (doesn't matter if partition col or not), or (2) kernel exposes it as just INT (microsec UTC) and then client has to handle. one challenge is Spark legacy behavior: data column spark applies TZ correctly, but if it is a partition column it doesn't (TD check i captured this correctly lol)
• from robert and tyler: keep utc as much as possible

A few design decisions

• timestamp support (doc coming soon from Alli: TODO link doc)

rust side

@roeap no major updates

• one meeting with Kusto
• query engine written in rust - one option: ask them to adopt kernel and contribute to it
• ask: thoughts? seems a good idea (thumbs up from TD/tyler et al)
• tyler expresses concern regarding commitment
• also OneLake guys are interested
  @schuermannator async/FFI
• duckDB prototype: for async ffi check [async_ffi][https://docs.rs/async-ffi/latest/async_ffi/]
• we have people that are comfortable with C(++) at an arms length
• we want to expose FFI that is usable for C++ guys
• can we get DuckDB user feedback?
• what about reactor/runtime for async FFI? we need a 'run loop' on engine side. tyler supposition: duckDB or redshift would leverage epoll or kevent (or maybe io_uring now??).
• risk: the runtime
• robert thought: we are going to be kinda IO bound so async is good?
• homework: man epoll (and learn about io_uring if you want extra credit)
• can we normalize the interface we have previously discussed in java land and rust Stream
• should we diagram threading model?
• zach get duckDB contact (how do they handle async in duckDB?)
• current thought is no runtime (tokio etc.) in rust and expose futures over boundary.
• discussing last time: Stream work allows us to implement async interfaces but without runtime.
• for delta-rs they can continue to depend on tokio (or get rid of it but someone in rust will) and just advance our futures with that
• zach thought: implication is this means ALL clients must have a runtime (and perhaps roll their own 'runtime' even if it is simple 'run loop')
• note: two big PRs in delta-rs one from robert and one from florian. robert's refactored delta-rs in a good way to prepare for kernel integration. florian started one which removes tokio from python side and uses python's 'run loop'