2024-08-15 kernel meeting notes

[zachschuermann]

summary

ryan is back! some discussion on debugging, opaque expressions, kernel/delta-rs, write plan

pull requests

• First pass on IN/Not In #270
• feat: add typed table configuration #309

attendees

@zachschuermann @roeap @hntd187 @ryan-johnson-databricks @rtyler

notes stream

• config PR from robert: feat: add typed table configuration #309 (adapted from delta-rs)
• bug ongoing Error reading from delta table with >= 100 commits #308
• we need better debugability (zach to look into it - create an issue)
• @hntd187 coming up next after IN/NOT IN:
  • Partition Filters in delta-kernel (a lot of copy paste from delta-rs)
  • Once landed in kernel, copy pasted back into delta-rs
• expressions that engine supports but kernel doesn't know about? opaque expressions. downside: we don't get data skipping
• IN list: (1) could do data skipping (2) needed for partition filtering, IN is known for huge pile of literal values: engine terrirtory. parsing lots of data has perf/OOM issues.
• PR from robert in delta-rs: adopting more kernel. data skipping
• when we do log replay: log as arrow arrays. need to pick things (part vals, parsed stats) from data, just use kernel expression language to probe at the log data. delta-rs doesn't have expression system. using kernels
• tyler: can we get signal on what expressions exist in the wild? starting point instead of just 'all of spark SQL'
• timestamp/timezone has similar issues.
• expressions in generated columns longstanding issue
• robert: start on checkpoints (V2) in delta-rs
• state handling in delta-rs is in arrow.
• mini-log-replay for physical conflict
• this isn't yet based on kernel log parsing
• can we use expressions to figure out if conflicts occur?
• check with java kernel side!
• conflicts: attempt commit, physical conflict, back-and-forth protocol, engine to resolve conflicts between version x and y, given that, then kernel will commit something else.
• to follow up
• blind appends: caution - serializability failures if you let blind appends cross metadata updates etc. check with Prakhar
• you have to read the schema of the table at a minimum: i.e. blind append is not blind