2023-09-21 kernel meeting notes

[zachschuermann]

summary

PR workflow defined: wait for 2 reviews, please ping for more attention. Assign @nicklan when in doubt. And bring up in weekly sync if PRs aren't getting enough eyes.

action items:

• list of connectors to target (add Clickhouse and pytorch)
• look into stacked PRs (graphite, can we use internal tool, etc.)
• create issue outlining goals before public release

attendees

@rtyler @wjones127 @roeap @tdas @ryan-johnson-databricks @nicklan @MrPowers @zachschuermann

notes stream

• discussion for PR workflow. ask: 2 reviews. robert: need to make sure we have active involvement
• tyler: 2 reviewers + timeout?
• will: from before - one databricks one external?
  • would be especially helpful to have someone review expressions PR if someone knows
  • tyler: if we can get JVM side involved too.
• stacked PRs. small PRs are good!

@MrPowers items

• October 3rd meeting. Tyler, Zach, Nick and I will be at the Databricks SF office and Ryan/Will/TD will dial in from 10:30-12 Pacific. The in-person folks will grab lunch after and then do some informal brainstorming in the afternoon.
• Can we think about making the first deliverable a way to read Delta tables via PyArrow interfaces? This will instantly give us a better connector for Polars, Dask, DataFusion, PyTorch, Ray, etc.
• Can we name the repo so it aligns with the PyPI package name? Perhaps deltakernel instead of delta-kernel-rs? I don’t like how we append -rs to these repo names. That’s an implementation detail that I usually want to hide from end users.
  • currently a private repo so doesn't matter?
  • leaving it as it is now
  • note distinction between repo name vs crate name
• ClickHouse could be a good C FFI dogfood customer. (covered this from before - have an AI above for tracking these somewhere)
  • from zach: curious if they have extension API (like DuckDB) or if we have to upstream our changes.
• Path to reader version 3, writer version 7 transaction log protocol support delta-rs#1568

from tyler: tables produced by databricks (DLT) don't have schemaString and aren't able to be read by delta-rs, etc.

• some databricks users have non-protocol-compliant tables
• how do we support?
• first commit by DLT is not valid by protocol (000..000.json). metadata action on DLT is wrong. has metadata but doesn't have a schemaString
• JVM kernel will fail to read these tables, delta-rs will fail to read these tables (actually @rtyler fixed it with a hack that pretends it's an empty schema)
• ask for documented/corrected release processes for DBR
• tyler concern: feature that is 'experimental' in delta but shipped via DBR
• Todo discuss next time
• spark people OSS use 2.4, DBR does 3.0.0 RC?