2023-09-28 kernel meeting notes

[zachschuermann]

summary

small meeting, synced on questions from @roeap and some misc discussion!

action items from last time:

• list of connectors to target (add Clickhouse and pytorch)
• look into stacked PRs (graphite, can we use internal tool, etc.)

attendees

@roeap @nicklan @MrPowers @zachschuermann @vkorukanti

notes stream

from @nicklan

• will review feat: support more expression types #26

from @roeap

• robert PR on top of Will's PR: client that creates the evaluator
• another PR for expression evaluator
• note java one does partition pruning but doesn't do pruning with stats
  • java issue: [Feature Request][Kernel] Support partition pruning in Kernel delta#2071
• depending on who creates the path, they might be different formats. hard to interact with object_store crate which has opinionated format/encodings.
• protocol: path should be URI encoded. does it specify exact character set or encodings etc.?
• double-encoding concern? we always encode when we write to actions, but it might have been encoded before.
• what characters does spark encode when it creates the partition file path?
• info: working on dagster, loves partitioning and can partition on timestamp

from @MrPowers

• Datafusion/Polars communities we need to target
• working with datafusion
• goal: we work with datafusion, gives delta support, then magically get delta support in influx, daskSQL magically gets delta support

from @zachschuermann

• scan.execute() returns a vector? Why not iterator?