2024-08-22 kernel meeting notes

[zachschuermann]

summary

light meeting. discussion: variant, collations, writes. pushing write review to next week when we have more attendance :)

pull requests/ongoing threads

• feat: add typed table configuration #309 (no updates - zach to review)
• Error reading from delta table with >= 100 commits #308 (no updates)

attendees

@zachschuermann @hntd187 @ryan-johnson-databricks

notes stream

• VARIANT discussion (early work from Steve)

collations

• collations: delta sharing server needs them. (need to pass them through correctly)
• implications: until we support they can't do data skipping on collated columns
• schema has collation attached to certain string columns (utf-8/binary)
• engnine free to take column and say 'i understand collations' or just say 'give me the utf-8/binary collations'
• kernel could say min/max on utf-8/binary collations or case-insensitive
• breaking API change
• for now: well-defined and we have a path forward
• existing clients who drop in a new .so should continue to work (and default should be now lack of collations means utf-8/binary)
• for new code we fail

write discussion

• API review, design sketch review