2024-10-10 kernel meeting notes

[zachschuermann]

summary

0.4 release delayed but still upcoming - need to plan/sync with delta-rs.

pull requests/ongoing threads

write path in progress, overall tracking issue at #377

• move transaction module into actions/ and rename to set_transaction #386
• [write] add Transaction with commit info and commit implementation #370

PRs

• nick semver things merged Adding cargo semver checks #369
• ryan's row group skipping merged Implement row group skipping for the default engine parquet readers #362
• note: deprioritizing Implement predicate pushdown for parquet reader #349
• expression visitor FFI in progress Add visitor for converting kernel expressions to engine expressions #363

attendees

@nicklan @zachschuermann @hntd187 @OussamaSaoudi-db @scovich

notes stream

• Semver checks merged!!
• didn't have any checks required but now we have required checks. how do we bypass if needed? admin would have to remove the 'required' then add it back
• sorting PR follow up: Ensure we return a sorted listing when using a local client #344
  • delta is designed with ordered listing in mind
  • proposal: merge PR and work on upstream object_store to expose whether or not list is ordered
  • note this PR should also change reverse sort to reverse iterator
  • on S3 side, except for directory buckets, everything has ordered listing and supports prefix list_from
    • DIRECTORY BUCKETS SUPPORT NEITHER
  • azure doesn't support ordering nor prefix list_from (woah?) they have some exact prefix matching
    • what does delta-spark do?
• quick PR board skim