kernel meeting notes 2025-04-17

[zachschuermann]

Summary

ongoing threads

• IN/NOT IN PR to merge @roeap
• predicate/expressions split @scovich (putting up a bunch of stacked commits - top-level feedback)
• merge opaque expressions @scovich
• merge IN/NOT IN deprecation @zachschuermann
• snapshot incremental API: Snapshot::try_new_from(Arc<Snapshot>) -> Arc<Snapshot> about to merge @zachschuermann
• @hntd187 targeting CDF reads, need timestamp work

Attendees

@hntd187 @zachschuermann @roeap @mightyshazam

notes stream

• going over open threads
• CREATE TABLE POC
• idempotent writes part 1 (with transaction id API)
  • part 2 coming soon is reading the transaction ids in snapshots SetTransactionScanner
  • TODO make this #[internal_api]
  • quick discussion on where to do app_id materialization
  • probably configure snapshot to track txn actions (if needed)
  • else you have to do another log replay
  • note app_ids are completely different than protocol + metadata
  • if cached, faster to go back to parquet for specific columns instead of just doing huge schema
  • engine is responsible for caching
  • note we could cache in default engine? need to make sure not to compete with the engine using default engine
• #[derive(IntoEngineData)] macro POC
• arrow 55
• writing file stats
• robert: incremental scan API
• sebastian: last 3 checkpoint write PRs