2023-11-16 kernel meeting notes

[zachschuermann]

summary

@nicklan went over short-term goals for the project. Goals will be tracked in #42

action items:

• review list of immediate actions and put in Public release goals #42

attendees

@wjones127 @nicklan @MrPowers @zachschuermann @roeap @ryan-johnson-databricks @tdas

notes stream

quick @roeap question:

• arbitrary string that is JSON? happens, but should be parseable (at least into Map<String, String>)
• in metadata, schema string, etc.

@nicklan

• made a list of stuff to hit in the near-term
• trying to target 'basic read support' before opening repository
• go over list (todo paste)

discussion on the list

• protocol and metadata: can't do anything meaningful without knowing P+M
• what is the use case for lazy?
• eager on P+M in standalone, downside though is that if you want to extend it (add more functionality) then you bring in eager evaluation
• could be nice if construction of the object doesn't require P+M eagerly
• use case: Flink uses standalone but we want it to use kernel (bit of a hack)
• don't return option: must always have protocol and metadata
• snapshot object does eager P+M
• high level thoughts from matthew: would like to have infra set up for IC work
• @wjones127 LGTMed: suggesting putting into github issues (probably Public release goals #42)
• @ryan-johnson-databricks posting columnvector PR review
• question: how do we get zero-overhead logging? macros!