-
Notifications
You must be signed in to change notification settings - Fork 14
Description
To support the deephaven-core use case of parsing a CSV into a blink table, it would be nice to add some sort of "sink coordination" options. For example, I might want the CSV reader to coordinate a "flush" every 100k rows. This would be a signal that all of the sinks have been written to and are "in-sync". (This does not mean we need to call Sink#write with arrays of that size, just that there needs to be a common coordination point.)
From an implementation standpoint, I suspect we would want this to limit how far ahead the reader gets as well. One of the main points of going into a blink table is that you can filter the data and limit how much memory you need to use.
It's unclear right now if this feature should require the caller to forgo inference.