Skip to content

Incremental sink coordination #228

@devinrsmith

Description

@devinrsmith

To support the deephaven-core use case of parsing a CSV into a blink table, it would be nice to add some sort of "sink coordination" options. For example, I might want the CSV reader to coordinate a "flush" every 100k rows. This would be a signal that all of the sinks have been written to and are "in-sync". (This does not mean we need to call Sink#write with arrays of that size, just that there needs to be a common coordination point.)

From an implementation standpoint, I suspect we would want this to limit how far ahead the reader gets as well. One of the main points of going into a blink table is that you can filter the data and limit how much memory you need to use.

It's unclear right now if this feature should require the caller to forgo inference.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions