Transactional Pipeline

[chubei]

Transactional Pipeline

The pipeline has ExecutorOperation::Commit now, but the commit message is generated by the SourceChannelManager, which tracks the number of uncommited operations and the time elapsed since the last commit. As a result, the commit does not necessarily align with source transaction boundaries.

In this proposal, we introduce IngestionMessage::Commit, which should be sent from the connector at the end of each source transaction. For the sources that do not support transactions, they should send IngestionMessage::Commit periodically, typically before beginning polling for new data.

Connector Example

Postgres

Postgres supports transactions, so in the replication phase. Postgres connector should send IngestionMessage::Commit when a transaction ends. In the snapshotting phase, Postgres connector should not send IngestionMessage::Commit.

Object store

Object store connector can send IngestionMessage::Commit after every file is ingested.