Skip To Initial Copy Data

[dwickyfp]

Feature Request

Currently, when setting up the data pipeline/ETL process, the system automatically attempts to perform an initial sync (snapshot) of the existing data in the table.

For use cases involving large datasets or high-throughput tables, this initial copy is expensive, time-consuming, and unnecessary when the goal is solely to react to new events (real-time triggers) or capture data changes (CDC) starting from the current moment. There is no easy way to opt-out of this historical backfill.

I would like to request a configuration option to skip the initial snapshot/copy.

[iambriccardo]

Hi, thanks for the issue. This is something that we want to support for sure.

Would you like this to be selective, aka per table or as a whole?

[dwickyfp]

My current immediate need is to configure this per pipeline (for the whole connector/job).

However, I agree that making it selective per table would be the superior approach for flexibility in the long run.

Ultimately, whether it is implemented at the table level or pipeline level, the core priority is simply having the capability to opt-out of the initial snapshot/backfill.

[iambriccardo]

This can be easily achieved with the current system, a way to do this would be to have a config param like skip_table_copy and if this is found, all table states are initialized by default to Ready instead of Init. The nice thing about this is that it can easily be made super flexible by implementing per-table skipping.

How does this sound to you?

[iambriccardo]

The only thing that we need to do is figure out how to do schema syncing properly. I will figure out a way to cleanly implement this as soon as I have the time.

Thanks for the proposal!

[dwickyfp]

That sounds fantastic! I completely agree with that approach. Using a config like skip_table_copy to initialize the table state directly to Ready seems like a clean and flexible solution.

Just to confirm one technical detail: by doing this, can we be sure that it completely skips the initial copy process? Specifically, does it ensure that the system skips creating the replication slot for initial data and skips the actual data copy operation entirely?

I am really looking forward to seeing this feature implemented soon!

[iambriccardo]

The implementation will likely be along these lines:

• ETL starts and detects all tables from the publication.
• It kickstarts parallel workers using all the limiting mechanism.
• Sees that it should not perform any table copy, thus it just loads the schema in a consistent way and starts immediately streaming.

This way, the system will immediately switch to syncing on the main worker and the table schema copy will be consistent.

We can already implement a mechanism inside that allows more specific sync patterns:

• SkipAllTables
• IncludeTables
• ExcludeTables

[dwickyfp]

I'm really looking forward to it, thank you

[iambriccardo]

PR is here: #537