Skip to content

Commit 8a3985e

Browse files
authored
Update controlling_sync.md
1 parent 05aa4b5 commit 8a3985e

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

docs/integrations/data-ingestion/clickpipes/postgres/controlling_sync.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,32 +20,40 @@ Database ClickPipes have an architecture that consists of two parallel processes
2020
There are two main ways to control the sync of a database ClickPipe. The ClickPipe will start pushing when one of the below settings kicks in.
2121

2222
### Sync interval {#interval-pg-sync}
23+
2324
The sync interval of the pipe is the amount of time (in seconds) for which the ClickPipe will pull records from the source database. The time to push what we have to ClickHouse is not included in this interval.
2425

2526
The default is **1 minute**.
2627
Sync interval can be set to any positive integer value, but it is recommended to keep it above 10 seconds.
2728

2829
### Pull batch size {#batch-size-pg-sync}
30+
2931
The pull batch size is the number of records that the ClickPipe will pull from the source database in one batch. Records mean inserts, updates and deletes done on the tables that are part of the pipe.
3032

3133
The default is **100,000** records.
3234
A safe maximum is 10 million.
3335

3436
### An exception: Long-running transactions on source {#transactions-pg-sync}
37+
3538
When a transaction is run on the source database, the ClickPipe waits until it receives the COMMIT of the transaction before it moves forward. This with **overrides** both the sync interval and the pull batch size.
3639

3740
### Configuring sync settings {#configuring-pg-sync}
41+
3842
You can set the sync interval and pull batch size when you create a ClickPipe or edit an existing one.
3943
When creating a ClickPipe it will be seen in the second step of the creation wizard, as shown below:
44+
4045
<Image src={create_sync_settings} alt="Create sync settings" size="md"/>
4146

4247
When editing an existing ClickPipe, you can head over to the **Settings** tab of the pipe, pause the pipe and then click on **Configure** here:
48+
4349
<Image src={edit_sync_button} alt="Edit sync button" size="md"/>
4450

4551
This will open a flyout with the sync settings, where you can change the sync interval and pull batch size:
52+
4653
<Image src={edit_sync_settings} alt="Edit sync settings" size="md"/>
4754

4855
### Tweaking the sync settings to help with replication slot growth {#tweaking-pg-sync}
56+
4957
Let's talk about how to use these settings to handle a large replication slot of a CDC pipe.
5058
The pushing time to ClickHouse does not scale linearly with the pulling time from the source database. This can be leveraged to reduce the size of a large replication slot.
5159
By increasing both the sync interval and pull batch size, the ClickPipe will pull a whole lot of data from the source database in one go, and then push it to ClickHouse.

0 commit comments

Comments
 (0)