Skip to content

Commit 40a3fc9

Browse files
authored
Update controlling_sync.md
1 parent f6c71f2 commit 40a3fc9

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

docs/integrations/data-ingestion/clickpipes/mysql/controlling_sync.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import edit_sync_button from '@site/static/images/integrations/data-ingestion/cl
99
import create_sync_settings from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/create_sync_settings.png'
1010
import edit_sync_settings from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/sync_settings_edit.png'
1111
import cdc_syncs from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/cdc_syncs.png'
12+
import Image from '@theme/IdealImage';
1213

1314
This document describes how to control the sync of a database ClickPipe (Postgres, MySQL etc.) when the ClickPipe is in **CDC (Running) mode**.
1415

@@ -19,32 +20,40 @@ Database ClickPipes have an architecture that consists of two parallel processes
1920
There are two main ways to control the sync of a database ClickPipe. The ClickPipe will start pushing when one of the below settings kicks in.
2021

2122
### Sync interval {#interval-mysql-sync}
23+
2224
The sync interval of the pipe is the amount of time (in seconds) for which the ClickPipe will pull records from the source database. The time to push what we have to ClickHouse is not included in this interval.
2325

2426
The default is **1 minute**.
2527
Sync interval can be set to any positive integer value, but it is recommended to keep it above 10 seconds.
2628

2729
### Pull batch size {#batch-size-mysql-sync}
30+
2831
The pull batch size is the number of records that the ClickPipe will pull from the source database in one batch. Records mean inserts, updates and deletes done on the tables that are part of the pipe.
2932

3033
The default is **100,000** records.
3134
A safe maximum is 10 million.
3235

3336
### An exception: Long-running transactions on source {#transactions-pg-sync}
37+
3438
When a transaction is run on the source database, the ClickPipe waits until it receives the COMMIT of the transaction before it moves forward. This with **overrides** both the sync interval and the pull batch size.
3539

3640
### Configuring sync settings {#configuring-mysql-sync}
41+
3742
You can set the sync interval and pull batch size when you create a ClickPipe or edit an existing one.
3843
When creating a ClickPipe it will be seen in the second step of the creation wizard, as shown below:
44+
3945
<Image src={create_sync_settings} alt="Create sync settings" size="md"/>
4046

4147
When editing an existing ClickPipe, you can head over to the **Settings** tab of the pipe, pause the pipe and then click on **Configure** here:
48+
4249
<Image src={edit_sync_button} alt="Edit sync button" size="md"/>
4350

4451
This will open a flyout with the sync settings, where you can change the sync interval and pull batch size:
52+
4553
<Image src={edit_sync_settings} alt="Edit sync settings" size="md"/>
4654

4755
### Monitoring sync control behaviour {#monitoring-mysql-sync}
56+
4857
You can see how long each batch takes in the **CDC Syncs** table in the **Metrics** tab of the ClickPipe. Note that the duration here includes push time and also if there are no rows incoming, the ClickPipe waits and the wait time is also included in the duration.
4958

5059
<Image src={cdc_syncs} alt="CDC Syncs table" size="md"/>

0 commit comments

Comments
 (0)