You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/clickpipes/postgres/faq.md
+12-5Lines changed: 12 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -131,7 +131,7 @@ In summary, while sync and normalize processes are terminated during a pause, it
131
131
132
132
### Can ClickPipe creation be automated or done via API or CLI? {#can-clickpipe-creation-be-automated-or-done-via-api-or-cli}
133
133
134
-
As of now, you can create a ClickPipe only via the UI. However, we are actively working on exposing OpenAPI and Terraform endpoints. We expect this to be released in the near future (within a month). If you are interested in becoming a design partner for this feature, please reach out to [email protected].
134
+
A Postgres ClickPipe can also be created and managed via [OpenAPI](https://clickhouse.com/docs/cloud/manage/openapi) endpoints. This feature is in beta, and the API reference can be found [here](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/beta). We are actively working on Terraform support to create Postgres ClickPipes as well.
135
135
136
136
### How do I speed up my initial load? {#how-do-i-speed-up-my-initial-load}
137
137
@@ -147,7 +147,7 @@ These adjustments should significantly enhance the performance of the initial lo
147
147
148
148
### How should I scope my publications when setting up replication? {#how-should-i-scope-my-publications-when-setting-up-replication}
149
149
150
-
You can let ClickPipes manage your publications (requires write access) or create them yourself. With ClickPipes-managed publications, we automatically handle table additions and removals as you edit the pipe. If self-managing, carefully scope your publications to only include tables you need to replicate - including unnecessary tables will slow down Postgres WAL decoding.
150
+
You can let ClickPipes manage your publications (requires additional permissions) or create them yourself. With ClickPipes-managed publications, we automatically handle table additions and removals as you edit the pipe. If self-managing, carefully scope your publications to only include tables you need to replicate - including unnecessary tables will slow down Postgres WAL decoding.
151
151
152
152
If you include any table in your publication, make sure it has either a primary key or `REPLICA IDENTITY FULL`. If you have tables without a primary key, creating a publication for all tables will cause DELETE and UPDATE operations to fail on those tables.
153
153
@@ -168,17 +168,24 @@ You have two options when dealing with tables without primary keys:
168
168
1. **Exclude tables without primary keys from ClickPipes**:
169
169
Create the publication with only the tables that have a primary key:
170
170
```sql
171
-
CREATE PUBLICATION my_publication FOR TABLE table_with_primary_key1, table_with_primary_key2, ...;
171
+
CREATE PUBLICATION clickpipes_publication FOR TABLE table_with_primary_key1, table_with_primary_key2, ...;
172
172
```
173
173
174
174
2. **Include tables without primary keys in ClickPipes**:
175
175
If you want to include tables without a primary key, you need to alter their replica identity to `FULL`. This ensures that UPDATE and DELETE operations work correctly:
176
176
```sql
177
177
ALTER TABLE table_without_primary_key1 REPLICA IDENTITY FULL;
178
178
ALTER TABLE table_without_primary_key2 REPLICA IDENTITY FULL;
179
-
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
179
+
CREATE PUBLICATION clickpipes_publication FOR TABLE <...>, <...>;
180
180
```
181
181
182
+
:::tip
183
+
If you're creating a publication manually instead of letting ClickPipes manage it, we don't recommend creating a publication `FOR ALL TABLES`, this leads to more traffic from Postgres to ClickPipes (to sending changes for other tables not in the pipe) and reduces overall efficiency.
184
+
185
+
For manually created publications, please add any tables you want to the publication before adding them to the pipe.
- **At Minimum:** Set [`max_slot_wal_keep_size`](https://www.postgresql.org/docs/devel/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE) to retain at least **two days' worth** of WAL data.
@@ -257,7 +264,7 @@ Note that if you're working with partitioned tables, make sure to create your pu
0 commit comments