Skip to content

Commit e0df17f

Browse files
authored
Merge pull request #3802 from ClickHouse/postgres-faq-openapi-beta
[clickpipes] update Postgres FAQ to highlight OpenAPI beta
2 parents 5ce9f33 + 1f520f1 commit e0df17f

File tree

1 file changed

+12
-5
lines changed
  • docs/integrations/data-ingestion/clickpipes/postgres

1 file changed

+12
-5
lines changed

docs/integrations/data-ingestion/clickpipes/postgres/faq.md

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ In summary, while sync and normalize processes are terminated during a pause, it
131131
132132
### Can ClickPipe creation be automated or done via API or CLI? {#can-clickpipe-creation-be-automated-or-done-via-api-or-cli}
133133
134-
As of now, you can create a ClickPipe only via the UI. However, we are actively working on exposing OpenAPI and Terraform endpoints. We expect this to be released in the near future (within a month). If you are interested in becoming a design partner for this feature, please reach out to [email protected].
134+
A Postgres ClickPipe can also be created and managed via [OpenAPI](https://clickhouse.com/docs/cloud/manage/openapi) endpoints. This feature is in beta, and the API reference can be found [here](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/beta). We are actively working on Terraform support to create Postgres ClickPipes as well.
135135
136136
### How do I speed up my initial load? {#how-do-i-speed-up-my-initial-load}
137137
@@ -147,7 +147,7 @@ These adjustments should significantly enhance the performance of the initial lo
147147
148148
### How should I scope my publications when setting up replication? {#how-should-i-scope-my-publications-when-setting-up-replication}
149149
150-
You can let ClickPipes manage your publications (requires write access) or create them yourself. With ClickPipes-managed publications, we automatically handle table additions and removals as you edit the pipe. If self-managing, carefully scope your publications to only include tables you need to replicate - including unnecessary tables will slow down Postgres WAL decoding.
150+
You can let ClickPipes manage your publications (requires additional permissions) or create them yourself. With ClickPipes-managed publications, we automatically handle table additions and removals as you edit the pipe. If self-managing, carefully scope your publications to only include tables you need to replicate - including unnecessary tables will slow down Postgres WAL decoding.
151151
152152
If you include any table in your publication, make sure it has either a primary key or `REPLICA IDENTITY FULL`. If you have tables without a primary key, creating a publication for all tables will cause DELETE and UPDATE operations to fail on those tables.
153153
@@ -168,17 +168,24 @@ You have two options when dealing with tables without primary keys:
168168
1. **Exclude tables without primary keys from ClickPipes**:
169169
Create the publication with only the tables that have a primary key:
170170
```sql
171-
CREATE PUBLICATION my_publication FOR TABLE table_with_primary_key1, table_with_primary_key2, ...;
171+
CREATE PUBLICATION clickpipes_publication FOR TABLE table_with_primary_key1, table_with_primary_key2, ...;
172172
```
173173
174174
2. **Include tables without primary keys in ClickPipes**:
175175
If you want to include tables without a primary key, you need to alter their replica identity to `FULL`. This ensures that UPDATE and DELETE operations work correctly:
176176
```sql
177177
ALTER TABLE table_without_primary_key1 REPLICA IDENTITY FULL;
178178
ALTER TABLE table_without_primary_key2 REPLICA IDENTITY FULL;
179-
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
179+
CREATE PUBLICATION clickpipes_publication FOR TABLE <...>, <...>;
180180
```
181181
182+
:::tip
183+
If you're creating a publication manually instead of letting ClickPipes manage it, we don't recommend creating a publication `FOR ALL TABLES`, this leads to more traffic from Postgres to ClickPipes (to sending changes for other tables not in the pipe) and reduces overall efficiency.
184+
185+
For manually created publications, please add any tables you want to the publication before adding them to the pipe.
186+
:::
187+
188+
182189
## Recommended `max_slot_wal_keep_size` Settings {#recommended-max_slot_wal_keep_size-settings}
183190
184191
- **At Minimum:** Set [`max_slot_wal_keep_size`](https://www.postgresql.org/docs/devel/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE) to retain at least **two days' worth** of WAL data.
@@ -257,7 +264,7 @@ Note that if you're working with partitioned tables, make sure to create your pu
257264

258265
```sql
259266
CREATE PUBLICATION clickpipes_publication
260-
FOR TABLE table_1, table_2, ...
267+
FOR TABLE <...>, <...>
261268
WITH (publish_via_partition_root = true);
262269
```
263270

0 commit comments

Comments
 (0)