Skip to content

Commit 1cb499e

Browse files
iskakaushikiskakaushik
authored
Add documentation for Aurora PostgreSQL CDC integration (#3548)
1 parent f061706 commit 1cb499e

File tree

3 files changed

+142
-6
lines changed

3 files changed

+142
-6
lines changed

‎docs/integrations/data-ingestion/clickpipes/postgres/index.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,17 +34,19 @@ To get started, you first need to make sure that your Postgres database is set u
3434

3535
1. [Amazon RDS Postgres](./postgres/source/rds)
3636

37-
2. [Supabase Postgres](./postgres/source/supabase)
37+
2. [Amazon Aurora Postgres](./postgres/source/aurora)
3838

39-
3. [Google Cloud SQL Postgres](./postgres/source/google-cloudsql)
39+
3. [Supabase Postgres](./postgres/source/supabase)
4040

41-
4. [Azure Flexible Server for Postgres](./postgres/source/azure-flexible-server-postgres)
41+
4. [Google Cloud SQL Postgres](./postgres/source/google-cloudsql)
4242

43-
5. [Neon Postgres](./postgres/source/neon-postgres)
43+
5. [Azure Flexible Server for Postgres](./postgres/source/azure-flexible-server-postgres)
4444

45-
6. [Crunchy Bridge Postgres](./postgres/source/crunchy-postgres)
45+
6. [Neon Postgres](./postgres/source/neon-postgres)
4646

47-
7. [Generic Postgres Source](./postgres/source/generic), if you are using any other Postgres provider or using a self-hosted instance
47+
7. [Crunchy Bridge Postgres](./postgres/source/crunchy-postgres)
48+
49+
8. [Generic Postgres Source](./postgres/source/generic), if you are using any other Postgres provider or using a self-hosted instance
4850

4951

5052
:::warning

‎docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
---
2+
sidebar_label: 'Amazon Aurora Postgres'
3+
description: 'Set up Amazon Aurora Postgres as a source for ClickPipes'
4+
slug: /integrations/clickpipes/postgres/source/aurora
5+
title: 'Aurora Postgres Source Setup Guide'
6+
---
7+
8+
import parameter_group_in_blade from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/parameter_group_in_blade.png';
9+
import change_rds_logical_replication from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/change_rds_logical_replication.png';
10+
import change_wal_sender_timeout from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/change_wal_sender_timeout.png';
11+
import modify_parameter_group from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/modify_parameter_group.png';
12+
import reboot_rds from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/reboot_rds.png';
13+
import security_group_in_rds_postgres from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/security_group_in_rds_postgres.png';
14+
import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png';
15+
import Image from '@theme/IdealImage';
16+
17+
# Aurora Postgres Source Setup Guide
18+
19+
## Supported Postgres versions {#supported-postgres-versions}
20+
21+
ClickPipes supports Aurora PostgreSQL-Compatible Edition version 12 and later.
22+
23+
## Enable Logical Replication {#enable-logical-replication}
24+
25+
You can skip this section if your Aurora instance already has the following settings configured:
26+
- `rds.logical_replication = 1`
27+
- `wal_sender_timeout = 0`
28+
29+
These settings are typically pre-configured if you previously used another data replication tool.
30+
31+
```text
32+
postgres=> SHOW rds.logical_replication ;
33+
rds.logical_replication
34+
-------------------------
35+
on
36+
(1 row)
37+
38+
postgres=> SHOW wal_sender_timeout ;
39+
wal_sender_timeout
40+
--------------------
41+
0
42+
(1 row)
43+
```
44+
45+
If not already configured, follow these steps:
46+
47+
1. Create a new parameter group for your Aurora PostgreSQL version with the required settings:
48+
- Set `rds.logical_replication` to 1
49+
- Set `wal_sender_timeout` to 0
50+
51+
<Image img={parameter_group_in_blade} alt="Where to find Parameter groups in Aurora" size="lg" border/>
52+
53+
<Image img={change_rds_logical_replication} alt="Changing rds.logical_replication" size="lg" border/>
54+
55+
<Image img={change_wal_sender_timeout} alt="Changing wal_sender_timeout" size="lg" border/>
56+
57+
2. Apply the new parameter group to your Aurora PostgreSQL cluster
58+
59+
<Image img={modify_parameter_group} alt="Modifying Aurora PostgreSQL with new parameter group" size="lg" border/>
60+
61+
3. Reboot your Aurora cluster to apply the changes
62+
63+
<Image img={reboot_rds} alt="Reboot Aurora PostgreSQL" size="lg" border/>
64+
65+
## Configure Database User {#configure-database-user}
66+
67+
Connect to your Aurora PostgreSQL writer instance as an admin user and execute the following commands:
68+
69+
1. Create a dedicated user for ClickPipes:
70+
71+
```sql
72+
CREATE USER clickpipes_user PASSWORD 'some-password';
73+
```
74+
75+
2. Grant schema permissions. The following example shows permissions for the `public` schema. Repeat these commands for each schema you want to replicate:
76+
77+
```sql
78+
GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
79+
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
80+
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;
81+
```
82+
83+
3. Grant replication privileges:
84+
85+
```sql
86+
GRANT rds_replication TO clickpipes_user;
87+
```
88+
89+
4. Create a publication for replication:
90+
91+
```sql
92+
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
93+
```
94+
95+
96+
## Configure Network Access {#configure-network-access}
97+
98+
### IP-based Access Control {#ip-based-access-control}
99+
100+
If you want to restrict traffic to your Aurora cluster, please add the [documented static NAT IPs](../../index.md#list-of-static-ips) to the `Inbound rules` of your Aurora security group.
101+
102+
<Image img={security_group_in_rds_postgres} alt="Where to find security group in Aurora PostgreSQL?" size="lg" border/>
103+
104+
<Image img={edit_inbound_rules} alt="Edit inbound rules for the above security group" size="lg" border/>
105+
106+
### Private Access via AWS PrivateLink {#private-access-via-aws-privatelink}
107+
108+
To connect to your Aurora cluster through a private network, you can use AWS PrivateLink. Follow our [AWS PrivateLink setup guide for ClickPipes](/knowledgebase/aws-privatelink-setup-for-clickpipes) to set up the connection.
109+
110+
### Aurora-Specific Considerations {#aurora-specific-considerations}
111+
112+
When setting up ClickPipes with Aurora PostgreSQL, keep these considerations in mind:
113+
114+
1. **Connection Endpoint**: Always connect to the writer endpoint of your Aurora cluster, as logical replication requires write access to create replication slots and must connect to the primary instance.
115+
116+
2. **Failover Handling**: In the event of a failover, Aurora will automatically promote a reader to be the new writer. ClickPipes will detect the disconnection and attempt to reconnect to the writer endpoint, which will now point to the new primary instance.
117+
118+
3. **Global Database**: If you're using Aurora Global Database, you should connect to the primary region's writer endpoint, as cross-region replication already handles data movement between regions.
119+
120+
4. **Storage Considerations**: Aurora's storage layer is shared across all instances in a cluster, which can provide better performance for logical replication compared to standard RDS.
121+
122+
### Dealing with Dynamic Cluster Endpoints {#dealing-with-dynamic-cluster-endpoints}
123+
124+
While Aurora provides stable endpoints that automatically route to the appropriate instance, here are some additional approaches for ensuring consistent connectivity:
125+
126+
1. For high-availability setups, configure your application to use the Aurora writer endpoint, which automatically points to the current primary instance.
127+
128+
2. If using cross-region replication, consider setting up separate ClickPipes for each region to reduce latency and improve fault tolerance.
129+
130+
## What's next? {#whats-next}
131+
132+
You can now [create your ClickPipe](../index.md) and start ingesting data from your Aurora PostgreSQL cluster into ClickHouse Cloud.
133+
Make sure to note down the connection details you used while setting up your Aurora PostgreSQL cluster as you will need them during the ClickPipe creation process.

‎sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -669,6 +669,7 @@ const sidebars = {
669669
label: "Source",
670670
items: [
671671
"integrations/data-ingestion/clickpipes/postgres/source/rds",
672+
"integrations/data-ingestion/clickpipes/postgres/source/aurora",
672673
"integrations/data-ingestion/clickpipes/postgres/source/supabase",
673674
"integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql",
674675
"integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres",

0 commit comments

Comments
 (0)