Skip to content

Commit 4985ecf

Browse files
refactor(clickpipes): rds private link docs (#3241)
* refactor(clickpipes): rds private link docs * chore: add newline
1 parent 642263d commit 4985ecf

File tree

2 files changed

+29
-13
lines changed

2 files changed

+29
-13
lines changed

docs/en/integrations/data-ingestion/clickpipes/postgres/faq.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ Yes! ClickPipes for Postgres offers two ways to connect to databases in private
4343
- us-east-1
4444
- us-east-2
4545
- eu-central-1
46-
- For detailed setup instructions, see our [PrivateLink documentation](https://clickhouse.com/docs/knowledgebase/aws-privatelink-setup-for-clickpipes#requirements)
46+
- For detailed setup instructions, see our [PrivateLink documentation](/docs/knowledgebase/aws-privatelink-setup-for-clickpipes#requirements)
4747
- For regions where PrivateLink is not available, please use SSH tunneling
4848

4949
### How do you handle UPDATEs and DELETEs?

knowledgebase/aws-privatelink-setup-for-clickpipes.mdx

Lines changed: 28 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -15,43 +15,59 @@ Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes.
1515

1616
## Requirements
1717

18-
The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1.
19-
(https://clickhouse.com/docs/en/integrations/clickpipes#list-of-static-ips)
18+
The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1
19+
(/docs/en/integrations/clickpipes#list-of-static-ips), additionally the ClickHouse Instance must
20+
be in the same region.
2021

2122
## Private link creation
2223

23-
Follow these steps to create a **VPC endpoint service** for your RDS instance. Repeat these steps if you have multiple instances that require endpoint services:
24+
Follow these steps to create a **VPC endpoint service** for your RDS instance. Repeat these steps if you have multiple
25+
RDS instances that require endpoint services (OR you may have different listener ports for different instances):
2426

2527
1. Locate Your VPC and [Create an NLB](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html)
26-
- Navigate to your target VPC and create a Network Load Balancer (NLB).
28+
- Navigate to your target VPC and create a Network Load Balancer (NLB). Note that the NLB should be internal (private) and not internet-facing (public).
2729

2830
2. Configure the Target Group
2931
- The target group should point to the RDS instance's endpoint IP and Port (typically 5432 for PostgreSQL or 3306 for MySQL).
30-
- Ensure that the TCP protocol is used to avoid TLS termination by the NLB.
32+
33+
:::note
34+
35+
If you would like to automate the process of updating the target group with the new RDS endpoint IP, you can use AWS Lambda functions or other automation tools.
36+
One of the terraform modules that can be used for this purpose is [this](https://github.com/MaterializeInc/terraform-aws-rds-privatelink).
37+
38+
:::
3139
- **IMPORTANT**: Make sure the RDS instance endpoint used in case of DB Cluster/Aurora is ONLY the WRITER Endpoint and NOT the common endpoint.
40+
- Ensure that the TCP protocol is used to avoid TLS termination by the NLB.
3241

3342
3. Set the Listener Port
3443
- The listener port of the load balancer must match the port used by the target group (typically 5432 for PostgreSQL or 3306 for MySQL).
3544

36-
4. Ensure the Load Balancer is Private
37-
- Configure the NLB to be private, ensuring it is only accessible within the VPC.
38-
39-
5. Create the VPC Endpoint Service
45+
4. Create the VPC Endpoint Service
4046
- In the VPC, create an endpoint service that points to the NLB.
4147
- Enable acceptance of connection requests from specific accounts.
4248

43-
6. Authorize ClickPipes to Use the Endpoint Service
49+
5. Authorize ClickPipes to Use the Endpoint Service
4450
- Grant permission to the ClickPipes account to request this endpoint service.
4551
- Configure allowed principals by adding the following principal ID:
4652
```
4753
arn:aws:iam::072088201116:root
4854
```
4955

56+
6. Disable "Enforce Security Group Inbound Rules on Private Link Traffic" on the NLB (if a security group is attached to the NLB)
57+
- Navigate to the NLB's settings and disable the "Enforce Security Group Inbound Rules on Private Link Traffic" setting if a security group is attached to the NLB.
58+
- If using Terraform, set the `enforce_security_group_inbound_rules_on_private_link_traffic` attribute to `false` for the NLB
59+
- This setting is **required** to allow traffic from the ClickPipes VPC to the NLB.
60+
5061
## Initiating connection
5162

52-
When it's done, share details such as private DNS name, VPC service name and availability zone.
63+
When it's done, share details such as AWS region, VPC service name and availability zone.
5364
ClickPipes team will initiate VPC endpoints creation in ClickPipes VPC. This will require [connection request acceptance](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests) on your side.
5465

5566
## Creating ClickPipes
5667

57-
Use your RDS's private DNS endpoints to create your ClickPipes.
68+
ClickPipes team will provide the DNS name to be used in the ClickPipe creation process. You can now [create your ClickPipe](/docs/en/integrations/clickpipes/postgres).
69+
70+
## Dynamically updating the RDS endpoint IP
71+
72+
When the RDS endpoint IP changes (in case of restarts/failovers/updates), you need to update the NLB target group with
73+
the new IP. You can automate this process using AWS Lambda functions or other automation tools.

0 commit comments

Comments
 (0)