From 66b60621883438a135cc61da2918effd2308ddab Mon Sep 17 00:00:00 2001 From: Shaun Struwig <41984034+Blargian@users.noreply.github.com> Date: Thu, 13 Nov 2025 08:54:25 +0100 Subject: [PATCH 1/4] revert accidental changes --- .../clickpipes/aws-privatelink.md | 61 +++++++++++++------ .../data-ingestion/clickpipes/mysql/faq.md | 11 +++- 2 files changed, 54 insertions(+), 18 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index b67fbfc15de..495834ca6d4 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -6,7 +6,6 @@ title: 'AWS PrivateLink for ClickPipes' doc_type: 'guide' keywords: ['aws privatelink', 'ClickPipes security', 'vpc endpoint', 'private connectivity', 'vpc resource'] --- - import cp_service from '@site/static/images/integrations/data-ingestion/clickpipes/cp_service.png'; import cp_step0 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step0.png'; import cp_rpe_select from '@site/static/images/integrations/data-ingestion/clickpipes/cp_rpe_select.png'; @@ -44,7 +43,7 @@ ClickPipes reverse private endpoint can be configured with one of the following ### VPC resource {#vpc-resource} -Your VPC resources can be accessed in ClickPipes using PrivateLink and [AWS VPC Lattice](https://docs.aws.amazon.com/vpc-lattice/latest/ug/what-is-vpc-lattice.html). This approach doesn't require setting up a load balancer in front of your data source. +Your VPC resources can be accessed in ClickPipes using [PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). This approach doesn't require setting up a load balancer in front of your data source. Resource configuration can be targeted with a specific host or RDS cluster ARN. Cross-region is not supported. @@ -56,11 +55,21 @@ To set up PrivateLink with VPC resource: 2. Create a resource configuration 3. Create a resource share -#### 1. Create a Resource-Gateway {#create-resource-gateway} + + +#### Create a resource gateway {#create-resource-gateway} + +Resource gateway is the point that receives traffic for specified resources in your VPC. -Resource-Gateway is the point that receives traffic for specified resources in your VPC. +:::note +Your resource gateway attached subnets are recommended to have sufficient IP addresses available. +It's recommended to have at least `/26` subnet mask for each subnet. -You can create a Resource-Gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command: +For each VPC endpoint (each Reverse Private Endpoint), AWS requires a consecutive block of 16 IP addresses per subnet. (`/28` subnet mask) +If this requirement is not met, Reverse Private Endpoint will transition to a failed state. +::: + +You can create a resource gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command: ```bash aws vpc-lattice create-resource-gateway \ @@ -70,18 +79,18 @@ aws vpc-lattice create-resource-gateway \ --name ``` -The output will contain a Resource-Gateway id, which you will need for the next step. +The output will contain a resource gateway id, which you will need for the next step. -Before you can proceed, you'll need to wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command: +Before you can proceed, you'll need to wait for the resource gateway to enter into an `Active` state. You can check the state by running the following command: ```bash aws vpc-lattice get-resource-gateway \ --resource-gateway-identifier ``` -#### 2. Create a VPC Resource-Configuration {#create-resource-configuration} +#### Create a VPC Resource-Configuration {#create-resource-configuration} -Resource-Configuration is associated with Resource-Gateway to make your resource accessible. +Resource-Configuration is associated with resource gateway to make your resource accessible. You can create a Resource-Configuration from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-configuration.html) or with the following command: @@ -105,13 +114,21 @@ aws vpc-lattice create-resource-configuration \ --resource-configuration-definition 'arnResource={arn=arn:aws:rds:us-east-1:123456789012:cluster:my-rds-cluster}' ``` +:::note +You can't create a resource configuration for a publicly accessible cluster. +If your cluster is publicly accessible, you must modify the cluster +to make it private before creating the resource configuration +or use [IP allow list](/integrations/clickpipes#list-of-static-ips) instead. +For more information, see the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/resource-configuration.html#resource-definition). +::: + The output will contain a Resource-Configuration ARN, which you will need for the next step. It will also contain a Resource-Configuration ID, which you will need to set up a ClickPipe connection with VPC resource. -#### 3. Create a Resource-Share {#create-resource-share} +#### Create a Resource-Share {#create-resource-share} Sharing your resource requires a Resource-Share. This is facilitated through the Resource Access Manager (RAM). -You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116`: +You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116` (arn:aws:iam::072088201116:root): ```bash aws ram create-resource-share \ @@ -129,6 +146,8 @@ You are ready to [create a ClickPipe with Reverse private endpoint](#creating-cl For more details on PrivateLink with VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). + + ### MSK multi-VPC connectivity {#msk-multi-vpc} The [Multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster. @@ -174,6 +193,8 @@ can be configured for ClickPipes. Add [your ClickPipe region](#aws-privatelink-r ## Creating a ClickPipe with reverse private endpoint {#creating-clickpipe} + + 1. Access the SQL Console for your ClickHouse Cloud Service. ClickPipes service @@ -193,7 +214,7 @@ can be configured for ClickPipes. Add [your ClickPipe region](#aws-privatelink-r 5. Select any of existing reverse private endpoints or create a new one. :::info -If cross-region access is required for RDS, you need to create a VPC endpoint service and +If cross-region access is required for RDS, you need to create a VPC endpoint service and [this guide should provide](/knowledgebase/aws-privatelink-setup-for-clickpipes) a good starting point to set it up. For same-region access, creating a VPC Resource is the recommended approach. @@ -228,21 +249,27 @@ For same-region access, creating a VPC Resource is the recommended approach. To see a full list of DNS names, access it in the cloud service settings. + + ## Managing existing reverse private endpoints {#managing-existing-endpoints} You can manage existing reverse private endpoints in the ClickHouse Cloud service settings: + + 1. On a sidebar find the `Settings` button and click on it. -ClickHouse Cloud settings + ClickHouse Cloud settings 2. Click on `Reverse private endpoints` in a `ClickPipe reverse private endpoints` section. -ClickHouse Cloud settings + ClickHouse Cloud settings + + Reverse private endpoint extended information is shown in the flyout. - Reverse private endpoint extended information is shown in the flyout. + Endpoint can be removed from here. It will affect any ClickPipes using this endpoint. - Endpoint can be removed from here. It will affect any ClickPipes using this endpoint. + ## Supported AWS regions {#aws-privatelink-regions} @@ -260,4 +287,4 @@ Currently, only VPC endpoint service supports cross-region connectivity. Private endpoints are linked to a specific ClickHouse service and are not transferable between services. -Multiple ClickPipes for a single ClickHouse service can reuse the same endpoint. +Multiple ClickPipes for a single ClickHouse service can reuse the same endpoint. \ No newline at end of file diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md index fb31cda80f2..f191542e05a 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md @@ -37,6 +37,15 @@ You have several options to resolve these issues: 4. **Skip certificate verification** - For self-hosted MySQL or MariaDB, whose default configurations provision a self-signed certificate we can't validate ([MySQL](https://dev.mysql.com/doc/refman/8.4/en/creating-ssl-rsa-files-using-mysql.html#creating-ssl-rsa-files-using-mysql-automatic), [MariaDB](https://mariadb.com/kb/en/securing-connections-for-client-and-server/#enabling-tls-for-mariadb-server)). Relying on this certificate encrypts the data in transit but runs the risk of server impersonation. We recommend properly signed certificates for production environments, but this option is useful for testing on a one-off instance or connecting to legacy infrastructure. + ### Do you support schema changes? {#do-you-support-schema-changes} -Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](./schema-changes) page for more information. \ No newline at end of file +Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](./schema-changes) page for more information. + +### Do you support replicating MySQL foreign key cascading deletes `ON DELETE CASCADE`? {#support-on-delete-cascade} + +Due to how MySQL [handles cascading deletes](https://dev.mysql.com/doc/refman/8.0/en/innodb-and-mysql-replication.html), they are not written to the binlog. Therefore it's not possible for ClickPipes (or any CDC tool) to replicate them. This can lead to inconsistent data. It's advised to use triggers instead for supporting cascading deletes. + +### Why can I not replicate my table which has a dot in it? {#replicate-table-dot} +PeerDB has a limitation currently where dots in source table identifiers - aka either schema name or table name - is not supported for replication as PeerDB cannot discern, in that case, what is the schema and what is the table as it splits on dot. +Effort is being made to support input of schema and table separately to get around this limitation. \ No newline at end of file From 317543ed6d0bdc30c14b2154dae58a4b07ec9de8 Mon Sep 17 00:00:00 2001 From: Shaun Struwig <41984034+Blargian@users.noreply.github.com> Date: Thu, 13 Nov 2025 09:01:38 +0100 Subject: [PATCH 2/4] remove blank line --- docs/integrations/data-ingestion/clickpipes/mysql/faq.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md index f191542e05a..e3d110efc21 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md @@ -37,7 +37,6 @@ You have several options to resolve these issues: 4. **Skip certificate verification** - For self-hosted MySQL or MariaDB, whose default configurations provision a self-signed certificate we can't validate ([MySQL](https://dev.mysql.com/doc/refman/8.4/en/creating-ssl-rsa-files-using-mysql.html#creating-ssl-rsa-files-using-mysql-automatic), [MariaDB](https://mariadb.com/kb/en/securing-connections-for-client-and-server/#enabling-tls-for-mariadb-server)). Relying on this certificate encrypts the data in transit but runs the risk of server impersonation. We recommend properly signed certificates for production environments, but this option is useful for testing on a one-off instance or connecting to legacy infrastructure. - ### Do you support schema changes? {#do-you-support-schema-changes} Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](./schema-changes) page for more information. From 6c4845a8343f496a67edcc6bead49eefee00491d Mon Sep 17 00:00:00 2001 From: Shaun Struwig <41984034+Blargian@users.noreply.github.com> Date: Thu, 13 Nov 2025 09:03:29 +0100 Subject: [PATCH 3/4] revert accidental removal of note --- docs/use-cases/observability/clickstack/config.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/use-cases/observability/clickstack/config.md b/docs/use-cases/observability/clickstack/config.md index 9a46d0baae8..6fa08870437 100644 --- a/docs/use-cases/observability/clickstack/config.md +++ b/docs/use-cases/observability/clickstack/config.md @@ -213,6 +213,10 @@ For example, below is the Logs source configured with correlated sources: ### Application configuration settings {#application-configuration-settings} +:::note HyperDX in ClickHouse Cloud +These settings cannot be modified when HyperDX is managed in ClickHouse Cloud. +::: + - `HYPERDX_API_KEY` - **Default:** None (required) - **Description:** Authentication key for the HyperDX API. From 9bdd7e0575d1412fe72ac768b7fe86475277b595 Mon Sep 17 00:00:00 2001 From: Shaun Struwig <41984034+Blargian@users.noreply.github.com> Date: Thu, 13 Nov 2025 09:16:06 +0100 Subject: [PATCH 4/4] fix docs check --- docs/integrations/data-ingestion/clickpipes/aws-privatelink.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 495834ca6d4..bd852362189 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -6,6 +6,7 @@ title: 'AWS PrivateLink for ClickPipes' doc_type: 'guide' keywords: ['aws privatelink', 'ClickPipes security', 'vpc endpoint', 'private connectivity', 'vpc resource'] --- + import cp_service from '@site/static/images/integrations/data-ingestion/clickpipes/cp_service.png'; import cp_step0 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step0.png'; import cp_rpe_select from '@site/static/images/integrations/data-ingestion/clickpipes/cp_rpe_select.png'; @@ -287,4 +288,4 @@ Currently, only VPC endpoint service supports cross-region connectivity. Private endpoints are linked to a specific ClickHouse service and are not transferable between services. -Multiple ClickPipes for a single ClickHouse service can reuse the same endpoint. \ No newline at end of file +Multiple ClickPipes for a single ClickHouse service can reuse the same endpoint.