diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index b67fbfc15de..bd852362189 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -44,7 +44,7 @@ ClickPipes reverse private endpoint can be configured with one of the following ### VPC resource {#vpc-resource} -Your VPC resources can be accessed in ClickPipes using PrivateLink and [AWS VPC Lattice](https://docs.aws.amazon.com/vpc-lattice/latest/ug/what-is-vpc-lattice.html). This approach doesn't require setting up a load balancer in front of your data source. +Your VPC resources can be accessed in ClickPipes using [PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). This approach doesn't require setting up a load balancer in front of your data source. Resource configuration can be targeted with a specific host or RDS cluster ARN. Cross-region is not supported. @@ -56,11 +56,21 @@ To set up PrivateLink with VPC resource: 2. Create a resource configuration 3. Create a resource share -#### 1. Create a Resource-Gateway {#create-resource-gateway} + -Resource-Gateway is the point that receives traffic for specified resources in your VPC. +#### Create a resource gateway {#create-resource-gateway} -You can create a Resource-Gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command: +Resource gateway is the point that receives traffic for specified resources in your VPC. + +:::note +Your resource gateway attached subnets are recommended to have sufficient IP addresses available. +It's recommended to have at least `/26` subnet mask for each subnet. + +For each VPC endpoint (each Reverse Private Endpoint), AWS requires a consecutive block of 16 IP addresses per subnet. (`/28` subnet mask) +If this requirement is not met, Reverse Private Endpoint will transition to a failed state. +::: + +You can create a resource gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command: ```bash aws vpc-lattice create-resource-gateway \ @@ -70,18 +80,18 @@ aws vpc-lattice create-resource-gateway \ --name ``` -The output will contain a Resource-Gateway id, which you will need for the next step. +The output will contain a resource gateway id, which you will need for the next step. -Before you can proceed, you'll need to wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command: +Before you can proceed, you'll need to wait for the resource gateway to enter into an `Active` state. You can check the state by running the following command: ```bash aws vpc-lattice get-resource-gateway \ --resource-gateway-identifier ``` -#### 2. Create a VPC Resource-Configuration {#create-resource-configuration} +#### Create a VPC Resource-Configuration {#create-resource-configuration} -Resource-Configuration is associated with Resource-Gateway to make your resource accessible. +Resource-Configuration is associated with resource gateway to make your resource accessible. You can create a Resource-Configuration from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-configuration.html) or with the following command: @@ -105,13 +115,21 @@ aws vpc-lattice create-resource-configuration \ --resource-configuration-definition 'arnResource={arn=arn:aws:rds:us-east-1:123456789012:cluster:my-rds-cluster}' ``` +:::note +You can't create a resource configuration for a publicly accessible cluster. +If your cluster is publicly accessible, you must modify the cluster +to make it private before creating the resource configuration +or use [IP allow list](/integrations/clickpipes#list-of-static-ips) instead. +For more information, see the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/resource-configuration.html#resource-definition). +::: + The output will contain a Resource-Configuration ARN, which you will need for the next step. It will also contain a Resource-Configuration ID, which you will need to set up a ClickPipe connection with VPC resource. -#### 3. Create a Resource-Share {#create-resource-share} +#### Create a Resource-Share {#create-resource-share} Sharing your resource requires a Resource-Share. This is facilitated through the Resource Access Manager (RAM). -You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116`: +You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116` (arn:aws:iam::072088201116:root): ```bash aws ram create-resource-share \ @@ -129,6 +147,8 @@ You are ready to [create a ClickPipe with Reverse private endpoint](#creating-cl For more details on PrivateLink with VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). + + ### MSK multi-VPC connectivity {#msk-multi-vpc} The [Multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster. @@ -174,6 +194,8 @@ can be configured for ClickPipes. Add [your ClickPipe region](#aws-privatelink-r ## Creating a ClickPipe with reverse private endpoint {#creating-clickpipe} + + 1. Access the SQL Console for your ClickHouse Cloud Service. ClickPipes service @@ -193,7 +215,7 @@ can be configured for ClickPipes. Add [your ClickPipe region](#aws-privatelink-r 5. Select any of existing reverse private endpoints or create a new one. :::info -If cross-region access is required for RDS, you need to create a VPC endpoint service and +If cross-region access is required for RDS, you need to create a VPC endpoint service and [this guide should provide](/knowledgebase/aws-privatelink-setup-for-clickpipes) a good starting point to set it up. For same-region access, creating a VPC Resource is the recommended approach. @@ -228,21 +250,27 @@ For same-region access, creating a VPC Resource is the recommended approach. To see a full list of DNS names, access it in the cloud service settings. + + ## Managing existing reverse private endpoints {#managing-existing-endpoints} You can manage existing reverse private endpoints in the ClickHouse Cloud service settings: + + 1. On a sidebar find the `Settings` button and click on it. -ClickHouse Cloud settings + ClickHouse Cloud settings 2. Click on `Reverse private endpoints` in a `ClickPipe reverse private endpoints` section. -ClickHouse Cloud settings + ClickHouse Cloud settings + + Reverse private endpoint extended information is shown in the flyout. - Reverse private endpoint extended information is shown in the flyout. + Endpoint can be removed from here. It will affect any ClickPipes using this endpoint. - Endpoint can be removed from here. It will affect any ClickPipes using this endpoint. + ## Supported AWS regions {#aws-privatelink-regions} diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md index fb31cda80f2..e3d110efc21 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md @@ -39,4 +39,12 @@ You have several options to resolve these issues: ### Do you support schema changes? {#do-you-support-schema-changes} -Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](./schema-changes) page for more information. \ No newline at end of file +Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](./schema-changes) page for more information. + +### Do you support replicating MySQL foreign key cascading deletes `ON DELETE CASCADE`? {#support-on-delete-cascade} + +Due to how MySQL [handles cascading deletes](https://dev.mysql.com/doc/refman/8.0/en/innodb-and-mysql-replication.html), they are not written to the binlog. Therefore it's not possible for ClickPipes (or any CDC tool) to replicate them. This can lead to inconsistent data. It's advised to use triggers instead for supporting cascading deletes. + +### Why can I not replicate my table which has a dot in it? {#replicate-table-dot} +PeerDB has a limitation currently where dots in source table identifiers - aka either schema name or table name - is not supported for replication as PeerDB cannot discern, in that case, what is the schema and what is the table as it splits on dot. +Effort is being made to support input of schema and table separately to get around this limitation. \ No newline at end of file diff --git a/docs/use-cases/observability/clickstack/config.md b/docs/use-cases/observability/clickstack/config.md index 9a46d0baae8..6fa08870437 100644 --- a/docs/use-cases/observability/clickstack/config.md +++ b/docs/use-cases/observability/clickstack/config.md @@ -213,6 +213,10 @@ For example, below is the Logs source configured with correlated sources: ### Application configuration settings {#application-configuration-settings} +:::note HyperDX in ClickHouse Cloud +These settings cannot be modified when HyperDX is managed in ClickHouse Cloud. +::: + - `HYPERDX_API_KEY` - **Default:** None (required) - **Description:** Authentication key for the HyperDX API.