Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 43 additions & 15 deletions docs/integrations/data-ingestion/clickpipes/aws-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@

### VPC resource {#vpc-resource}

Your VPC resources can be accessed in ClickPipes using PrivateLink and [AWS VPC Lattice](https://docs.aws.amazon.com/vpc-lattice/latest/ug/what-is-vpc-lattice.html). This approach doesn't require setting up a load balancer in front of your data source.
Your VPC resources can be accessed in ClickPipes using [PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). This approach doesn't require setting up a load balancer in front of your data source.

Resource configuration can be targeted with a specific host or RDS cluster ARN.
Cross-region is not supported.
Expand All @@ -56,11 +56,21 @@
2. Create a resource configuration
3. Create a resource share

#### 1. Create a Resource-Gateway {#create-resource-gateway}
<VerticalStepper headerLevel="h4">

Resource-Gateway is the point that receives traffic for specified resources in your VPC.
#### Create a resource gateway {#create-resource-gateway}

You can create a Resource-Gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command:
Resource gateway is the point that receives traffic for specified resources in your VPC.

:::note
Your resource gateway attached subnets are recommended to have sufficient IP addresses available.
It's recommended to have at least `/26` subnet mask for each subnet.

For each VPC endpoint (each Reverse Private Endpoint), AWS requires a consecutive block of 16 IP addresses per subnet. (`/28` subnet mask)
If this requirement is not met, Reverse Private Endpoint will transition to a failed state.
:::

You can create a resource gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command:

```bash
aws vpc-lattice create-resource-gateway \
Expand All @@ -70,18 +80,18 @@
--name <RESOURCE_GATEWAY_NAME>
```

The output will contain a Resource-Gateway id, which you will need for the next step.
The output will contain a resource gateway id, which you will need for the next step.

Check warning on line 83 in docs/integrations/data-ingestion/clickpipes/aws-privatelink.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'will need', use present tense.

Check warning on line 83 in docs/integrations/data-ingestion/clickpipes/aws-privatelink.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'will contain', use present tense.

Before you can proceed, you'll need to wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command:
Before you can proceed, you'll need to wait for the resource gateway to enter into an `Active` state. You can check the state by running the following command:

Check warning on line 85 in docs/integrations/data-ingestion/clickpipes/aws-privatelink.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'you'll need', use present tense.

```bash
aws vpc-lattice get-resource-gateway \
--resource-gateway-identifier <RESOURCE_GATEWAY_ID>
```

#### 2. Create a VPC Resource-Configuration {#create-resource-configuration}
#### Create a VPC Resource-Configuration {#create-resource-configuration}

Resource-Configuration is associated with Resource-Gateway to make your resource accessible.
Resource-Configuration is associated with resource gateway to make your resource accessible.

You can create a Resource-Configuration from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-configuration.html) or with the following command:

Expand All @@ -105,13 +115,21 @@
--resource-configuration-definition 'arnResource={arn=arn:aws:rds:us-east-1:123456789012:cluster:my-rds-cluster}'
```

:::note
You can't create a resource configuration for a publicly accessible cluster.
If your cluster is publicly accessible, you must modify the cluster
to make it private before creating the resource configuration
or use [IP allow list](/integrations/clickpipes#list-of-static-ips) instead.
For more information, see the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/resource-configuration.html#resource-definition).
:::

The output will contain a Resource-Configuration ARN, which you will need for the next step. It will also contain a Resource-Configuration ID, which you will need to set up a ClickPipe connection with VPC resource.

#### 3. Create a Resource-Share {#create-resource-share}
#### Create a Resource-Share {#create-resource-share}

Sharing your resource requires a Resource-Share. This is facilitated through the Resource Access Manager (RAM).

You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116`:
You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116` (arn:aws:iam::072088201116:root):

```bash
aws ram create-resource-share \
Expand All @@ -129,6 +147,8 @@

For more details on PrivateLink with VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html).

</VerticalStepper>

### MSK multi-VPC connectivity {#msk-multi-vpc}

The [Multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster.
Expand Down Expand Up @@ -174,6 +194,8 @@

## Creating a ClickPipe with reverse private endpoint {#creating-clickpipe}

<VerticalStepper headerLevel="list">

1. Access the SQL Console for your ClickHouse Cloud Service.

<Image img={cp_service} alt="ClickPipes service" size="md" border/>
Expand All @@ -193,7 +215,7 @@
5. Select any of existing reverse private endpoints or create a new one.

:::info
If cross-region access is required for RDS, you need to create a VPC endpoint service and
If cross-region access is required for RDS, you need to create a VPC endpoint service and
[this guide should provide](/knowledgebase/aws-privatelink-setup-for-clickpipes) a good starting point to set it up.

For same-region access, creating a VPC Resource is the recommended approach.
Expand Down Expand Up @@ -228,21 +250,27 @@

To see a full list of DNS names, access it in the cloud service settings.

</VerticalStepper>

## Managing existing reverse private endpoints {#managing-existing-endpoints}

You can manage existing reverse private endpoints in the ClickHouse Cloud service settings:

<VerticalStepper headerLevel="list">

1. On a sidebar find the `Settings` button and click on it.

<Image img={cp_rpe_settings0} alt="ClickHouse Cloud settings" size="lg" border/>
<Image img={cp_rpe_settings0} alt="ClickHouse Cloud settings" size="lg" border/>

2. Click on `Reverse private endpoints` in a `ClickPipe reverse private endpoints` section.

<Image img={cp_rpe_settings1} alt="ClickHouse Cloud settings" size="md" border/>
<Image img={cp_rpe_settings1} alt="ClickHouse Cloud settings" size="md" border/>

Reverse private endpoint extended information is shown in the flyout.

Reverse private endpoint extended information is shown in the flyout.
Endpoint can be removed from here. It will affect any ClickPipes using this endpoint.

Check warning on line 271 in docs/integrations/data-ingestion/clickpipes/aws-privatelink.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'will affect', use present tense.

Endpoint can be removed from here. It will affect any ClickPipes using this endpoint.
</VerticalStepper>

## Supported AWS regions {#aws-privatelink-regions}

Expand Down
10 changes: 9 additions & 1 deletion docs/integrations/data-ingestion/clickpipes/mysql/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,4 +39,12 @@

### Do you support schema changes? {#do-you-support-schema-changes}

Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](./schema-changes) page for more information.
Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](./schema-changes) page for more information.

Check warning on line 42 in docs/integrations/data-ingestion/clickpipes/mysql/faq.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Colons

': S' should be in lowercase.

### Do you support replicating MySQL foreign key cascading deletes `ON DELETE CASCADE`? {#support-on-delete-cascade}

Due to how MySQL [handles cascading deletes](https://dev.mysql.com/doc/refman/8.0/en/innodb-and-mysql-replication.html), they are not written to the binlog. Therefore it's not possible for ClickPipes (or any CDC tool) to replicate them. This can lead to inconsistent data. It's advised to use triggers instead for supporting cascading deletes.

### Why can I not replicate my table which has a dot in it? {#replicate-table-dot}
PeerDB has a limitation currently where dots in source table identifiers - aka either schema name or table name - is not supported for replication as PeerDB cannot discern, in that case, what is the schema and what is the table as it splits on dot.
Effort is being made to support input of schema and table separately to get around this limitation.
4 changes: 4 additions & 0 deletions docs/use-cases/observability/clickstack/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,10 @@ For example, below is the Logs source configured with correlated sources:

### Application configuration settings {#application-configuration-settings}

:::note HyperDX in ClickHouse Cloud
These settings cannot be modified when HyperDX is managed in ClickHouse Cloud.
:::

- `HYPERDX_API_KEY`
- **Default:** None (required)
- **Description:** Authentication key for the HyperDX API.
Expand Down