You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/kafka/index.md
+14-7Lines changed: 14 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,10 @@
1
+
---
2
+
sidebar_label: 'Integrating Kafka with ClickHouse'
3
+
sidebar_position: 1
4
+
slug: /integrations/kafka
5
+
description: 'Introduction to Kafka with ClickHouse'
6
+
title: 'Integrating Kafka with ClickHouse'
7
+
---
1
8
2
9
# Integrating Kafka with ClickHouse
3
10
@@ -17,22 +24,22 @@ For a more detailed comparison between these options, see [Choosing an option](#
17
24
18
25
### ClickPipes for Kafka {#clickpipes-for-kafka}
19
26
20
-
[ClickPipes](../clickpipes/index.md) is a managed integration platform that makes ingesting data from a diverse set of sources as simple as clicking a few buttons. Because it is fully managed and purpose-built for production workloads, ClickPipes significantly lowers infrastructure (CAPEX) and operational (OPEC'S) costs, removing the need for external data streaming and ETL tools.
27
+
[ClickPipes](../clickpipes/index.md) is a managed integration platform that makes ingesting data from a diverse set of sources as simple as clicking a few buttons. Because it is fully managed and purpose-built for production workloads, ClickPipes significantly lowers infrastructure and operational costs, removing the need for external data streaming and ETL tools.
21
28
22
29
:::tip
23
30
This is the recommended option if you're a ClickHouse Cloud user. ClickPipes is **fully managed** and purpose-built to deliver the **best performance** in Cloud environments.
24
31
:::
25
32
26
33
#### Main features {#clickpipes-for-kafka-main-features}
27
34
28
-
[//]: #"TODO(morsapaes) It isn't optimal to link to a static alpha-release of the Terraform provider. Link to a Terraform guide once that's available."
35
+
[//]: #"TODO It isn't optimal to link to a static alpha-release of the Terraform provider. Link to a Terraform guide once that's available."
29
36
30
37
* Optimized for ClickHouse Cloud, delivering blazing-fast performance
31
38
* Horizontal and vertical scalability for high-throughput workloads
32
39
* Built-in fault tolerance with configurable replicas and automatic retries
33
40
* Deployment and management via ClickHouse Cloud UI, [Open API](../../../cloud/manage/api/api-overview.md), or [Terraform](https://registry.terraform.io/providers/ClickHouse/clickhouse/3.3.3-alpha2/docs/resources/clickpipe)
34
41
* Enterprise-grade security with support for cloud-native authorization (IAM) and private connectivity (PrivateLink)
35
-
*Support for a wide range of [data sources](../clickpipes/kafka.md#supported-data-sources), including Confluent Cloud, Amazon MSK, Redpanda Cloud, and Azure Event Hubs
42
+
*Supports a wide range of [data sources](../clickpipes/kafka.md#supported-data-sources), including Confluent Cloud, Amazon MSK, Redpanda Cloud, and Azure Event Hubs
36
43
37
44
#### Getting started {#clickpipes-for-kafka-getting-started}
38
45
@@ -57,7 +64,7 @@ This is the recommended option if you're already a Kafka Connect user. The Kafka
57
64
58
65
To get started using the ClickHouse Kafka Connect Sink, see the [reference documentation](./kafka-clickhouse-connect-sink.md).
59
66
60
-
### Kafka Table Engine {#kafka-table-engine}
67
+
### Kafka table engine {#kafka-table-engine}
61
68
62
69
The [Kafka table engine](./kafka-table-engine.md) can be used to read data from and write data to Apache Kafka and other Kafka API-compatible brokers. This option is bundled with open-source ClickHouse and is available across all deployment types.
63
70
@@ -71,17 +78,17 @@ This is the recommended option if you're self-hosting ClickHouse and need a **lo
71
78
* Bundled with open-source ClickHouse
72
79
* Supports all ClickHouse data types
73
80
74
-
#### Getting started {kafka-table-engine-getting-started}
81
+
#### Getting started {#kafka-table-engine-getting-started}
75
82
76
-
To get started using the Kafka Table Engine, see the [reference documentation](./kafka-table-engine.md).
83
+
To get started using the Kafka table engine, see the [reference documentation](./kafka-table-engine.md).
77
84
78
85
### Choosing an option {#choosing-an-option}
79
86
80
87
| Product | Deployment | Strengths | Weaknesses |
81
88
|---------|------------|-----------|------------|
82
89
|**ClickPipes for Kafka**|[CH Cloud]| • Scalable architecture for high throughput and low latency<br/>• Built-in monitoring and schema management<br/>• Private networking connections (via PrivateLink)<br/>• Supports SSL/TLS authentication and IAM authorization<br/>• Supports programmatic configuration (Terraform, API endpoints) | • Does not support pushing data to Kafka<br/>• At-least-once semantics |
83
90
|**Kafka Connect Sink**|[CH Cloud]<br/>[CH BYOC]<br/>[CH OSS]| • Exactly-once semantics<br/>• Allows granular control over data transformation, batching and error handling<br/>• Can be deployed in private networks<br/>• Allows real-time replication from databases not yet supported in ClickPipes via Debezium | • Does not support pushing data to Kafka<br/>• Operationally complex to set up and maintain<br/>• Requires Kafka and Kafka Connect expertise |
84
-
|**Kafka Table Engine**|[CH Cloud]<br/>[CH BYOC]<br/>[CH OSS]| • Supports pushing data to Kafka<br/>• Allows real-time replication from databases not yet supported in ClickPipes via Debezium | • At-least-once semantics<br/>• Limited horizontal scaling for consumers. Cannot be scaled independently from the CH server<br/>• Limited error handling and debugging options<br/>• Requires Kafka expertise |
91
+
|**Kafka table engine**|[CH Cloud]<br/>[CH BYOC]<br/>[CH OSS]| • Supports pushing data to Kafka<br/>• Allows real-time replication from databases not yet supported in ClickPipes via Debezium | • At-least-once semantics<br/>• Limited horizontal scaling for consumers. Cannot be scaled independently from the CH server<br/>• Limited error handling and debugging options<br/>• Requires Kafka expertise |
0 commit comments