You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/kafka/index.md
+20-20Lines changed: 20 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,11 +14,11 @@ title: 'Integrating Kafka with ClickHouse'
14
14
15
15
Choosing the right option for your use case depends on multiple factors, including your ClickHouse deployment type, data flow direction and operational requirements.
16
16
17
-
|Option | Deployment type | Kafka to ClickHouse | ClickHouse to Kafka| Fully managed |
17
+
|Option | Deployment type |Fully managed |Kafka to ClickHouse | ClickHouse to Kafka |
For a more detailed comparison between these options, see [Choosing an option](#choosing-an-option).
24
24
@@ -40,6 +40,7 @@ This is the recommended option if you're a ClickHouse Cloud user. ClickPipes is
40
40
* Deployment and management via ClickHouse Cloud UI, [Open API](../../../cloud/manage/api/api-overview.md), or [Terraform](https://registry.terraform.io/providers/ClickHouse/clickhouse/3.3.3-alpha2/docs/resources/clickpipe)
41
41
* Enterprise-grade security with support for cloud-native authorization (IAM) and private connectivity (PrivateLink)
42
42
* Supports a wide range of [data sources](../clickpipes/kafka.md#supported-data-sources), including Confluent Cloud, Amazon MSK, Redpanda Cloud, and Azure Event Hubs
43
+
* Supports most common serialization formats (JSON, Avro, Protobuf coming soon!)
43
44
44
45
#### Getting started {#clickpipes-for-kafka-getting-started}
45
46
@@ -50,14 +51,13 @@ To get started using ClickPipes for Kafka, see the [reference documentation](../
50
51
Kafka Connect is an open-source framework that works as a centralized data hub for simple data integration between Kafka and other data systems. The [ClickHouse Kafka Connect Sink](https://github.com/ClickHouse/clickhouse-kafka-connect) connector provides a scalable and highly-configurable option to read data from Apache Kafka and other Kafka API-compatible brokers.
51
52
52
53
:::tip
53
-
This is the recommended option if you're already a Kafka Connect user. The Kafka Connect Sink offers a rich set of features and configuration options for **advanced tuning**.
54
+
This is the recommended option if you prefer **high configurability** or are already a Kafka Connect user.
54
55
:::
55
56
56
57
#### Main features {#kafka-connect-sink-main-features}
57
58
58
59
* Can be configured to support exactly-once semantics
59
-
* Supports all ClickHouse data types
60
-
* Handles structured data with declared schemas and unstructured JSON data
60
+
* Supports most common serialization formats (JSON, Avro, Protobuf)
61
61
* Tested continuously against ClickHouse Cloud
62
62
63
63
#### Getting started {#kafka-connect-sink-getting-started}
@@ -74,32 +74,32 @@ This is the recommended option if you're self-hosting ClickHouse and need a **lo
74
74
75
75
#### Main features {#kafka-table-engine-main-features}
76
76
77
-
* Can be used for reading and writing data
77
+
* Can be used for [reading](./kafka-table-engine.md/#kafka-to-clickhouse) and [writing](./kafka-table-engine.md/#clickhouse-to-kafka) data
78
78
* Bundled with open-source ClickHouse
79
-
* Supports all ClickHouse data types
79
+
* Supports most common serialization formats (JSON, Avro, Protobuf)
80
80
81
81
#### Getting started {#kafka-table-engine-getting-started}
82
82
83
83
To get started using the Kafka table engine, see the [reference documentation](./kafka-table-engine.md).
84
84
85
85
### Choosing an option {#choosing-an-option}
86
86
87
-
| Product |Deployment |Strengths | Weaknesses |
88
-
|---------|------------|-----------|------------|
89
-
|**ClickPipes for Kafka**|[CH Cloud]|• Scalable architecture for high throughput and low latency<br/>• Built-in monitoring and schema management<br/>• Private networking connections (via PrivateLink)<br/>• Supports SSL/TLS authentication and IAM authorization<br/>• Supports programmatic configuration (Terraform, API endpoints) | • Does not support pushing data to Kafka<br/>• At-least-once semantics |
90
-
|**Kafka Connect Sink**|[CH Cloud]<br/>[CH BYOC]<br/>[CH OSS]|• Exactly-once semantics<br/>• Allows granular control over data transformation, batching and error handling<br/>• Can be deployed in private networks<br/>• Allows real-time replication from databases not yet supported in ClickPipes via Debezium | • Does not support pushing data to Kafka<br/>• Operationally complex to set up and maintain<br/>• Requires Kafka and Kafka Connect expertise |
91
-
|**Kafka table engine**|[CH Cloud]<br/>[CH BYOC]<br/>[CH OSS]|• Supports pushing data to Kafka<br/>• Allows real-time replication from databases not yet supported in ClickPipes via Debezium | • At-least-once semantics<br/>• Limited horizontal scaling for consumers. Cannot be scaled independently from the CH server<br/>• Limited error handling and debugging options<br/>• Requires Kafka expertise |
87
+
| Product | Strengths | Weaknesses |
88
+
|---------|-----------|------------|
89
+
|**ClickPipes for Kafka**| • Scalable architecture for high throughput and low latency<br/>• Built-in monitoring and schema management<br/>• Private networking connections (via PrivateLink)<br/>• Supports SSL/TLS authentication and IAM authorization<br/>• Supports programmatic configuration (Terraform, API endpoints) | • Does not support pushing data to Kafka<br/>• At-least-once semantics |
90
+
|**Kafka Connect Sink**| • Exactly-once semantics<br/>• Allows granular control over data transformation, batching and error handling<br/>• Can be deployed in private networks<br/>• Allows real-time replication from databases not yet supported in ClickPipes via Debezium | • Does not support pushing data to Kafka<br/>• Operationally complex to set up and maintain<br/>• Requires Kafka and Kafka Connect expertise |
91
+
|**Kafka table engine**| • Supports [pushing data to Kafka](./kafka-table-engine.md/#clickhouse-to-kafka)<br/>• Operationally simple to set up | • At-least-once semantics<br/>• Limited horizontal scaling for consumers. Cannot be scaled independently from the ClickHouse server<br/>• Limited error handling and debugging options<br/>• Requires Kafka expertise |
92
92
93
93
### Other options {#other-options}
94
94
95
-
*[**Confluent Cloud**](./confluent/index.md) - Confluent platform provides an option to upload and [run ClickHouse Connector Sink on Confluent Cloud](./confluent/custom-connector.md) or use [HTTP Sink Connector for Confluent Platform](./confluent/kafka-connect-http.md) that integrates Apache Kafka with an API via HTTP or HTTPS.
95
+
*[**Confluent Cloud**](./confluent/index.md) - Confluent Platform provides an option to upload and [run ClickHouse Connector Sink on Confluent Cloud](./confluent/custom-connector.md) or use [HTTP Sink Connector for Confluent Platform](./confluent/kafka-connect-http.md) that integrates Apache Kafka with an API via HTTP or HTTPS.
96
96
97
-
*[**Vector**](./kafka-vector.md) - Vector is a vendoragnostic data pipeline. With the ability to read from Kafka, and send events to ClickHouse, this represents a robust integration option.
97
+
*[**Vector**](./kafka-vector.md) - Vector is a vendor-agnostic data pipeline. With the ability to read from Kafka, and send events to ClickHouse, this represents a robust integration option.
98
98
99
99
*[**JDBC Connect Sink**](./kafka-connect-jdbc.md) - The Kafka Connect JDBC Sink connector allows you to export data from Kafka topics to any relational database with a JDBC driver.
100
100
101
-
***Custom code** - Custom code using respective client libraries for Kafka and ClickHouse may be appropriate cases where custom processing of events is required. This is beyond the scope of this documentation.
101
+
***Custom code** - Custom code using Kafka and ClickHouse [client libraries](../../language-clients/index.md)may be appropriate in cases where custom processing of events is required.
0 commit comments