Merge pull request #3998 from morsapaes/docs-kafka_table_engine

integrations: update Kafka integration overview

Author: Blargian

docs/integrations/data-ingestion/clickpipes/kafka.md

@@ -112,7 +112,7 @@ without an embedded schema id, then the specific schema ID or subject must be sp
 | Azure Event Hubs     |<Azureeventhubssvg class="image" alt="Azure Event Hubs logo" style={{width: '3rem'}}/>|Streaming| Stable          | Configure ClickPipes and start ingesting streaming data from Azure Event Hubs into ClickHouse Cloud. |
 | WarpStream           |<Warpstreamsvg class="image" alt="WarpStream logo" style={{width: '3rem'}}/>|Streaming| Stable          | Configure ClickPipes and start ingesting streaming data from WarpStream into ClickHouse Cloud.       |
 
-More connectors are will get added to ClickPipes, you can find out more by [contacting us](https://clickhouse.com/company/contact?loc=clickpipes).
+More connectors will get added to ClickPipes in the future. You can find out more by [contacting us](https://clickhouse.com/company/contact?loc=clickpipes).
 
 ## Supported data formats {#supported-data-formats}
 

docs/integrations/data-ingestion/kafka/index.md

@@ -8,47 +8,98 @@ title: 'Integrating Kafka with ClickHouse'
 
 # Integrating Kafka with ClickHouse
 
-[Apache Kafka](https://kafka.apache.org/) is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. In most cases involving Kafka and ClickHouse, users will wish to insert Kafka based data into ClickHouse. Below we outline several options for both use cases, identifying the pros and cons of each approach.
+[Apache Kafka](https://kafka.apache.org/) is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. ClickHouse provides multiple options to **read from** and **write to** Kafka and other Kafka API-compatible brokers (e.g., Redpanda, Amazon MSK).
 
-## Choosing an option {#choosing-an-option}
+## Available options {#available-options}
 
-When integrating Kafka with ClickHouse, you will need to make early architectural decisions about the high-level approach used. We outline the most common strategies below:
+Choosing the right option for your use case depends on multiple factors, including your ClickHouse deployment type, data flow direction and operational requirements.
 
-### ClickPipes for Kafka (ClickHouse Cloud) {#clickpipes-for-kafka-clickhouse-cloud}
-* [**ClickPipes**](../clickpipes/kafka.md) offers the easiest and most intuitive way to ingest data into ClickHouse Cloud. With support for Apache Kafka, Confluent Cloud and Amazon MSK today, and many more data sources coming soon.
+|Option   | Deployment type | Fully managed | Kafka to ClickHouse | ClickHouse to Kafka |
+|---------|------------|:-------------------:|:-------------------:|:------------------:|
+| [ClickPipes for Kafka](../clickpipes/kafka.md) | [Cloud], [BYOC] (coming soon!)   | ✅ | ✅ |   |
+| [Kafka Connect Sink](./kafka-clickhouse-connect-sink.md)   | [Cloud], [BYOC], [Self-hosted] | | ✅ |   |
+| [Kafka table engine](./kafka-table-engine.md)   | [Cloud], [BYOC], [Self-hosted] | | ✅ | ✅ |
 
-### 3rd-Party Cloud-based Kafka Connectivity {#3rd-party-cloud-based-kafka-connectivity}
-* [**Confluent Cloud**](./confluent/index.md) - Confluent platform provides an option to upload and [run ClickHouse Connector Sink on Confluent Cloud](./confluent/custom-connector.md) or use [HTTP Sink Connector for Confluent Platform](./confluent/kafka-connect-http.md) that integrates Apache Kafka with an API via HTTP or HTTPS.
+For a more detailed comparison between these options, see [Choosing an option](#choosing-an-option).
 
-* [**Amazon MSK**](./msk/index.md) - support Amazon MSK Connect framework to forward data from Apache Kafka clusters to external systems such as ClickHouse. You can install ClickHouse Kafka Connect on Amazon MSK.
+### ClickPipes for Kafka {#clickpipes-for-kafka}
 
-* [**Redpanda Cloud**](https://cloud.redpanda.com/) - Redpanda is a Kafka API-compatible streaming data platform that can be used as an upstream data source for ClickHouse. The hosted cloud platform, Redpanda Cloud, integrates with ClickHouse over Kafka protocol, enabling real-time data ingestion for streaming analytics workloads
+[ClickPipes](../clickpipes/index.md) is a managed integration platform that makes ingesting data from a diverse set of sources as simple as clicking a few buttons. Because it is fully managed and purpose-built for production workloads, ClickPipes significantly lowers infrastructure and operational costs, removing the need for external data streaming and ETL tools.
 
-### Self-managed Kafka Connectivity {#self-managed-kafka-connectivity}
-* [**Kafka Connect**](./kafka-clickhouse-connect-sink.md) - Kafka Connect is a free, open-source component of Apache Kafka that works as a centralized data hub for simple data integration between Kafka and other data systems.  Connectors provide a simple means of scalable and reliably streaming data to and from Kafka.  Source Connectors inserts data to Kafka topics from other systems, whilst Sink Connectors delivers data from Kafka topics into other data stores such as ClickHouse.
-* [**Vector**](./kafka-vector.md) - Vector is a vendor agnostic data pipeline. With the ability to read from Kafka, and send events to ClickHouse, this represents a robust integration option.
-* [**JDBC Connect Sink**](./kafka-connect-jdbc.md) - The Kafka Connect JDBC Sink connector allows you to export data from Kafka topics to any relational database with a JDBC driver
-* **Custom code** - Custom code using respective client libraries for Kafka and ClickHouse may be appropriate cases where custom processing of events is required. This is beyond the scope of this documentation.
-* [**Kafka table engine**](./kafka-table-engine.md) provides a Native ClickHouse integration (not available on ClickHouse Cloud). This table engine **pulls** data from the source system. This requires ClickHouse to have direct access to Kafka.
-* [**Kafka table engine with named collections**](./kafka-table-engine-named-collections.md) - Using named collections provides native ClickHouse integration with Kafka. This approach allows secure connections to multiple Kafka clusters, centralizing configuration management and improving scalability and security.
+:::tip
+This is the recommended option if you're a ClickHouse Cloud user. ClickPipes is **fully managed** and purpose-built to deliver the **best performance** in Cloud environments.
+:::
 
-### Choosing an approach {#choosing-an-approach}
-It comes down to a few decision points:
+#### Main features {#clickpipes-for-kafka-main-features}
 
-* **Connectivity** - The Kafka table engine needs to be able to pull from Kafka if ClickHouse is the destination. This requires bi-directional connectivity. If there is a network separation, e.g. ClickHouse is in the Cloud and Kafka is self-managed, you may be hesitant to remove this for compliance and security reasons. (This approach is not currently supported in ClickHouse Cloud.) The Kafka table engine utilizes resources within ClickHouse itself, utilizing threads for the consumers. Placing this resource pressure on ClickHouse may not be possible due to resource constraints, or your architects may prefer a separation of concerns. In this case, tools such as Kafka Connect, which run as a separate process and can be deployed on different hardware may be preferable. This allows the process responsible for pulling Kafka data to be scaled independently of ClickHouse.
+[//]: # "TODO It isn't optimal to link to a static alpha-release of the Terraform provider. Link to a Terraform guide once that's available."
 
-* **Hosting on Cloud** - Cloud vendors may set limitations on Kafka components available on their platform. Follow the guide to explore recommended options for each Cloud vendor.
+* Optimized for ClickHouse Cloud, delivering blazing-fast performance
+* Horizontal and vertical scalability for high-throughput workloads
+* Built-in fault tolerance with configurable replicas and automatic retries
+* Deployment and management via ClickHouse Cloud UI, [Open API](../../../cloud/manage/api/api-overview.md), or [Terraform](https://registry.terraform.io/providers/ClickHouse/clickhouse/3.3.3-alpha2/docs/resources/clickpipe)
+* Enterprise-grade security with support for cloud-native authorization (IAM) and private connectivity (PrivateLink)
+* Supports a wide range of [data sources](../clickpipes/kafka.md#supported-data-sources), including Confluent Cloud, Amazon MSK, Redpanda Cloud, and Azure Event Hubs
+* Supports most common serialization formats (JSON, Avro, Protobuf coming soon!)
 
-* **External enrichment** - Whilst messages can be manipulated before insertion into ClickHouse, through the use of functions in the select statement of the materialized view, users may prefer to move complex enrichment external to ClickHouse.
+#### Getting started {#clickpipes-for-kafka-getting-started}
 
-* **Data flow direction** - Vector only supports the transfer of data from Kafka to ClickHouse.
+To get started using ClickPipes for Kafka, see the [reference documentation](../clickpipes/kafka.md) or navigate to the `Data Sources` tab in the ClickHouse Cloud UI.
 
-## Assumptions {#assumptions}
+### Kafka Connect Sink {#kafka-connect-sink}
 
-The user guides linked above assume the following:
+Kafka Connect is an open-source framework that works as a centralized data hub for simple data integration between Kafka and other data systems. The [ClickHouse Kafka Connect Sink](https://github.com/ClickHouse/clickhouse-kafka-connect) connector provides a scalable and highly-configurable option to read data from Apache Kafka and other Kafka API-compatible brokers.
 
-* You are familiar with the Kafka fundamentals, such as producers, consumers and topics.
-* You have a topic prepared for these examples. We assume all data is stored in Kafka as JSON, although the principles remain the same if using Avro.
-* We utilise the excellent [kcat](https://github.com/edenhill/kcat) (formerly kafkacat) in our examples to publish and consume Kafka data.
-* Whilst we reference some python scripts for loading sample data, feel free to adapt the examples to your dataset.
-* You are broadly familiar with ClickHouse materialized views.
+:::tip
+This is the recommended option if you prefer **high configurability** or are already a Kafka Connect user.
+:::
+
+#### Main features {#kafka-connect-sink-main-features}
+
+* Can be configured to support exactly-once semantics
+* Supports most common serialization formats (JSON, Avro, Protobuf)
+* Tested continuously against ClickHouse Cloud
+
+#### Getting started {#kafka-connect-sink-getting-started}
+
+To get started using the ClickHouse Kafka Connect Sink, see the [reference documentation](./kafka-clickhouse-connect-sink.md).
+
+### Kafka table engine {#kafka-table-engine}
+
+The [Kafka table engine](./kafka-table-engine.md) can be used to read data from and write data to Apache Kafka and other Kafka API-compatible brokers. This option is bundled with open-source ClickHouse and is available across all deployment types.
+
+:::tip
+This is the recommended option if you're self-hosting ClickHouse and need a **low entry barrier** option, or if you need to **write** data to Kafka.
+:::
+
+#### Main features {#kafka-table-engine-main-features}
+
+* Can be used for [reading](./kafka-table-engine.md/#kafka-to-clickhouse) and [writing](./kafka-table-engine.md/#clickhouse-to-kafka) data
+* Bundled with open-source ClickHouse
+* Supports most common serialization formats (JSON, Avro, Protobuf)
+
+#### Getting started {#kafka-table-engine-getting-started}
+
+To get started using the Kafka table engine, see the [reference documentation](./kafka-table-engine.md).
+
+### Choosing an option {#choosing-an-option}
+
+| Product | Strengths | Weaknesses |
+|---------|-----------|------------|
+| **ClickPipes for Kafka** | • Scalable architecture for high throughput and low latency<br/>• Built-in monitoring and schema management<br/>• Private networking connections (via PrivateLink)<br/>• Supports SSL/TLS authentication and IAM authorization<br/>• Supports programmatic configuration (Terraform, API endpoints) | • Does not support pushing data to Kafka<br/>• At-least-once semantics |
+| **Kafka Connect Sink** | • Exactly-once semantics<br/>• Allows granular control over data transformation, batching and error handling<br/>• Can be deployed in private networks<br/>• Allows real-time replication from databases not yet supported in ClickPipes via Debezium | • Does not support pushing data to Kafka<br/>• Operationally complex to set up and maintain<br/>• Requires Kafka and Kafka Connect expertise |
+| **Kafka table engine** | • Supports [pushing data to Kafka](./kafka-table-engine.md/#clickhouse-to-kafka)<br/>• Operationally simple to set up | • At-least-once semantics<br/>• Limited horizontal scaling for consumers. Cannot be scaled independently from the ClickHouse server<br/>• Limited error handling and debugging options<br/>• Requires Kafka expertise |
+
+### Other options {#other-options}
+
+* [**Confluent Cloud**](./confluent/index.md) - Confluent Platform provides an option to upload and [run ClickHouse Connector Sink on Confluent Cloud](./confluent/custom-connector.md) or use [HTTP Sink Connector for Confluent Platform](./confluent/kafka-connect-http.md) that integrates Apache Kafka with an API via HTTP or HTTPS.
+
+* [**Vector**](./kafka-vector.md) - Vector is a vendor-agnostic data pipeline. With the ability to read from Kafka, and send events to ClickHouse, this represents a robust integration option.
+
+* [**JDBC Connect Sink**](./kafka-connect-jdbc.md) - The Kafka Connect JDBC Sink connector allows you to export data from Kafka topics to any relational database with a JDBC driver.
+
+* **Custom code** - Custom code using Kafka and ClickHouse [client libraries](../../language-clients/index.md) may be appropriate in cases where custom processing of events is required.
+
+[BYOC]: ../../../cloud/reference/byoc.md
+[Cloud]: ../../../cloud-index.md
+[Self-hosted]: ../../../intro.md

docs/tutorial.md

@@ -12,7 +12,7 @@ show_related_blogs: true
 
 ## Overview {#overview}
 
-Learn how to ingest and query data in ClickHouse using a New York City taxi example dataset. 
+Learn how to ingest and query data in ClickHouse using the New York City taxi example dataset. 
 
 ### Prerequisites {#prerequisites}
 

scripts/aspell-dict-file.txt

@@ -919,6 +919,8 @@ supabase
 artitecture
 emqx
 mqttx
+--docs/integrations/data-ingestion/kafka/index.md--
+configurability
 --docs/integrations/data-ingestion/kafka/confluent/custom-connector.md--
 AddCustomConnectorPlugin
 --docs/integrations/data-ingestion/kafka/confluent/kafka-connect-http.md--

styles/ClickHouse/Headings.yml

@@ -62,6 +62,8 @@ exceptions:
   - Time To Live
   - Docker Compose
   - Kafka
+  - Kafka Connect
+  - Kafka Connect Sink
   - Google Cloud Run
   - NPM
   - OTel