Skip to content

Commit 0d5bbb2

Browse files
authored
Merge pull request #142352 from clemensv/eh-doc-patch-4
Support statement clarifications for Kafka Connect
2 parents 14cc41d + 66f7813 commit 0d5bbb2

File tree

2 files changed

+24
-6
lines changed

2 files changed

+24
-6
lines changed

articles/event-hubs/event-hubs-kafka-connect-debezium.md

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,25 @@
11
---
2-
title: Integrate Apache Kafka Connect on Azure Event Hubs (Preview) with Debezium for Change Data Capture
2+
title: Integrate Apache Kafka Connect on Azure Event Hubs with Debezium for Change Data Capture
33
description: This article provides information on how to use Debezium with Azure Event Hubs for Kafka.
44
ms.topic: how-to
55
author: abhirockzz
66
ms.author: abhishgu
7-
ms.date: 08/11/2020
7+
ms.date: 01/06/2021
88
---
99

10-
# Integrate Apache Kafka Connect support on Azure Event Hubs (Preview) with Debezium for Change Data Capture
10+
# Integrate Apache Kafka Connect support on Azure Event Hubs with Debezium for Change Data Capture
1111

1212
**Change Data Capture (CDC)** is a technique used to track row-level changes in database tables in response to create, update, and delete operations. [Debezium](https://debezium.io/) is a distributed platform that builds on top of Change Data Capture features available in different databases (for example, [logical decoding in PostgreSQL](https://www.postgresql.org/docs/current/static/logicaldecoding-explanation.html)). It provides a set of [Kafka Connect connectors](https://debezium.io/documentation/reference/1.2/connectors/index.html) that tap into row-level changes in database table(s) and convert them into event streams that are then sent to [Apache Kafka](https://kafka.apache.org/).
1313

14+
> [!WARNING]
15+
> Use of the Apache Kafka Connect framework as well as the Debezium platform and its connectors are **not eligible for product support through Microsoft Azure**.
16+
>
17+
> Apache Kafka Connect assumes for its dynamic configuration to be held in compacted topics with otherwise unlimited retention. Azure Event Hubs [does not implement compaction as a broker feature](event-hubs-federation-overview.md#log-projections) and always imposes a time-based retention limit on retained events, rooting from the principle that Azure Event Hubs is a real-time event streaming engine and not a long-term data or configuration store.
18+
>
19+
> While the Apache Kafka project might be comfortable with mixing these roles, Azure believes that such information is best managed in a proper database or configuration store.
20+
>
21+
> Many Apache Kafka Connect scenarios will be functional, but these conceptual differences between Apache Kafka's and Azure Event Hubs' retention models may cause certain configurations not to work as expected.
22+
1423
This tutorial walks you through how to set up a change data capture based system on Azure using [Azure Event Hubs](./event-hubs-about.md?WT.mc_id=devto-blog-abhishgu) (for Kafka), [Azure DB for PostgreSQL](../postgresql/overview.md) and Debezium. It will use the [Debezium PostgreSQL connector](https://debezium.io/documentation/reference/1.2/connectors/postgresql.html) to stream database modifications from PostgreSQL to Kafka topics in Azure Event Hubs
1524

1625
> [!NOTE]

articles/event-hubs/event-hubs-kafka-connect-tutorial.md

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,20 @@
22
title: Integrate with Apache Kafka Connect- Azure Event Hubs | Microsoft Docs
33
description: This article provides information on how to use Kafka Connect with Azure Event Hubs for Kafka.
44
ms.topic: how-to
5-
ms.date: 06/23/2020
5+
ms.date: 01/06/2021
66
---
77

8-
# Integrate Apache Kafka Connect support on Azure Event Hubs (Preview)
9-
As ingestion for business needs increases, so does the requirement to ingest for various external sources and sinks. [Apache Kafka Connect](https://kafka.apache.org/documentation/#connect) provides such framework to connect and import/export data from/to any external system such as MySQL, HDFS, and file system through a Kafka cluster. This tutorial walks you through using Kafka Connect framework with Event Hubs.
8+
# Integrate Apache Kafka Connect support on Azure Event Hubs
9+
[Apache Kafka Connect](https://kafka.apache.org/documentation/#connect) is a framework to connect and import/export data from/to any external system such as MySQL, HDFS, and file system through a Kafka cluster. This tutorial walks you through using Kafka Connect framework with Event Hubs.
10+
11+
> [!WARNING]
12+
> Use of the Apache Kafka Connect framework and its connectors is **not eligible for product support through Microsoft Azure**.
13+
>
14+
> Apache Kafka Connect assumes for its dynamic configuration to be held in compacted topics with otherwise unlimited retention. Azure Event Hubs [does not implement compaction as a broker feature](event-hubs-federation-overview.md#log-projections) and always imposes a time-based retention limit on retained events, rooting from the principle that Azure Event Hubs is a real-time event streaming engine and not a long-term data or configuration store.
15+
>
16+
> While the Apache Kafka project might be comfortable with mixing these roles, Azure believes that such information is best managed in a proper database or configuration store.
17+
>
18+
> Many Apache Kafka Connect scenarios will be functional, but these conceptual differences between Apache Kafka's and Azure Event Hubs' retention models may cause certain configurations not to work as expected.
1019
1120
This tutorial walks you through integrating Kafka Connect with an event hub and deploying basic FileStreamSource and FileStreamSink connectors. This feature is currently in preview. While these connectors are not meant for production use, they demonstrate an end-to-end Kafka Connect scenario where Azure Event Hubs acts as a Kafka broker.
1221

0 commit comments

Comments
 (0)