Skip to content

Commit bb1fa4d

Browse files
megvanhuygenmegvanhuygen
authored
Merge pull request #110119 from spelluru/ehubkafkamigratelatest
Migration Guide for Apache Kafka
2 parents 9052a48 + 3c88e94 commit bb1fa4d

File tree

5 files changed

+214
-1
lines changed

5 files changed

+214
-1
lines changed

‎articles/event-hubs/TOC.yml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,11 @@
165165
- name: Integrate with Apache Kafka
166166
items:
167167
- name: Kafka developer guide for Event Hubs
168-
href: apache-kafka-developer-guide.md
168+
href: apache-kafka-developer-guide.md
169+
- name: Kafka migration guide for Event Hubs
170+
href: apache-kafka-migration-guide.md
171+
- name: Kafka troubleshooting guide for Event Hubs
172+
href: apache-kafka-troubleshooting-guide.md
169173
- name: Mirror a Kafka broker in an event hub
170174
href: event-hubs-kafka-mirror-maker-tutorial.md
171175
- name: Connect Apache Spark to an event hub
@@ -176,6 +180,8 @@
176180
href: event-hubs-kafka-connect-tutorial.md
177181
- name: Connect Akka Streams to an event hub
178182
href: event-hubs-kafka-akka-streams-tutorial.md
183+
- name: FAQ - Event Hubs for Kafka
184+
href: apache-kafka-frequently-asked-questions.md
179185
- name: Manage and monitor
180186
items:
181187
- name: Monitor Event Hubs with Azure Monitoring

‎articles/event-hubs/apache-kafka-developer-guide.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,11 @@ See the following How-to guides in our documentation:
7575
## Next steps
7676
Review samples in the GitHub repo [azure-event-hubs-for-kafka](https://github.com/Azure/azure-event-hubs-for-kafka) under quickstart and tutorials folders.
7777

78+
Also, see the following articles:
79+
80+
- [Apache Kafka troubleshooting guide for Event Hubs](apache-kafka-troubleshooting-guide.md)
81+
- [Frequently asked questions - Event Hubs for Apache Kafka](apache-kafka-frequently-asked-questions.md)
82+
- [Apache Kafka migration guide for Event Hubs](apache-kafka-migration-guide.md)
83+
84+
85+

‎articles/event-hubs/apache-kafka-frequently-asked-questions.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: Frequently asked questions - Azure Event Hubs for Apache Kafka
3+
description: This article shows how consumers and producers that use different protocols (AMQP, Apache Kafka, and HTTPS) can exchange events when using Azure Event Hubs.
4+
services: event-hubs
5+
documentationcenter: ''
6+
author: ShubhaVijayasarathy
7+
manager:
8+
9+
ms.service: event-hubs
10+
ms.devlang: na
11+
ms.topic: article
12+
ms.tgt_pltfrm: na
13+
ms.workload: na
14+
ms.date: 04/01/2020
15+
ms.author: shvija
16+
17+
---
18+
19+
# Frequently asked questions - Event Hubs for Apache Kafka
20+
This article provides answers to some of the frequently asked questions on migrating to Event Hubs for Apache Kafka.
21+
22+
## Do you run Apache Kafka?
23+
24+
No. We execute Kafka API operations against Event Hubs infrastructure. Because there's a tight correlation between Apache Kafka and Event Hubs AMQP functionality (that is, produce, receive, management, so on.), we're able to bring the known reliability of Event Hubs to the Kafka PaaS space.
25+
26+
## Event Hubs consumer group vs. Kafka consumer group
27+
What's the difference between an Event Hub consumer group and a Kafka consumer group on Event Hubs? Kafka consumer groups on Event Hubs are fully distinct from standard Event Hubs consumer groups.
28+
29+
**Event Hubs consumer groups**
30+
31+
- They are Managed with create, retrieve, update, and delete (CRUD) operations via portal, SDK, or Azure Resource Manager templates. Event Hubs consumer groups can't be autocreated.
32+
- They are children entities of an event hub. It means that the same consumer group name can be reused between event hubs in the same namespace because they're separate entities.
33+
- They aren't used for storing offsets. Orchestrated AMQP consumption is done using external offset storage, for example, Azure Storage.
34+
35+
**Kafka consumer groups**
36+
37+
- They are autocreated. Kafka groups can be managed via the Kafka consumer group APIs.
38+
- They can store offsets in the Event Hubs service.
39+
- They are used as keys in what is effectively an offset key-value store. For a unique pair of `group.id` and `topic-partition`, we store an offset in Azure Storage (3x replication). Event Hubs users don't incur extra storage costs from storing Kafka offsets. Offsets are manipulable via the Kafka consumer group APIs, but the offset storage *accounts* aren't directly visible or manipulable for Event Hub users.
40+
- They span a namespace. Using the same Kafka group name for multiple applications on multiple topics means that all applications and their Kafka clients will be rebalanced whenever only a single application needs rebalancing. Choose your group names wisely.
41+
- They fully distinct from Event Hubs consumer groups. You **don't** need to use '$Default', nor do you need to worry about Kafka clients interfering with AMQP workloads.
42+
- They aren't viewable in the Azure portal. Consumer group info is accessible via Kafka APIs.
43+
44+
## Next steps
45+
To learn more about Event Hubs and Event Hubs for Kafka, see the following articles:
46+
47+
- [Apache Kafka developer guide for Event Hubs](apache-kafka-developer-guide.md)
48+
- [Apache Kafka migration guide for Event Hubs](apache-kafka-migration-guide.md)
49+
- [Apache Kafka troubleshooting guide for Event Hubs](apache-kafka-troubleshooting-guide.md)
50+
- [Recommended configurations](https://github.com/Azure/azure-event-hubs-for-kafka/blob/master/CONFIGURATION.md)
51+

‎articles/event-hubs/apache-kafka-migration-guide.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
---
2+
title: Migrate to Azure Event Hubs for Apache Kafka
3+
description: This article shows how consumers and producers that use different protocols (AMQP, Apache Kafka, and HTTPS) can exchange events when using Azure Event Hubs.
4+
services: event-hubs
5+
documentationcenter: ''
6+
author: ShubhaVijayasarathy
7+
manager:
8+
9+
ms.service: event-hubs
10+
ms.devlang: na
11+
ms.topic: article
12+
ms.tgt_pltfrm: na
13+
ms.workload: na
14+
ms.date: 04/01/2020
15+
ms.author: shvija
16+
17+
---
18+
19+
# Migrate to Azure Event Hubs for Apache Kafka Ecosystems
20+
Azure Event Hubs exposes an Apache Kafka endpoint, which enables you to connect to Event Hubs using the Kafka protocol. By making minimal changes to your existing Kafka application, you can connect to Azure Event Hubs and reap the benefits of the Azure ecosystem. Event Hubs for Kafka support [Apache Kafka version 1.0](https://kafka.apache.org/10/documentation.html) and later.
21+
22+
## Pre-migration
23+
24+
### Create an Azure account
25+
If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.
26+
27+
### Create an Event Hubs namespace
28+
Follow step-by-step instructions in the [Create an event hub](event-hubs-create.md) article to create an Event Hubs namespace and an event hub.
29+
30+
### Connection string
31+
Follow steps from the [Get connection string from the portal](event-hubs-get-connection-string.md#get-connection-string-from-the-portal) article. And, note down the connection string for later use.
32+
33+
### Fully qualified domain name (FQDN)
34+
You may also need the FQDN that points to your Event Hub namespace. The FQDN can be found within your connection string as follows:
35+
36+
`Endpoint=sb://`**`mynamespace.servicebus.windows.net`**`/;SharedAccessKeyName=XXXXXX;SharedAccessKey=XXXXXX`
37+
38+
If your Event Hubs namespace is deployed on a non-public cloud, your domain name may differ (for example, \*.servicebus.chinacloudapi.cn, \*.servicebus.usgovcloudapi.net, or \*.servicebus.cloudapi.de).
39+
40+
## Migration
41+
42+
### Update your Kafka client configuration
43+
44+
To connect to a Kafka-enabled Event Hub, you'll need to update the Kafka client configurations. If you're having trouble finding yours, try searching for where `bootstrap.servers` is set in your application.
45+
46+
Insert the following configs wherever makes sense in your application. Make sure to update the `bootstrap.servers` and `sasl.jaas.config` values to direct the client to your Event Hubs Kafka endpoint with the correct authentication.
47+
48+
```
49+
bootstrap.servers={MYNAMESPACE}.servicebus.windows.net:9093
50+
request.timeout.ms=60000
51+
security.protocol=SASL_SSL
52+
sasl.mechanism=PLAIN
53+
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{CONNECTION STRING TO YOUR NAMESPACE}";
54+
```
55+
56+
If `sasl.jaas.config` isn't a supported configuration in your framework, find the configurations that are used to set the SASL username and password and use them instead. Set the username to `$ConnectionString` and the password to your Event Hubs connection string.
57+
58+
## Post-migration
59+
Run your Kafka application that sends events to the event hub. Then, verify that the event hub receives the events using the Azure portal. On the **Overview** page of your Event Hubs namespace, switch to the **Messages** view in the **Metrics** section. Refresh the page to update the chart. It may take a few seconds for it to show that the messages have been received.
60+
61+
[![Verify that the event hub received the messages](./media/getstarted-dotnet-standard-send-v2/verify-messages-portal.png)](./media/getstarted-dotnet-standard-send-v2/verify-messages-portal.png#lightbox)
62+
63+
64+
## Next steps
65+
To learn more about Event Hubs and Event Hubs for Kafka, see the following articles:
66+
67+
- [Apache Kafka troubleshooting guide for Event Hubs](apache-kafka-troubleshooting-guide.md)
68+
- [Frequently asked questions - Event Hubs for Apache Kafka](apache-kafka-frequently-asked-questions.md)
69+
- [Apache Kafka developer guide for Azure Event Hubs](apache-kafka-developer-guide.md)
70+
- [Recommended configurations](https://github.com/Azure/azure-event-hubs-for-kafka/blob/master/CONFIGURATION.md)
71+
-

‎articles/event-hubs/apache-kafka-troubleshooting-guide.md

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
title: Troubleshoot issues with Azure Event Hubs for Apache Kafka
3+
description: This article shows how to troubleshoot issues with Azure Event Hubs for Apache Kafka
4+
services: event-hubs
5+
documentationcenter: ''
6+
author: ShubhaVijayasarathy
7+
manager:
8+
9+
ms.service: event-hubs
10+
ms.devlang: na
11+
ms.topic: article
12+
ms.tgt_pltfrm: na
13+
ms.workload: na
14+
ms.date: 04/01/2020
15+
ms.author: shvija
16+
17+
---
18+
19+
# Apache Kafka troubleshooting guide for Event Hubs
20+
This article provides troubleshooting tips for issues that you may run into when using Event Hubs for Apache Kafka.
21+
22+
## Server Busy exception
23+
You may receive Server Busy exception due to Kafka throttling. With AMQP clients, Event Hubs immediately returns a **Server Busy** exception upon service throttling. It's equivalent to a "try again later" message. In Kafka, messages are delayed before being completed. The delay length is returned in milliseconds as `throttle_time_ms` in the produce/fetch response. In most cases, these delayed requests aren't logged as ServerBusy exceptions on Event Hubs dashboards. Instead, the response's `throttle_time_ms` value should be used as an indicator that throughput has exceeded the provisioned quota.
24+
25+
If the traffic is excessive, the service has the following behavior:
26+
27+
- If produce request's delay exceeds request timeout, Event Hubs returns **Policy Violation** error code.
28+
- If fetch request's delay exceeds request timeout, Event Hubs logs the request as throttled and responds with empty set of records and no error code.
29+
30+
[Dedicated clusters](event-hubs-dedicated-overview.md) don't have throttling mechanisms. You're free to consume all of your cluster resources.
31+
32+
## No records received
33+
You may see consumers not getting any records and constantly rebalancing. In this scenario, consumers don't get any records and constantly rebalance. There's no exception or error when it happens, but the Kafka logs will show that the consumers are stuck trying to rejoin the group and assign partitions. There are a few possible causes:
34+
35+
- Make sure that your `request.timeout.ms` is at least the recommended value of 60000 and your `session.timeout.ms` is at least the recommended value of 30000. Having these settings too low could cause consumer timeouts, which then cause rebalances (which then cause more timeouts, which cause more rebalancing, and so on)
36+
- If your configuration matches those recommended values, and you're still seeing constant rebalancing, feel free to open up an issue (make sure to include your entire configuration in the issue so that we can help debug)!
37+
38+
## Compression/Message format version issue
39+
Kafka supports compression, and Event Hubs for Kafka currently doesn't. Errors that mention a message-format version (for example, `The message format version on the broker does not support the request.`) are caused when a client tries to send compressed Kafka messages to our brokers.
40+
41+
If compressed data is necessary, compressing your data before sending it to the brokers and decompressing after receiving is a valid workaround. The message body is just a byte array to the service, so client-side compression/decompression won't cause any issues.
42+
43+
## UnknownServerException
44+
You may receive an UnknownServerException from Kafka client libraries similar to the following example:
45+
46+
```
47+
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
48+
```
49+
50+
Open a ticket with Microsoft support. Debug-level logging and exception timestamps in UTC are helpful in debugging the issue.
51+
52+
## Other issues
53+
Check the following items if you see issues when using Kafka on Event Hubs.
54+
55+
- **Firewall blocking traffic** - Make sure that port **9093** isn't blocked by your firewall.
56+
- **TopicAuthorizationException** - The most common causes of this exception are:
57+
- A typo in the connection string in your configuration file, or
58+
- Trying to use Event Hubs for Kafka on a Basic tier namespace. Event Hubs for Kafka is [only supported for Standard and Dedicated tier namespaces](https://azure.microsoft.com/pricing/details/event-hubs/).
59+
- **Kafka version mismatch** - Event Hubs for Kafka Ecosystems supports Kafka versions 1.0 and later. Some applications using Kafka version 0.10 and later could occasionally work because of the Kafka protocol's backwards compatibility, but we strongly recommend against using old API versions. Kafka versions 0.9 and earlier don't support the required SASL protocols and can't connect to Event Hubs.
60+
- **Strange encodings on AMQP headers when consuming with Kafka** - when sending events to an event hub over AMQP, any AMQP payload headers are serialized in AMQP encoding. Kafka consumers don't deserialize the headers from AMQP. To read header values, manually decode the AMQP headers. Alternatively, you can avoid using AMQP headers if you know that you'll be consuming via Kafka protocol. For more information, see [this GitHub issue](https://github.com/Azure/azure-event-hubs-for-kafka/issues/56).
61+
- **SASL authentication** - Getting your framework to cooperate with the SASL authentication protocol required by Event Hubs can be more difficult than meets the eye. See if you can troubleshoot the configuration using your framework's resources on SASL authentication.
62+
63+
## Limits
64+
Apache Kafka vs. Event Hubs Kafka. For the most part, the Event Hubs for Kafka Ecosystems has the same defaults, properties, error codes, and general behavior that Apache Kafka does. The instances where these two explicitly differ (or where Event Hubs imposes a limit that Kafka doesn't) are listed below:
65+
66+
- The max length of the `group.id` property is 256 characters
67+
- The max size of `offset.metadata.max.bytes` is 1024 bytes
68+
- Offset commits are throttled at 4 calls/second per partition with a max internal log size of 1 MB
69+
70+
71+
## Next steps
72+
To learn more about Event Hubs and Event Hubs for Kafka, see the following articles:
73+
74+
- [Apache Kafka developer guide for Event Hubs](apache-kafka-developer-guide.md)
75+
- [Apache Kafka migration guide for Event Hubs](apache-kafka-migration-guide.md)
76+
- [Frequently asked questions - Event Hubs for Apache Kafka](apache-kafka-frequently-asked-questions.md)
77+
- [Recommended configurations](https://github.com/Azure/azure-event-hubs-for-kafka/blob/master/CONFIGURATION.md)

0 commit comments

Comments
 (0)