Acrolynx

Author: spelluru

articles/event-hubs/apache-kafka-configurations.md

@@ -4,7 +4,7 @@ description: This article provides recommended Apache Kafka configurations for c
 ms.topic: reference
 ms.subservice: kafka
 ms.custom: devx-track-extended-java
-ms.date: 03/30/2022
+ms.date: 03/06/2025
 ---
 
 # Recommended configurations for Apache Kafka clients
@@ -17,16 +17,16 @@ Here are the recommended configurations for using Azure Event Hubs from Apache K
 Property | Recommended values | Permitted range | Notes
 ---|---:|-----:|---
 `metadata.max.age.ms` | 180000 (approximate) | < 240000 | Can be lowered to pick up metadata changes sooner.
-`connections.max.idle.ms`	| 180000 | < 240000 | Azure closes inbound Transmission Control Protocol (TCP) idle > 240,000 ms, which can result in sending on dead connections (shown as expired batches because of send timeout).
+`connections.max.idle.ms`	| 180000 | < 240000 | Azure closes inbound Transmission Control Protocol (TCP) idle > 240,000 ms, which can result in sending on dead connections (shown as expired batches because of send time-out).
 
 ### Producer configurations only
 Producer configs can be found [here](https://kafka.apache.org/documentation/#producerconfigs).
 
 |Property | Recommended Values | Permitted Range | Notes|
 |---|---:|---:|---|
 |`max.request.size` | 1000000 | < 1046528 | The service closes connections if requests larger than 1,046,528 bytes are sent. *This value **must** be changed and causes issues in high-throughput produce scenarios.*|
-|`retries` | > 0 | | Might require increasing delivery.timeout.ms value, see documentation.|
-|`request.timeout.ms` | 30000 .. 60000 | > 20000| Event Hubs internally defaults to a minimum of 20,000 ms. *While requests with lower timeout values are accepted, client behavior isn't guaranteed.* <p>Make sure that your **request.timeout.ms** is at least the recommended value of 60000 and your **session.timeout.ms** is at least the recommended value of 30000. Having these settings too low could cause consumer timeouts, which then cause rebalances (which then cause more timeouts, which cause more rebalancing, and so on).</p>|
+|`retries` | > 0 | | Might require increasing `delivery.timeout.ms` value, see documentation.|
+|`request.timeout.ms` | 30000 .. 60000 | > 20000| Event Hubs internally defaults to a minimum of 20,000 ms. *While requests with lower time out values are accepted, client behavior isn't guaranteed.* <p>Make sure that your **request.timeout.ms** is at least the recommended value of 60000 and your **session.timeout.ms** is at least the recommended value of 30000. Having these settings too low could cause consumer time-outs, which then cause rebalances (which then cause more time-outs, which cause more rebalancing, and so on).</p>|
 |`metadata.max.idle.ms` | 180000 | > 5000 | Controls how long the producer caches metadata for a topic that's idle. If the elapsed time since a topic was last produced exceeds the metadata idle duration, then the topic's metadata is forgotten and the next access to it will force a metadata fetch request.|
 |`linger.ms` | > 0 | | For high throughput scenarios, linger value should be equal to the highest tolerable value to take advantage of batching.|
 |`delivery.timeout.ms` | | | Set according to the formula (`request.timeout.ms` + `linger.ms`) * `retries`.|
@@ -38,8 +38,8 @@ Consumer configs can be found [here](https://kafka.apache.org/documentation/#con
 Property | Recommended Values | Permitted Range | Notes
 ---|---:|-----:|---
 `heartbeat.interval.ms` | 3000 | | 3000 is the default value and shouldn't be changed.
-`session.timeout.ms` | 30000 |6000 .. 300000| Start with 30000, increase if seeing frequent rebalancing because of missed heartbeats.<p>Make sure that your request.timeout.ms is at least the recommended value of 60000 and your session.timeout.ms is at least the recommended value of 30000. Having these settings too low could cause consumer timeouts, which then cause rebalances (which then cause more timeouts, which cause more rebalancing, and so on).</p>
-`max.poll.interval.ms` | 300000 (default) |>session.timeout.ms| Used for rebalance timeout, so it shouldn't be set too low. Must be greater than session.timeout.ms.
+`session.timeout.ms` | 30000 |6000 .. 300000| Start with 30000, increase if seeing frequent rebalancing because of missed heartbeats.<p>Make sure that your request.timeout.ms is at least the recommended value of 60000 and your session.timeout.ms is at least the recommended value of 30000. Having these settings too low could cause consumer time-outs, which then cause rebalances (which then cause more time-outs, which cause more rebalancing, and so on).</p>
+`max.poll.interval.ms` | 300000 (default) |>session.timeout.ms| Used for rebalance time-out, so it shouldn't be set too low. Must be greater than session.timeout.ms.
 
 ## librdkafka configuration properties
 The main `librdkafka` configuration file ([link](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)) contains extended descriptions for the properties described in the following sections.
@@ -56,7 +56,7 @@ Property | Recommended Values | Permitted Range | Notes
 |Property | Recommended Values | Permitted Range | Notes|
 |---|---:|-----:|---|
 |`retries` | > 0 | | Default is 2147483647.|
-|`request.timeout.ms` | 30000 .. 60000 | > 20000| Event Hubs internally defaults to a minimum of 20,000 ms. `librdkafka` default value is 5000, which can be problematic. *While requests with lower timeout values are accepted, client behavior isn't guaranteed.*|
+|`request.timeout.ms` | 30000 .. 60000 | > 20000| Event Hubs internally defaults to a minimum of 20,000 ms. `librdkafka` default value is 5000, which can be problematic. *While requests with lower time-out values are accepted, client behavior isn't guaranteed.*|
 |`partitioner` | `consistent_random` | See librdkafka documentation | `consistent_random` is default and best. Empty and null keys are handled ideally for most cases.|
 |`compression.codec` | `none, gzip` || Only gzip compression is currently supported.|
 
@@ -66,7 +66,7 @@ Property | Recommended Values | Permitted Range | Notes
 ---|---:|-----:|---
 `heartbeat.interval.ms` | 3000 || 3000 is the default value and shouldn't be changed.
 `session.timeout.ms` | 30000 |6000 .. 300000| Start with 30000, increase if seeing frequent rebalancing because of missed heartbeats.
-`max.poll.interval.ms` | 300000 (default) |>session.timeout.ms| Used for rebalance timeout, so it shouldn't be set too low. Must be greater than session.timeout.ms.
+`max.poll.interval.ms` | 300000 (default) |>session.timeout.ms| Used for rebalance time out, so it shouldn't be set too low. Must be greater than session.timeout.ms.
 
 
 ## Further notes
@@ -75,7 +75,7 @@ Check the following table of common configuration-related error scenarios.
 
 Symptoms | Problem | Solution
 ----|---|-----
-Offset commit failures because of rebalancing | Your consumer is waiting too long in between calls to poll() and the service is kicking the consumer out of the group. | You have several options: <ul><li>Increase poll processing timeout (`max.poll.interval.ms`)</li><li>Decrease message batch size to speed up processing</li><li>Improve processing parallelization to avoid blocking consumer.poll()</li></ul> Applying some combination of the three is likely wisest.
+Offset commit failures because of rebalancing | Your consumer is waiting too long in between calls to poll() and the service is kicking the consumer out of the group. | You have several options: <ul><li>Increase poll processing time out (`max.poll.interval.ms`)</li><li>Decrease message batch size to speed up processing</li><li>Improve processing parallelization to avoid blocking consumer.poll()</li></ul> Applying some combination of the three is likely wisest.
 Network exceptions at high produce throughput | If you're using Java client + default max.request.size, your requests might be too large. | See Java configs mentioned earlier.
 
 ## Next steps

articles/event-hubs/apache-kafka-developer-guide.md

@@ -1,7 +1,7 @@
 ---
 title: Apache Kafka developer guide for Event Hubs
 description: This article provides links to articles that describe how to integrate your Kafka applications with Azure Event Hubs.
-ms.date: 12/18/2024
+ms.date: 03/06/2025
 ms.subservice: kafka
 ms.topic: article
 ---

articles/event-hubs/apache-kafka-frequently-asked-questions.yml

@@ -5,7 +5,7 @@ metadata:
   ms.topic: faq
   ms.subservice: kafka
 
-  ms.date: 10/14/2022
+  ms.date: 03/06/2025
 title: Frequently asked questions - Event Hubs for Apache Kafka 
 summary: This article provides answers to some of the frequently asked questions on migrating to Event Hubs for Apache Kafka.
 
@@ -34,8 +34,8 @@ sections:
           - They're autocreated.  Kafka groups can be managed via the Kafka consumer group APIs.
           - They can store offsets in the Event Hubs service.
           - They're used as keys in what is effectively an offset key-value store. For a unique pair of `group.id` and `topic-partition`, we store an offset in Azure Storage (3x replication). Event Hubs users don't incur extra storage costs from storing Kafka offsets. Offsets are manipulable via the Kafka consumer group APIs, but the offset storage *accounts* aren't directly visible or manipulable for Event Hubs users.  
-          - They span a namespace. Using the same Kafka group name for multiple applications on multiple topics means that all applications and their Kafka clients will be rebalanced whenever only a single application needs rebalancing.  Choose your group names wisely.
-          - They fully distinct from Event Hubs consumer groups. You **don't** need to use '$Default', nor do you need to worry about Kafka clients interfering with AMQP workloads.
+          - They span a namespace. Using the same Kafka group name for multiple applications on multiple topics means that all applications and their Kafka clients are rebalanced whenever only a single application needs rebalancing. Choose your group names wisely.
+          - They fully distinct from Event Hubs consumer groups. You **don't** need to use `$Default`, nor do you need to worry about Kafka clients interfering with AMQP workloads.
           - They aren't viewable in the Azure portal. Consumer group info is accessible via Kafka APIs.
           
       - question: |

articles/event-hubs/apache-kafka-migration-guide.md

@@ -3,7 +3,7 @@ title: Migrate to Azure Event Hubs for Apache Kafka
 description: This article explains how to migrate clients from Apache Kafka to Azure Event Hubs. 
 ms.topic: article
 ms.subservice: kafka
-ms.date: 12/18/2024
+ms.date: 03/06/2025
 ---
 
 # Migrate to Azure Event Hubs for Apache Kafka Ecosystems

articles/event-hubs/apache-kafka-streams.md

@@ -3,7 +3,7 @@ title: Kafka Streams for Apache Kafka in Event Hubs on Azure Cloud
 description: Learn about how to use the Apache Kafka Streams API with Event Hubs service on Azure Cloud.
 ms.topic: overview
 ms.subservice: kafka
-ms.date: 04/29/2024
+ms.date: 03/06/2025
 ---
 
 # Kafka Streams for Azure Event Hubs
@@ -27,21 +27,21 @@ Azure Event Hubs natively supports both the AMQP and Kafka protocol. However, to
 | Property | Default behavior for Event Hubs | Modified behavior for Kafka streams | Explanation |
 | ----- | ---- | ----| ---- |
 | `messageTimestampType` | set to `AppendTime` | should be set to `CreateTime` | Kafka Streams relies on creation timestamp rather than append timestamp |
-| `message.timestamp.difference.max.ms` | max allowed value is 90 days | Property is used to govern past timestamps only. Future time is set to 1 hour and can't be changed. | This is in line with the Kafka protocol specification |
+| `message.timestamp.difference.max.ms` | max allowed value is 90 days | Property is used to govern past timestamps only. Future time is set to 1 hour and can't be changed. | It is in line with the Kafka protocol specification |
 | `min.compaction.lag.ms` | | max allowed value is two days ||
 | Infinite retention topics | | size based truncation of 250 GB for each topic-partition||
-| Delete record API for infinite retention topics| | Not implemented. As a workaround, the topic can be updated and a finite retention time can be set.| This will be done in GA |
+| Delete record API for infinite retention topics| | Not implemented. As a workaround, the topic can be updated and a finite retention time can be set.| This functionality will be supported in GA |
 
 ### Other considerations
 
 Here are some of the other considerations to keep in mind.
 
   * Kafka streams client applications must be granted management, read, and write permissions for the entire namespaces to be able to create temporary topics for stream processing.
-  * Temporary topics and partitions count towards the quota for the given namespace. These should be kept under consideration when provisioning the namespace or cluster.
-  * Infinite retention time for "Offset" Store is limited by max message retention time of the SKU. Check [Event Hubs Quotas](event-hubs-quotas.md) for these tier specific values.
+  * Temporary topics and partitions count towards the quota for the given namespace. They should be kept under consideration when provisioning the namespace or cluster.
+  * Infinite retention time for "Offset" Store is limited by max message retention time of the Stock Keeping Unit (SKU). Check [Event Hubs Quotas](event-hubs-quotas.md) for these tier specific values.
 
 
-These include, updating the topic configuration in the `messageTimestampType` to use the `CreateTime` (that is, Event creation time) instead of the `AppendTime` (that is, log append time).
+They include, updating the topic configuration in the `messageTimestampType` to use the `CreateTime` (that is, Event creation time) instead of the `AppendTime` (that is, log append time).
 
 To override the default behavior (required), the below setting must be set in Azure Resource Manager (ARM).
 
@@ -73,7 +73,7 @@ To override the default behavior (required), the below setting must be set in Az
 
 ## Kafka Streams concepts
 
-Kafka streams provides a simple abstraction layer over the Kafka producer and consumer APIs to help developers get started with real time streaming scenarios faster. The light-weight library depends on an **Apache Kafka compatible broker** (like Azure Event Hubs) for the internal messaging layer, and manages a **fault tolerant local state store**. With the transactional API, the Kafka streams library supports rich processing features such as **exactly once processing** and **one record at a time processing**.
+Kafka streams provide a simple abstraction layer over the Kafka producer and consumer APIs to help developers get started with real time streaming scenarios faster. The light-weight library depends on an **Apache Kafka compatible broker** (like Azure Event Hubs) for the internal messaging layer, and manages a **fault tolerant local state store**. With the transactional API, the Kafka streams library supports rich processing features such as **exactly once processing** and **one record at a time processing**.
 
 Records arriving out of order benefit from **event-time based windowing operations**.
 
@@ -100,7 +100,7 @@ Stream processing topology can be defined either with the [Kafka Streams DSL](ht
 
 Streams and tables are 2 different but useful abstractions provided by the [Kafka Streams DSL](https://kafka.apache.org/37/documentation/streams/developer-guide/dsl-api.html), modeling both time series and relational data formats that must coexist for stream processing use-cases. 
 
-Kafka extends this further and introduces a duality between streams and tables, where a
+Kafka extends it further and introduces a duality between streams and tables, where a
   * A **stream** can be considered as a changelog of a **table**, and
   * A **table** can be considered as a snapshot of the latest value of each key in a **stream**.
 
@@ -113,11 +113,11 @@ For example
 
 ### Time
 
-Kafka Streams allows windowing and grace functions to allow for out of order data records to be ingested and still be included in the processing. To ensure that this behavior is deterministic, there are additional notions of time in Kafka streams. These include: 
+Kafka Streams allows windowing and grace functions to allow for out of order data records to be ingested and still be included in the processing. To ensure that this behavior is deterministic, there are more notions of time in Kafka streams. They include: 
 
-  * Creation time (also known as 'Event time') - This is the time when the event occurred and the data record was created.
-  * Processing time - This is the time when the data record is processed by the stream processing application (or when it's consumed).
-  * Append time (also known as 'Creation time') - This is the time when the data is stored and committed to the storage of the Kafka broker. This differs from the creation time because of the time difference between the creation of the event and the actual ingestion by the broker.
+  * Creation time (also known as 'Event time') - It's the time when the event occurred and the data record was created.
+  * Processing time - It's the time when the data record is processed by the stream processing application (or when it's consumed).
+  * Append time (also known as 'Creation time') - It's the time when the data is stored and committed to the storage of the Kafka broker. It differs from the creation time because of the time difference between the creation of the event and the actual ingestion by the broker.
 
 
 
@@ -130,7 +130,7 @@ Stateful transformations in the DSL include:
   * [Aggregating](https://kafka.apache.org/37/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-aggregating)
   * [Joining](https://kafka.apache.org/37/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-joins)
   * [Windowing (as part of aggregations and joins)](https://kafka.apache.org/37/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-windowing)
-  * [Applying custom processors and transformers](https://kafka.apache.org/37/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-process), which may be stateful, for Processor API integration
+  * [Applying custom processors and transformers](https://kafka.apache.org/37/documentation/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-process), which can be stateful, for Processor API integration
 
 ### Window and grace
 
@@ -143,7 +143,7 @@ Applications must utilize the windowing and grace period controls to improve fau
 
 ### Processing guarantees
 
-Business and technical users seek to extract key business insights from the output of stream processing workloads, which translate to high transactional guarantee requirements. Kafka streams works together with Kafka transactions to ensure transactional processing guarantees by integrating with the Kafka compatible brokers' (such as Azure Event Hubs) underlying storage system to ensure that offset commits and state store updates are written atomically.
+Business and technical users seek to extract key business insights from the output of stream processing workloads, which translate to high transactional guarantee requirements. Kafka streams work together with Kafka transactions to ensure transactional processing guarantees by integrating with the Kafka compatible brokers' (such as Azure Event Hubs) underlying storage system to ensure that offset commits and state store updates are written atomically.
 
 To ensure transactional processing guarantees, the `processing.guarantee` setting in the Kafka Streams configs must be updated from the default value of `at_least_once` to `exactly_once_v2` (for client versions at or after Apache Kafka 2.5) or `exactly_once` (for client versions before Apache Kafka 2.5.x).