Merge pull request #214533 from kasun04/main

Log compaction support

Author: v-regandowner

articles/event-hubs/TOC.yml

@@ -92,6 +92,8 @@
         href: event-hubs-capture-overview.md
       - name: Explore captured Avro files
         href: explore-captured-avro-files.md
+    - name: Log compaction
+      href: log-compaction.md
     - name: Application groups
       href: resource-governance-overview.md     
     - name: Tiers 
@@ -173,8 +175,6 @@
         href: event-hubs-amqp-troubleshoot.md
 - name: How-to guides
   items:
-    - name: Create and manage application groups
-      href: resource-governance-with-app-groups.md
     - name: Develop
       items:
         - name: Get Event Hubs connection string
@@ -203,6 +203,11 @@
               href: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/eventhub/azure-eventhub/README.md
             - name: JavaScript in GitHub (azure/event-hubs)
               href: https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/eventhub/event-hubs/README.md
+    - name: Use log compaction
+      href: use-log-compaction.md
+    - name: Create and manage application groups
+      href: resource-governance-with-app-groups.md
+
     - name: Process data
       items:
         - name: Capture Event Hubs data in Parquet format

articles/event-hubs/event-hubs-features.md

@@ -33,7 +33,7 @@ Any entity that sends data to an event hub is an *event publisher* (synonymously
 
 You can publish an event via AMQP 1.0, the Kafka protocol, or HTTPS. The Event Hubs service provides [REST API](/rest/api/eventhub/) and [.NET](event-hubs-dotnet-standard-getstarted-send.md), [Java](event-hubs-java-get-started-send.md), [Python](event-hubs-python-get-started-send.md), [JavaScript](event-hubs-node-get-started-send.md), and [Go](event-hubs-go-get-started-send.md) client libraries for publishing events to an event hub. For other runtimes and platforms, you can use any AMQP 1.0 client, such as [Apache Qpid](https://qpid.apache.org/). 
 
-The choice to use AMQP or HTTPS is specific to the usage scenario. AMQP requires the establishment of a persistent bidirectional socket in addition to transport level security (TLS) or SSL/TLS. AMQP has higher network costs when initializing the session, however HTTPS requires additional TLS overhead for every request. AMQP has significantly higher performance for frequent publishers and can achieve much lower latencies when used with asynchronous publishing code.
+The choice to use AMQP or HTTPS is specific to the usage scenario. AMQP requires the establishment of a persistent bidirectional socket in addition to transport level security (TLS) or SSL/TLS. AMQP has higher network costs when initializing the session, however HTTPS requires extra TLS overhead for every request. AMQP has higher performance for frequent publishers and can achieve much lower latencies when used with asynchronous publishing code.
 
 You can publish events individually or batched. A single publication has a limit of 1 MB, regardless of whether it's a single event or a batch. Publishing events larger than this threshold will be rejected. 
 
@@ -163,6 +163,14 @@ If a reader disconnects from a partition, when it reconnects it begins reading a
 > - [JavaScript](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/eventhub/eventhubs-checkpointstore-blob/samples/v1/javascript) or  [TypeScript](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/eventhub/eventhubs-checkpointstore-blob/samples/v1/typescript)
 > - [Python](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/eventhub/azure-eventhub-checkpointstoreblob-aio/samples/)
 
+
+### Log compaction 
+
+Azure Event Hubs supports compacting event log to retain the latest events of a given event key. With compacted event hubs/Kafka topic, you can use key-baesd retention rather than using the coarser-grained time-based retention. 
+
+For more information on log compaction, see [Log compaction](log-compaction.md). 
+
+
 ### Common consumer tasks
 
 All Event Hubs consumers connect via an AMQP 1.0 session, a state-aware bidirectional communication channel. Each partition has an AMQP 1.0 session that facilitates the transport of events segregated by partition.

articles/event-hubs/event-hubs-for-kafka-ecosystem-overview.md

@@ -120,10 +120,6 @@ This feature is fundamentally at odds with Azure Event Hubs' multi-protocol mode
 
 The payload of any Event Hubs event is a byte stream and the content can be compressed with an algorithm of your choosing. The Apache Avro encoding format supports compression natively.
 
-### Log Compaction
-
-Apache Kafka log compaction is a feature that allows evicting all but the last record of each key from a partition, which effectively turns an Apache Kafka topic into a key-value store where the last value added overrides the previous one. This feature is presently not implemented by Azure Event Hubs. The key-value store pattern, even with frequent updates, is far better supported by database services like [Azure Cosmos DB](../cosmos-db/introduction.md). For more information, see [Log Projection](event-hubs-federation-overview.md#log-projections). 
-
 ### Kafka Streams
 
 Kafka Streams is a client library for stream analytics that is part of the Apache Kafka open-source project, but is separate from the Apache Kafka event stream broker. 

articles/event-hubs/log-compaction.md

@@ -0,0 +1,49 @@
+---
+title: Log compaction
+description: This article describes how the log compaction feature works in Event Hubs.
+ms.topic: article
+ms.date: 10/7/2022
+ms.custom: ignite-2022
+---
+
+# Log compaction
+
+Log compaction is a way of retaining data in Event Hubs using event key based retention. By default, each event hub/Kafka topic is created with time-based retention or *delete* cleanup policy, where events are purged upon the expiration of the retention time. Rather using coarser-grained time based retention, you can use event key-based retention mechanism where Event Hubs retrains the last known value for each event key of an event hub or a Kafka topic. 
+
+> [!NOTE] 
+> Log compaction feature is available only in **premium** and **dedicated** tiers. 
+
+As shown below, an event log (of an event hub partition) may have multiple events with the same key. If you're using a compacted event hub, then Event Hubs service will take care of purging old events and only keeping the latest events of a given event key. 
+
+:::image type="content" source="./media/event-hubs-log-compaction/log-compaction.png" alt-text="Diagram showing how a topic gets compacted." lightbox="./media/event-hubs-resource-governance-overview/app-groups.png":::
+
+### Compaction key
+The partition key that you set with each event is used as the compaction key. 
+
+### Tombstones
+Client application can mark existing events of an event hub to be deleted during compaction job. These markers are known as *Tombstones*. Tombstones are set by the client applications by sending a new event with an existing key and a `null` event payload. 
+
+## How log compaction works
+
+You can enable log compaction at each event hub/Kafka topic level. You can ingest events to a compacted article from any support protocol. Azure Event Hubs service runs a compaction job for each compacted event hub. Compaction job cleans each event hub partition log by only retaining the latest event of a given event key. 
+
+:::image type="content" source="./media/event-hubs-log-compaction/how-compaction-work.png" alt-text="Diagram showing how log compaction works." lightbox="./media/event-hubs-log-compaction/how-compaction-work.png":::
+
+At a given time the event log of a compacted event hub can have a *cleaned* portion and *dirty* portion. The clean portion contains the events that are compacted by the compaction job while the dirty portion comprises the events that are yet to be compacted. 
+
+The execution of the compaction job is managed by the Event Hubs service and user can't control it. Therefore, Event Hubs service determines when to start compaction and how fast it compact a given compacted event hub. 
+
+## Compaction guarantees
+Log compaction feature of Event Hubs provides the following guarantee: 
+- Ordering of the messages is always maintained at the key and partition level. Compaction job doesn't alter ordering of messages but it just discards the old events of the same key. 
+- The sequence number and offset of a message never changes. 
+- Any consumer progressing from the start of the event log will see at least the final state of all events in the order they were written. 
+- Events that the user mark to be deleted can still be seen by consumers for time defined by *Tombstone Retention Time(hours)*. 
+
+
+## Log compaction use cases
+Log compaction can be useful in scenarios where you stream the same set of updatable events. As compacted event hubs only keep the latest events, users don't need to worry about the growth of the event storage. Therefore log compaction is commonly used in scenarios such as Change Data Capture(CDC), maintaining event in tables for stream processing applications and event caching. 
+
+
+## Next steps
+For instructions on how to use log compaction in Event Hubs, see [Use log compaction](./use-log-compaction.md)

articles/event-hubs/media/event-hubs-log-compaction/enabling-compaction.png

@@ -0,0 +1,90 @@
+---
+title: Use log compaction
+description: Learn how to use log compaction. 
+ms.topic: how-to
+ms.custom: log-compaction
+ms.date: 10/7/2022
+---
+
+# Use log compaction
+This article shows you how to use log compaction feature in Event Hubs. To understand the details of log compaction, see [Log Compaction](log-compaction.md). 
+
+In this article you'll, follow these key steps:
+- Create a compacted event hub/Kafka topic.
+- Publish events to a compacted event hub. 
+- Consume events from a compacted event hub. 
+
+> [!NOTE] 
+> Log compaction feature is available only in **premium** and **dedicated** tiers. 
+
+## Create a compacted event hub/Kafka topic
+This section shows you how to create a compacted event hub using Azure portal and an Azure Resource Manager (ARM) template. 
+
+### [Azure portal](#tab/portal)
+You can create a compacted event hub using the Azure portal by following these steps. 
+
+1. Navigate to your Event Hubs namespace.
+1. On the Event Hubs Namespace page, select Event Hubs in the left menu.
+1. At the top of the window, select + Event Hubs.
+    :::image type="content" source="./media/event-hubs-quickstart-portal/create-event-hub4.png" alt-text="Screenshot of event hub creation UI.":::
+1. Type a *name* for your event hub, and specify the *partition count*. Since we're creating a compacted event hub, select *compaction policy* as *compaction* and provide the desired value for *tombstone retention time*. 
+    :::image type="content" source="./media/event-hubs-log-compaction/enabling-compaction.png" alt-text="Screenshot of the event hubs creation UI with compaction related attributes.":::
+1. Select *create* and create the compacted event hub. 
+
+### [ARM template](#tab/arm)
+The following example shows how to create a compacted event hub/Kafka topic using an ARM template. 
+
+```json
+"resources": [
+  {
+    "apiVersion": "2017-04-01",
+    "name": "[parameters('eventHubName')]",
+    "type": "eventhubs",
+    "dependsOn": [
+      "[resourceId('Microsoft.EventHub/namespaces/', parameters('eventHubNamespaceName'))]"
+    ],
+    "properties": {
+      "partitionCount": "[parameters('partitionCount')]",
+      "retentionDescription": {
+        "cleanupPolicy": "compact",
+        "tombstoneRetentionTimeInHours": "24"
+      }
+    }
+  }
+]
+```
+
+
+---
+
+## Triggering compaction 
+Event Hubs service determines when the compaction job of a given compacted event hub should be executed. Compacted event hub reaches the compaction threshold when there are considerable number of events or the total size of a given event log grows significantly. 
+
+## Publish event to a compacted topic
+Publishing events to a compacted event hub is the same as publishing events to a regular event hub. As the client application you only need to determine the compaction key, which you set using partition key. 
+
+### Using Event Hubs SDK(AMQP)
+With Event Hubs SDK, you can set partition key and publish events as shown below:
+
+```csharp
+var enqueueOptions = new EnqueueEventOptions
+{
+    PartitionKey = "Key-1"
+    
+};
+await producer.EnqueueEventAsync(eventData, enqueueOptions);
+```
+
+### Using Kafka
+With Kafka you can set the partition key when you create the `ProducerRecord` as shown below: 
+```java
+ProducerRecord<String, String> record = new ProducerRecord<String, String>(TOPIC, "Key-1" , "Value-1");
+```
+
+
+## Consuming events from a compacted topic
+There are no changes required at the consumer side to consume events from a compacted event hub. So, you can use any of the existing consumer applications to consume data from a compacted event hub. 
+
+## Next steps
+
+- For conceptual information on how log compaction work, see [Log compaction](log-compaction.md).