Add log compaction docs

Author: kasun04

articles/event-hubs/log-compaction.md

@@ -0,0 +1,49 @@
+---
+title: Log Compaction
+description: This article describes how log compaction feature works in Event Hubs.
+ms.topic: article
+ms.date: 10/7/2022
+ms.custom: ignite-2022
+---
+
+# Log Compaction
+
+Log compaction is a way of retaining data in Event Hubs using event key based retention. By default, each event hub/Kafka topic is created with time-based retention or deletes* cleanup policy, where events are purged upon the expiration of the retention time. Rather using coarser-grained time based retention, you can use event key-based retention mechanism where Event Hubs retrains the last known value for each event key of an event hub or a Kafka topic. 
+
+> [!NOTE] 
+> Log compaction feature is available only in **premium** and **dedicated** tiers. 
+
+As shown below, an event log (of an event hub partition) may have multiple events with the same key. If you're using a compacted event hub, then Event Hubs service will take care of purging old events and only keeping the latest events of a given event key. 
+
+:::image type="content" source="./media/event-hubs-log-compaction/log-compaction.png" alt-text="Image showing capturing of Event Hubs data into Azure Storage or Azure Data Lake Storage." lightbox="./media/event-hubs-resource-governance-overview/app-groups.png":::
+
+### Compaction key
+The partition key that you set with each event is used as the compaction key. 
+
+### Tombstones
+Client application can mark existing events of an event hub to be deleted during compaction job. These markers are known as *Tombstones*. Tombstones are set by the client applications by sending a new event with an existing key and a `null` event payload. 
+
+## How log compaction works
+
+You can enable log compaction at each event hub/Kafka topic level. You can ingest events to a compacted article from any support protocol. Azure Event Hubs service runs a compaction job for each compacted event hub. Compaction job cleans each event hub partition log by only retaining the latest event of a given event key. 
+
+:::image type="content" source="./media/event-hubs-log-compaction/how-compaction-work.png" alt-text="Image showing capturing of Event Hubs data into Azure Storage or Azure Data Lake Storage." lightbox="./media/event-hubs-log-compaction/how-compaction-work.png":::
+
+At a given time the event log of a compacted event hub can have a *cleaned* portion and *dirty* portion. The clean portion contains the events that are compacted by the compaction job while the dirty portion comprises the events that are yet to be compacted. 
+
+The execution of the compaction job is managed by the Event Hubs service and user can't control it. Therefore, Event Hubs service determines when to start compaction and how fast it compact a given compacted event hub. 
+
+## Compaction guarantees
+Log compaction feature of Event Hubs provides the following guarantee: 
+- Ordering of the messages is always maintained at the key and partition level. Compaction job doesn't alter ordering of messages but it just discards the old events of the same key. 
+- The sequence number and offset of a message never changes. 
+- Any consumer progressing from the start of the event log will see at least the final state of all events in the order they were written. 
+- Events that the user mark to be deleted can still be seen by consumers for time defined by *Tombstone Retention Time(hours)*. 
+
+
+## Log compaction use cases
+Log compaction can be useful in scenarios where you stream the same set of updatable events. As compacted event hubs only keep the latest events, users don't need to worry about the growth of the event storage. Therefore log compaction is commonly used in scenarios such as Change Data Capture(CDC), maintaining event in tables for stream processing applications and event caching. 
+
+
+## Next steps
+For instructions on how to use log compaction in Event Hubs, see [Use log compaction](user-log-compaction.md)

articles/event-hubs/media/event-hubs-log-compaction/enabling-compaction.png

@@ -0,0 +1,90 @@
+---
+title: Use log compaction
+description: Learn how to use log compaction. 
+ms.topic: how-to
+ms.custom: log-compaction
+ms.date: 10/7/2022
+---
+
+# Use log compaction
+This article shows you how to create log compaction feature in Event Hubs. To understand the details of log compaction, see [Log Compaction](log-compaction.md). 
+
+In this article you'll, follow these key steps:
+- Create a compacted event hub/Kafka topic.
+- Publish events to a compacted event hub. 
+- Consume events from a compacted event hub. 
+
+> [!NOTE] 
+> Log compation feature is available only in **premium** and **dedicated** tiers. 
+
+## Create a compacted event hub/Kafka topic
+This section shows you how to create a compacted event hub using Azure portal and an Azure Resource Manager (ARM) template. 
+
+### [Azure portal](#tab/portal)
+You can create a compacted event hub using the Azure portal by following these steps. 
+
+1. Navigate to your Event Hubs namespace.
+1. On the Event Hubs Namespace page, select Event Hubs in the left menu.
+1. At the top of the window, select + Event Hubs.
+    :::image type="content" source="./media/event-hubs-quickstart-portal/create-event-hub4.png" alt-text="Screenshot of the Application Groups page in the Azure portal.":::
+1. Type a *name* for your event hub, and specify the *partition count*. Since we're creating a compacted event hub, select *compaction policy* as *compaction* and provide the desired value for *tombstone retention time*. 
+    :::image type="content" source="./media/event-hubs-log-compaction/enabling-compaction.png" alt-text="Screenshot of the Application Groups page in the Azure portal.":::
+1. Select *create* and create the compacted event hub. 
+
+### [ARM template](#tab/arm)
+The following example shows how to create a compacted event hub/Kafka topic using an ARM template. 
+
+```json
+"resources": [
+  {
+    "apiVersion": "2017-04-01",
+    "name": "[parameters('eventHubName')]",
+    "type": "eventhubs",
+    "dependsOn": [
+      "[resourceId('Microsoft.EventHub/namespaces/', parameters('eventHubNamespaceName'))]"
+    ],
+    "properties": {
+      "partitionCount": "[parameters('partitionCount')]",
+      "retentionDescription": {
+        "cleanupPolicy": "compact",
+        "tombstoneRetentionTimeInHours": "24"
+      }
+    }
+  }
+]
+```
+
+
+---
+
+## Triggering compaction 
+Event Hubs service determines when the compaction job of a given compacted event hub should be executed. Compacted event hub reaches the compaction threshold when there are considerable number of events or the total size of a given event log grows significantly. 
+
+## Publish event to a compacted topic
+Publishing events to a compacted event hub is the same as publishing events to a regular event hub. As the client application you only need to determine the compaction key, which you set using partition key. 
+
+### Using Event Hubs SDK(AMQP)
+With Event Hubs SDK, you can set partition key and publish events as shown below:
+
+```csharp
+var enqueueOptions = new EnqueueEventOptions
+{
+    PartitionKey = "Key-1"
+    
+};
+await producer.EnqueueEventAsync(eventData, enqueueOptions);
+```
+
+### Using Kafka
+With Kafka you can set the partition key when you create the `ProducerRecord` as shown below: 
+```java
+ProducerRecord<String, String> record = new ProducerRecord<String, String>(TOPIC, "Key-1" , "Value-1");
+```
+
+
+## Consuming events from a compacted topic
+There are no changes required at the consumer side to consume events from a compacted event hub. So, you can use any of the existing consumer applications to consume data from a compacted event hub. 
+
+## Next steps
+
+- For conceptual information on how log compaction work, see [Log compaction](log-compaction.md).