You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/event-hubs/event-hubs-capture-enable-through-portal.md
+61-38Lines changed: 61 additions & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,53 +2,72 @@
2
2
title: Event Hubs - Capture streaming events using Azure portal
3
3
description: This article describes how to enable capturing of events streaming through Azure Event Hubs by using the Azure portal.
4
4
ms.topic: quickstart
5
-
ms.date: 09/16/2021
5
+
ms.date: 10/27/2021
6
6
ms.custom: mode-ui
7
7
---
8
8
9
9
# Enable capturing of events streaming through Azure Event Hubs
10
10
11
-
Azure [Event Hubs Capture][capture-overview] enables you to automatically deliver the streaming data in Event Hubs to an [Azure Blob storage](https://azure.microsoft.com/services/storage/blobs/) or [Azure Data Lake Storage Gen1 or Gen 2](https://azure.microsoft.com/services/data-lake-store/) account of your choice.
12
-
13
-
You can configure Capture at the event hub creation time using the [Azure portal](https://portal.azure.com). You can either capture the data to an Azure [Blob storage](https://azure.microsoft.com/services/storage/blobs/) container, or to an [Azure Data Lake Storage Gen 1 or Gen 2](https://azure.microsoft.com/services/data-lake-store/) account.
14
-
15
-
For more information, see the [Event Hubs Capture overview][capture-overview].
11
+
Azure [Event Hubs Capture][capture-overview] enables you to automatically deliver the streaming data in Event Hubs to an [Azure Blob storage](https://azure.microsoft.com/services/storage/blobs/) or [Azure Data Lake Storage Gen1 or Gen 2](https://azure.microsoft.com/services/data-lake-store/) account of your choice.You can configure capture settings using the [Azure portal](https://portal.azure.com) when creating an event hub or for an existing event hub. For conceptual information on this feature, see [Event Hubs Capture overview][capture-overview].
16
12
17
13
> [!IMPORTANT]
18
14
> - The destination storage (Azure Storage or Azure Data Lake Storage) account must be in the same subscription as the event hub.
19
15
> - Event Hubs doesn't support capturing events in a **premium** storage account.
20
16
21
17
22
-
## Capture data to Azure Storage
18
+
## Enable Capture when you create an event hub
23
19
24
-
When you create an event hub, you can enable Capture by clicking the **On**button in the **Create Event Hub** portal screen. You then specify a Storage Account and container by clicking**Azure Storage**in the **Capture Provider** box. Because Event Hubs Capture uses service-to-service authentication with storage, you do not need to specify a storage connection string. The resource picker selects the resource URI for your storage account automatically. If you use Azure Resource Manager, you must supply this URI explicitly as a string.
20
+
If you don't have an Event Hubs namespace to work with, create a **standard**tier namespace by following steps from the article: [Create an Event Hubs namespace](event-hubs-create.md#create-an-event-hubs-namespace). Make sure that you select**Standard**for the **pricing tier**. The basic tier doesn't support the Capture feature.
25
21
26
-
The default time window is 5 minutes. The minimum value is 1, the maximum 15. The **Size** window has a range of 10-500 MB.
22
+
To create an event hub within the namespace, follow these steps:
27
23
28
-
You can enable or disable emitting empty files when no events occur during the Capture window.
24
+
1. On the **Overview** page for your namespace, select **+ Event hub** on the command bar.
25
+
26
+
:::image type="content" source="./media/event-hubs-quickstart-portal/create-event-hub4.png" lightbox="./media/event-hubs-quickstart-portal/create-event-hub4.png" alt-text="Screenshot of the selection of Add event hub button on the command bar.":::
27
+
2. On the **Create event hub** page, type a name for your event hub, then select **Next: Capture** at the bottom of the page.
28
+
29
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/create-event-hub-basics-page.png" alt-text="Screenshot of the Create event hub page.":::
30
+
1. On the **Capture** tab, select **On** for **Capture**.
31
+
1. Drag the slider to set the **Time window** in minutes. The default time window is 5 minutes. The minimum value is 1 and the maximum is 15.
32
+
1. Drag the slider to set the **Size window (MB)**. The default value is 300 MB. The minimum value is 10 MB and the maximum value is 500 MB.
33
+
1. Specify whether you want Event Hubs to **emit empty files when no events occur during the Capture time window**.
34
+
35
+
See one of the following sections based on the type of storage you want to use to store captured files.
36
+
37
+
## Capture data to Azure Storage
29
38
30
-
![Time window for capture][1]
39
+
1. For **Capture Provider**, select **Azure Storage Account** (default).
40
+
1. For **Azure Storage Container**, click the **Select the container** link.
31
41
42
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/select-container-link.png" alt-text="Screenshot that shows the Create event hub page with the Select container link.":::
43
+
1. On the **Storage accounts** page, select the storage account that you want to use to capture data.
44
+
1. On the **Containers** page, select the container where you want to store captured files, and then click **Select**.
32
45
46
+
Because Event Hubs Capture uses service-to-service authentication with storage, you don't need to specify a storage connection string. The resource picker selects the resource URI for your storage account automatically. If you use Azure Resource Manager, you must supply this URI explicitly as a string.
47
+
1. Now, on the **Create event hub** page, confirm that the selected container shows up.
48
+
1. For **Capture file name format**, specify format for the captured file names.
49
+
1. Select **Review + create** at the bottom of the page.
50
+
1. On the **Review + create** page, review settings, and select **Create** to create the event hub.
51
+
33
52
## Capture data to Azure Data Lake Storage Gen 2
34
53
35
-
1. Follow [Create a storage account](../storage/common/storage-account-create.md?tabs=azure-portal#create-a-storage-account) article to create an Azure Storage account. Set **Hierarchical namespace** to **Enabled** on the **Advanced** tab to make it an Azure Data Lake Storage Gen 2 account. The Azure Storage account must be in the same subscription as the event hub.
36
-
2. When creating an event hub, do the following steps:
54
+
Follow [Create a storage account](../storage/common/storage-account-create.md?tabs=azure-portal#create-a-storage-account) article to create an Azure Storage account. Set **Hierarchical namespace** to **Enabled** on the **Advanced** tab to make it an Azure Data Lake Storage Gen 2 account. The Azure Storage account must be in the same subscription as the event hub.
37
55
38
-
1. Select **On** for **Capture**.
39
-
2. Select **Azure Storage** as the capture provider. The **Azure Data Lake Store** option you see for the **Capture provider** is for the Gen 1 of Azure Data Lake Storage. To use a Gen 2 of Azure Data Lake Storage, you select **Azure Storage**.
40
-
2. Select the **Select Container** button.
56
+
1. Select **Azure Storage** as the capture provider. The **Azure Data Lake Store** option you see for the **Capture provider** is for the Gen 1 of Azure Data Lake Storage. To use a Gen 2 of Azure Data Lake Storage, you select **Azure Storage**.
57
+
2. For **Azure Storage Container**, click the **Select the container** link.
41
58
42
-

59
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/select-container-link.png" alt-text="Screenshot that shows the Create event hub page with the Select container link.":::
43
60
3. Select the **Azure Data Lake Storage Gen 2** account from the list.
44
61
45
-

46
-
4. Select the **container** (file system in Data Lake Storage Gen 2).
62
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/select-data-lake-storage-gen2.png" alt-text="Screenshot showing the selection of Data Lake Storage Gen 2 account.":::
63
+
4. Select the **container** (file system in Data Lake Storage Gen 2), and then click **Select** at the bottom of the page.
47
64
48
-

49
-
5. On the **Create Event Hub** page, select **Create**.
65
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/select-file-system-data-lake-storage.png" alt-text="Screenshot that shows the Containers page.":::
66
+
1. For **Capture file name format**, specify format for the captured file names.
67
+
1. Select **Review + create** at the bottom of the page.
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/create-event-hub-data-lake-storage.png" alt-text="Screenshot that shows the Create event hub page with all the fields specified.":::
70
+
1. On the **Review + create** page, review settings, and select **Create** to create the event hub.
52
71
53
72
> [!NOTE]
54
73
> The container you create in a Azure Data Lake Storage Gen 2 using this user interface (UI) is shown under **File systems** in **Storage Explorer**. Similarly, the file system you create in a Data Lake Storage Gen 2 account shows up as a container in this UI.
@@ -58,6 +77,9 @@ You can enable or disable emitting empty files when no events occur during the C
58
77
59
78
To capture data to Azure Data Lake Storage Gen 1, you create a Data Lake Storage Gen 1 account, and an event hub:
60
79
80
+
> [!IMPORTANT]
81
+
> On Feb 29, 2024 Azure Data Lake Storage Gen1 will be retired. For more information, see the [official announcement](https://azure.microsoft.com/updates/action-required-switch-to-azure-data-lake-storage-gen2-by-29-february-2024/). If you use Azure Data Lake Storage Gen1, make sure to migrate to Azure Data Lake Storage Gen2 prior to that date. For more information, see [Azure Data Lake Storage migration guidelines and patterns](../storage/blobs/data-lake-storage-migrate-gen1-to-gen2.md).
82
+
61
83
### Create an Azure Data Lake Storage Gen 1 account and folders
62
84
63
85
1. Create a Data Lake Storage account, following the instructions in [Get started with Azure Data Lake Storage Gen 1 using the Azure portal](../data-lake-store/data-lake-store-get-started-portal.md).
@@ -67,31 +89,32 @@ To capture data to Azure Data Lake Storage Gen 1, you create a Data Lake Storage
67
89
### Create an event hub
68
90
69
91
1. The event hub must be in the same Azure subscription as the Azure Data Lake Storage Gen 1 account you created. Create the event hub, clicking the **On** button under **Capture** in the **Create Event Hub** portal page.
70
-
2.In the **Create Event Hub** portal page, select **Azure Data Lake Store** from the **Capture Provider** box.
92
+
2.On the **Create Event Hub** page, select **Azure Data Lake Store** from the **Capture Provider** box.
71
93
3. In **Select Store** next to the **Data Lake Store** drop-down list, specify the Data Lake Storage Gen 1 account you created previously, and in the **Data Lake Path** field, enter the path to the data folder you created.
72
94
73
-
![Select Data Lake Storage account][3]
74
-
75
-
76
-
## Add or configure Capture on an existing event hub
77
-
78
-
You can configure Capture on existing event hubs that are in Event Hubs namespaces. To enable Capture on an existing event hub, or to change your Capture settings, click the namespace to load the overview screen, then click the event hub for which you want to enable or change the Capture setting. Finally, click the **Capture** option on the left side of the open page and then edit the settings, as shown in the following figures:
95
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/event-hubs-capture3.png" alt-text="Screenshot showing the selection of Data Lake Storage Account Gen 1.":::
79
96
80
-
### Azure Blob Storage
81
97
82
-
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/event-hubs-capture2.png" alt-text="Configure capture to an Azure storage for an existing event hub.":::
98
+
## Configure Capture for an existing event hub
83
99
84
-
### Azure Data Lake Storage Gen 2
85
-
It's same as above (for Azure Blob Storage) except that you will be selecting a container from an Azure Data Lake Storage Gen 2 account.
100
+
You can configure Capture on existing event hubs that are in Event Hubs namespaces. To enable Capture on an existing event hub, or to change your Capture settings, follow these steps:
86
101
87
-
### Azure Data Lake Storage Gen 1
102
+
1. On the home page for your namespace, select **Event Hubs** under **Entities** on the left menu.
103
+
1. Select the event hub for which you want to configure the Capture feature.
88
104
89
-
![Configure Azure Data Lake Storage][4]
105
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/select-event-hub.png" alt-text="Screenshot showing the selection of an event hub in the list of event hubs.":::
106
+
1. On the **Event Hubs Instance** page, select **Capture** on the left menu.
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/view-capture-page.png" alt-text="Screenshot showing the Capture page for your event hub.":::
109
+
1. On the **Capture** page, select **Avro** for **Output event serialization format**. The **Parquet** format is supported only via Azure Stream Analytics integration. For more information, see [Capture Event Hubs data in parquet format and analyze with Azure Synapse Analytics](../stream-analytics/event-hubs-parquet-capture-tutorial.md).
110
+
1. Select **On** for **Capture**.
94
111
112
+
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/enable-capture.png" alt-text="Screenshot showing the Capture page for your event hub with the Capture feature enabled.":::
113
+
1. To configure other settings, see the sections:
114
+
-[Capture data to Azure Storage](#capture-data-to-azure-storage)
115
+
-[Capture data to Azure Data Lake Storage Gen 2](#capture-data-to-azure-data-lake-storage-gen-2)
116
+
-[Capture data to Azure Data Lake Storage Gen 1](#capture-data-to-azure-data-lake-storage-gen-1)
117
+
95
118
## Next steps
96
119
97
120
- Learn more about Event Hubs capture by reading the [Event Hubs Capture overview][capture-overview].
0 commit comments