You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/event-grid/event-hubs-integration.md
+34-34Lines changed: 34 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,11 @@
1
1
---
2
2
title: 'Tutorial: Send Event Hubs data to data warehouse - Event Grid'
3
-
description: Shows how to migrate Event Hubs captured data from Azure Blob Storage to Azure Synapse Analytics, specifically a dedicated SQL pool, using Azure Event Grid and Azure Functions.
3
+
description: Learn to migrate Event Hubs captured data from Azure Blob Storage to Azure Synapse Analytics using Azure Event Grid and Azure Functions.
4
4
ms.topic: tutorial
5
-
ms.date: 01/31/2024
5
+
ms.date: 06/26/2025
6
6
ms.devlang: csharp
7
7
ms.custom: devx-track-csharp
8
+
#customer intent: As a developer, I want to learn to migrate data captured by event hubs from storage to analytics to support apps.
8
9
---
9
10
10
11
# Tutorial: Migrate Event Hubs captured data from Azure Storage to Azure Synapse Analytics using Azure Event Grid and Azure Functions
@@ -31,22 +32,21 @@ In this article, you take the following steps:
31
32
> - Verify captured data in Azure Synapse Analytics
32
33
33
34
## Prerequisites
34
-
To complete this tutorial, you must have:
35
35
36
-
-This article assumes that you're familiar with Event Grid and Event Hubs (especially the Capture feature). If you aren't familiar with Azure Event Grid, see [Introduction to Azure Event Grid](overview.md). To learn about the Capture feature of Azure Event Hubs, see [Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage](../event-hubs/event-hubs-capture-overview.md).
36
+
-Knowledge of Azure Event Grid and Azure Event Hubs, especially the Capture feature. If you aren't familiar with Azure Event Grid, see [Introduction to Azure Event Grid](overview.md). To learn about the Capture feature of Azure Event Hubs, see [Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage](../event-hubs/event-hubs-capture-overview.md).
37
37
- An Azure subscription. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/) before you begin.
38
-
-[Visual Studio](https://www.visualstudio.com/vs/) with workloads for: .NET desktop development, Azure development, ASP.NET and web development, Node.js development, and Python development.
39
-
-Download the [EventHubsCaptureEventGridDemo sample project](https://github.com/Azure/azure-event-hubs/tree/master/samples/e2e/EventHubsCaptureEventGridDemo) to your computer.
38
+
-[Visual Studio](https://www.visualstudio.com/vs/) with workloads for .NET desktop development, Azure development, ASP.NET and web development, Node.js development, and Python development.
39
+
-The [EventHubsCaptureEventGridDemo sample project](https://github.com/Azure/azure-event-hubs/tree/master/samples/e2e/EventHubsCaptureEventGridDemo). The sample includes:
40
40
41
-
- WindTurbineDataGenerator – A simple publisher that sends sample wind turbine data to an event hub with the Capture feature enabled.
42
-
- FunctionDWDumper – An Azure function that receives a notification from Azure Event Grid when an Avro file is captured to the Azure Storage blob. It receives the blob’s URI path, reads its contents, and pushes this data to Azure Synapse Analytics (dedicated SQL pool).
41
+
- WindTurbineDataGenerator. A simple publisher that sends sample wind turbine data to an event hub with the Capture feature enabled.
42
+
- FunctionDWDumper. An Azure function that receives a notification from Azure Event Grid when an Avro file is captured to the Azure Storage blob. It receives the blob URI, reads its contents, and pushes this data to Azure Synapse Analytics.
43
43
44
44
## Deploy the infrastructure
45
45
46
-
In this step, you deploy the required infrastructure with a [Resource Manager template](https://github.com/Azure/azure-docs-json-samples/blob/master/event-grid/EventHubsDataMigration.json). When you deploy the template, the following resources are created:
46
+
In this section, deploy the required infrastructure with a [Resource Manager template](https://github.com/Azure/azure-docs-json-samples/blob/master/event-grid/EventHubsDataMigration.json). When you deploy the template, the following resources are created:
47
47
48
-
- Event hub with the Capture feature enabled.
49
-
- Storage account for the captured files.
48
+
- Event hub with the Capture feature enabled
49
+
- Storage account for the captured files
50
50
- App service plan for hosting the function app
51
51
- Function app for processing the event
52
52
- SQL Server for hosting the data warehouse
@@ -69,7 +69,7 @@ In this step, you deploy the required infrastructure with a [Resource Manager te
69
69
70
70
1. Create a storage account by selecting **Create storage**. Azure Cloud Shell requires an Azure storage account to store some files.
71
71
72
-
::image type="content" source="media/event-hubs-functions-synapse-analytics/create-storage-cloud-shell.png" alt-text="Screenshot showing the creation of storage for Cloud Shell.":::
72
+
:::image type="content" source="media/event-hubs-functions-synapse-analytics/create-storage-cloud-shell.png" alt-text="Screenshot showing the creation of storage for Cloud Shell.":::
73
73
74
74
1. Wait until the Cloud Shell is initialized.
75
75
@@ -102,14 +102,14 @@ In this step, you deploy the required infrastructure with a [Resource Manager te
102
102
"tags": null,
103
103
"type": "Microsoft.Resources/resourceGroups"
104
104
}
105
-
```
105
+
```
106
106
107
-
1. Deploy all the resources mentioned in the previous section (event hub, storage account, functions app, Azure Synapse Analytics) by running the following CLI command:
107
+
1. Deploy all the resources mentioned in the previous section by running the following CLI command. The resources include an event hub, storage account, functions app, and Azure Synapse Analytics.
108
108
109
-
1. Copy and paste the command into the Cloud Shell window. Alternatively, you can copy/paste into an editor of your choice, set values, and then copy the command to the Cloud Shell. If you see an error due to an Azure resource name, delete the resource group, fix the name, and retry the command again.
109
+
1. Copy and paste the command into the Cloud Shell window. Alternatively, you can copy into a text editor, set values, and then copy the command to the Cloud Shell. If you see an error due to an Azure resource name, delete the resource group, fix the name, and retry the command.
110
110
111
111
> [!IMPORTANT]
112
-
> Specify values for the following entities before running the command:
112
+
> Specify values for the following entities before you run the command:
113
113
>
114
114
> - Name of the resource group you created earlier.
115
115
> - Name for the event hub namespace.
@@ -128,14 +128,14 @@ In this step, you deploy the required infrastructure with a [Resource Manager te
1. Press **ENTER** in the Cloud Shell window to run the command. This process might take a while since you're creating a bunch of resources. In the result of the command, ensure that there are no failures.
131
+
1. Press **Enter** in the Cloud Shell window to run the command. This process might take a while since you're creating a bunch of resources. In the result of the command, ensure that there are no failures.
132
132
133
-
1. Close the Cloud Shell by selecting the **Cloud Shell** button in the portal (or) **X** button in the top-right corner of the Cloud Shell window.
133
+
1. Close the Cloud Shell by selecting the **Cloud Shell** button in the portal or the **X** button in the top-right corner of the Cloud Shell window.
134
134
135
135
### Verify that the resources are created
136
136
137
137
1. In the Azure portal, in the search box, enter *Resource groups*. From the results, select **Resource groups**.
138
-
1. Filter the list of resource groups by entering the name of your resource group in the search box.
138
+
1. If necessary, in the search box, enter the name of your resource group to filter the resource groups.
139
139
1. Select your resource group in the list.
140
140
141
141
:::image type="content" source="media/event-hubs-functions-synapse-analytics/select-resource-group.png" alt-text="Screenshot showing the selection of your resource group.":::
@@ -148,12 +148,12 @@ In this step, you deploy the required infrastructure with a [Resource Manager te
148
148
149
149
In this section, you create a table in the dedicated SQL pool you created earlier.
150
150
151
-
1. In the list of resources in the resource group, select your **dedicated SQL pool**.
152
-
1. On the **Dedicated SQL pool** page, in the **Common Tasks** section on the left menu, select **Query editor (preview)**.
151
+
1. In the list of resources in the resource group, select your *dedicated SQL pool*.
152
+
1. On the **Dedicated SQL pool** page, under **Common Tasks**, select **Query editor (preview)**.
153
153
154
154
:::image type="content" source="media/event-hubs-functions-synapse-analytics/sql-data-warehouse-page.png" alt-text="Screenshot showing the selection of Query Editor on a Dedicated SQL pool page in the Azure portal.":::
155
155
156
-
1. Enter the name of **user** and **password** for the SQL server, and select **OK**. If you see a message about allowing your client to access the SQL server, select **Allowlist IP <your IP Address> on server <your SQL server>**, and then select **OK**.
156
+
1. Enter the user name and password for the SQL server, and select **OK**. If you see a message about allowing your client to access the SQL server, select **Allowlist IP <your IP Address> on server <your SQL server>**, and then select **OK**.
157
157
1. In the query window, copy and run the following SQL script:
158
158
159
159
```sql
@@ -177,11 +177,11 @@ First, get the publish profile for the Functions app from the Azure portal. Then
177
177
178
178
### Get the publish profile
179
179
180
-
1. On the **Resource Group** page, select the **Azure Functions app** in the list of resources.
180
+
1. On the **Resource Group** page, select the **Azure Functions app**.
181
181
182
182
:::image type="content" source="media/event-hubs-functions-synapse-analytics/select-function-app.png" lightbox="media/event-hubs-functions-synapse-analytics/select-function-app.png" alt-text="Screenshot showing the selection of the function app in the list of resources for a resource group.":::
183
183
184
-
1. On the **Function App** page for your app, select **Get publish profile** on the command bar.
184
+
1. On the **Function App** page for your app, select **Get publish profile**.
185
185
186
186
:::image type="content" source="media/event-hubs-functions-synapse-analytics/get-publish-profile.png" lightbox="media/event-hubs-functions-synapse-analytics/get-publish-profile.png" alt-text="Screenshot showing the selection of the **Get Publish Profile** button on the command bar of the function app page.":::
187
187
@@ -197,7 +197,7 @@ First, get the publish profile for the Functions app from the Azure portal. Then
197
197
198
198
:::image type="content" source="media/event-hubs-functions-synapse-analytics/import-profile.png" alt-text="Screenshot showing the selection **Import Profile** on the **Publish** dialog box.":::
199
199
200
-
1. On the **Import profile** tab, select the publish settings file that you saved earlier in the **FunctionEGDWDumper** folder, and then select **Finish**.
200
+
1. On the **Import profile** tab, select the publish settings file that you saved earlier in the **FunctionEGDWDumper** folder. Then select **Finish**.
201
201
1. When Visual Studio configures the profile, select **Publish**. Confirm that the publishing succeeded.
202
202
1. In the web browser that has the **Azure Function** page open, select **Functions** in the middle pane. Confirm that the **EventGridTriggerMigrateData** function shows up in the list. If you don't see it, try publishing from Visual Studio again, and then refresh the page in the portal.
203
203
@@ -208,18 +208,18 @@ After publishing the function, you're ready to subscribe to the event.
208
208
## Subscribe to the event
209
209
210
210
1. In a new tab or new window of a web browser, sign in to the [Azure portal](https://portal.azure.com).
211
-
1. In the Azure portal, select **Resource groups** on the left menu.
212
-
1.Filter the list of resource groups by entering the name of your resource group in the search box.
211
+
1. In the Azure portal, search for and select **Resource groups**.
212
+
1.If necessary, in the search box, enter the name of your resource group to filter the resource groups.
213
213
1. Select your resource group in the list.
214
214
1. Select the **Event Hubs namespace** from the list of resources.
215
-
1. On the **Event Hubs Namespace** page, select **Events** on the left menu, and then select **+ Event Subscription** on the toolbar.
215
+
1. On the **Event Hubs Namespace** page, select **Events**, and then select **+ Event Subscription** on the toolbar.
216
216
217
217
:::image type="content" source="media/event-hubs-functions-synapse-analytics/event-hub-add-subscription-link.png" alt-text="Screenshot of the Events page for an Event Hubs namespace with Add event subscription link selected. ":::
218
218
219
219
1. On the **Create Event Subscription** page, follow these steps:
220
220
221
221
1. Enter a name for the **event subscription**.
222
-
1. Enter a name for the **system topic**. A system topic provides an endpoint for the sender to send events. For more information, see [System topics](system-topics.md)
222
+
1. Enter a name for the **system topic**. A system topic provides an endpoint for the sender to send events. For more information, see [System topics](system-topics.md).
223
223
1. For **Endpoint Type**, select **Azure Function**.
224
224
1. For **Endpoint**, select the link.
225
225
1. On the **Select Azure Function** page, follow these steps if they aren't automatically filled.
@@ -241,16 +241,16 @@ After publishing the function, you're ready to subscribe to the event.
241
241
242
242
## Run the app to generate data
243
243
244
-
You finished setting up your event hub, dedicate SQL pool (formerly SQL Data Warehouse), Azure function app, and event subscription. Before running an application that generates data for event hub, you need to configure a few values.
244
+
You finished setting up your event hub, dedicate SQL pool (formerly SQL Data Warehouse), Azure function app, and event subscription. Before you run an application that generates data for event hub, configure a few values.
245
245
246
246
1. In the Azure portal, navigate to your resource group as you did earlier.
247
247
1. Select the Event Hubs namespace.
248
-
1. In the **Event Hubs Namespace** page, select **Shared access policies** on the left menu.
248
+
1. In the **Event Hubs Namespace** page, select **Shared access policies**.
249
249
1. Select **RootManageSharedAccessKey** in the list of policies.
250
250
251
251
:::image type="content" source="media/event-hubs-functions-synapse-analytics/event-hub-namespace-shared-access-policies.png" alt-text="Screenshot showing the Shared access policies page for an Event Hubs namespace.":::
252
252
253
-
1. Select the copy button next to the **Connection string-primary key** text box.
253
+
1. Select the copy button next to **Primary connection string**.
254
254
1. Go back to your Visual Studio solution.
255
255
1. Right-click **WindTurbineDataGenerator** project, and select **Set as Startup project**.
256
256
1. In the WindTurbineDataGenerator project, open **program.cs**.
@@ -272,7 +272,7 @@ You finished setting up your event hub, dedicate SQL pool (formerly SQL Data War
272
272
:::image type="content" source="media/event-hubs-functions-synapse-analytics/query-results.png" alt-text="Screenshot showing the query results.":::
273
273
274
274
> [!IMPORTANT]
275
-
> We use connection string to authenticate to Azure Event Hubs namespace to keep the tutorial simple. We recommend that you use Microsoft Entra ID authentication in production environments. When using an application, you can enable managed identity for the application and assign the identity an appropriate role (Azure Event Hubs Owner, Azure Event Hubs Data Sender, or Azure Event Hubs Data Receiver) on the Event Hubs namespace. For more information, see [Authorize access to Event Hubs using Microsoft Entra ID](../event-hubs/authorize-access-azure-active-directory.md).
275
+
> In this example, using the connection string to authenticate to Azure Event Hubs namespace keeps the tutorial simple. We recommend that you use Microsoft Entra ID authentication in production environments. When you use an application, enable managed identity for the application and assign the identity an appropriate role (Azure Event Hubs Owner, Azure Event Hubs Data Sender, or Azure Event Hubs Data Receiver) on the Event Hubs namespace. For more information, see [Authorize access to Event Hubs using Microsoft Entra ID](../event-hubs/authorize-access-azure-active-directory.md).
276
276
277
277
## Monitor the solution
278
278
@@ -281,7 +281,7 @@ This section helps you with monitoring or troubleshooting the solution.
281
281
### View captured data in the storage account
282
282
283
283
1. Navigate to the resource group and select the storage account used for capturing event data.
284
-
1. On the **Storage account** page, select **Storage browser** on the left menu.
284
+
1. On the **Storage account** page, select **Storage browser**.
285
285
1. Expand **BLOB CONTAINERS**, and select **windturbinecapture**.
286
286
1. Open the folder named same as your **Event Hubs namespace** in the right pane.
287
287
1. Open the folder named same as your event hub (**hubdatamigration**).
0 commit comments