Skip to content

Commit 486de21

Browse files
committed
updated
1 parent 313a8b1 commit 486de21

File tree

1 file changed

+17
-15
lines changed

1 file changed

+17
-15
lines changed

articles/event-hubs/store-captured-data-data-warehouse.md

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,18 @@
11
# Process and migrate captured Event Hubs data to a SQL Data Warehouse using Event Grid and Azure Functions
22

3-
Event Hubs [Capture](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview) is the easiest way to automatically deliver streamed data in Event Hubs to an Azure Blob storage or Azure Data Lake store. You can subsequently process and deliver the data to any other storage destinations of your choice, such as SQL Data Warehouse or Cosmos DB. In this tutorial, you learn how you to capture data from your event hub into a SQL Database Warehouse by using an [Event Grid](https://docs.microsoft.com/azure/event-grid/overview) triggered Azure Function.
3+
Event Hubs [Capture](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview) is the easiest way to automatically deliver streamed data in Event Hubs to an Azure Blob storage or Azure Data Lake store. You can subsequently process and deliver the data to any other storage destinations of your choice, such as SQL Data Warehouse or Cosmos DB. In this tutorial, you learn how you to capture data from your event hub into a SQL data warehouse by using an Azure function triggered by an [event grid](https://docs.microsoft.com/azure/event-grid/overview).
44

55
![Visual Studio](./media/store-captured-data-data-warehouse/EventGridIntegrationOverview.PNG)
66

7-
* First, you create an event hub with the Capture feature enabled and set an Azure blob storage as the destination. Data generated by WindTurbineGenerator is streamed into the event hub and is automatically captured into Azure Storage as Avro files.
7+
* First, you create an event hub with the **Capture** feature enabled and set an Azure blob storage as the destination. Data generated by WindTurbineGenerator is streamed into the event hub and is automatically captured into Azure Storage as Avro files.
88
* Next, you create an Azure Event Grid subscription with the Event Hubs namespace as its source and the Azure Function endpoint as its destination.
9-
* Whenever a new Avro file is delivered to the Azure Storage blob by Event Hubs Capture, Event Grid notifies the Azure Function with the blob URI. The Function then migrates the data from the Storage blob to a SQL data warehouse.
9+
* Whenever a new Avro file is delivered to the Azure Storage blob by the Event Hubs Capture feature, Event Grid notifies the Azure Function with the blob URI. The Function then migrates data from the blob to a SQL data warehouse.
1010

1111
In this tutorial, you do the following actions:
1212

1313
> [!div class="checklist"]
1414
> * Deploy the infrastructure
15-
> * Publish code to the Functions App
15+
> * Publish code to a Functions App
1616
> * Create an Event Grid subscription from the Functions app
1717
> * Stream sample data into Event Hub.
1818
> * Verify captured data in SQL Data Warehouse
@@ -25,7 +25,7 @@ In this tutorial, you do the following actions:
2525
- *FunctionDWDumper* – An Azure Function that receives an Event Grid notification when an Avro file is captured to the Azure Storage blob. It receives the blob’s URI path, reads its contents, and pushes this data to a SQL Data Warehouse.
2626

2727
### Deploy the infrastructure
28-
Deploy the infrastructure needed for this tutorial by using this [Azure Resource Manager template](https://raw.githubusercontent.com/Azure/azure-docs-json-samples/master/event-grid/EventHubsDataMigration.json). This template creates the following resources:
28+
Use Azure PowerShell or Azure CLI to deploy the infrastructure needed for this tutorial by using this [Azure Resource Manager template](https://raw.githubusercontent.com/Azure/azure-docs-json-samples/master/event-grid/EventHubsDataMigration.json). This template creates the following resources:
2929

3030
- Event Hub with the Capture feature enabled
3131
- Storage account for the captured event data
@@ -36,7 +36,8 @@ Deploy the infrastructure needed for this tutorial by using this [Azure Resource
3636

3737
The following sections provide Azure CLI and Azure PowerShell commands for deploying the infrastructure required for the tutorial. Update names of the following objects before running the commands:
3838

39-
- Azure resource group
39+
- Azure resource group
40+
- Region for the resource group
4041
- Event Hubs namespace
4142
- Event hub
4243
- Azure SQL server
@@ -45,13 +46,13 @@ The following sections provide Azure CLI and Azure PowerShell commands for deplo
4546
- Azure Storage
4647
- Azure Functions App
4748

48-
These scripts take a while to create all the Azure artifcts. Wait until the script completes before proceeding further.
49+
These scripts take some time to create all the Azure artifacts. Wait until the script completes before proceeding further. If the deployment fails for some reason, delete the resource group, fix the reported issue, and rerun the command.
4950

5051
#### Azure CLI
5152
To deploy the template using Azure CLI, use the following commands:
5253

5354
```azurecli-interactive
54-
az group create -l westcentralus -n rgDataMigrationSample
55+
az group create -l westus -n rgDataMigrationSample
5556
5657
az group deployment create \
5758
--resource-group rgDataMigrationSample \
@@ -68,8 +69,9 @@ New-AzureRmResourceGroup -Name rgDataMigration -Location westcentralus
6869
New-AzureRmResourceGroupDeployment -ResourceGroupName rgDataMigration -TemplateUri https://raw.githubusercontent.com/Azure/azure-docs-json-samples/master/event-grid/EventHubsDataMigration.json -eventHubNamespaceName <event-hub-namespace> -eventHubName hubdatamigration -sqlServerName <sql-server-name> -sqlServerUserName <user-name> -sqlServerDatabaseName <database-name> -storageName <unique-storage-name> -functionAppName <app-name>
6970
```
7071

72+
7173
### Create a table in SQL Data Warehouse
72-
Create a table in your SQL data warehouse by running the [CreateDataWarehouseTable.sql](https://github.com/Azure/azure-event-hubs/blob/master/samples/e2e/EventHubsCaptureEventGridDemo/scripts/CreateDataWarehouseTable.sql) script using Visual Studio or the Query Editor in the portal.
74+
Create a table in your SQL data warehouse by running the [CreateDataWarehouseTable.sql](https://github.com/Azure/azure-event-hubs/blob/master/samples/e2e/EventHubsCaptureEventGridDemo/scripts/CreateDataWarehouseTable.sql) script using [Visual Studio](../sql-data-warehouse/sql-data-warehouse-query-visual-studio.md), [SQL Server Management Studio](../sql-data-warehouse/sql-data-warehouse-query-ssms.md), or the Query Editor in the portal.
7375

7476
```sql
7577
CREATE TABLE [dbo].[Fact_WindTurbineMetrics] (
@@ -102,7 +104,7 @@ WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = ROUND_ROBIN);
102104

103105
![Select publish](./media/store-captured-data-data-warehouse/select-publish.png)
104106

105-
After publishing the function, you are ready to subscribe to the Capture event!
107+
After publishing the function, you are ready to subscribe to the capture event from Event Hubs!
106108

107109

108110
## Create an Event Grid subscription from the Functions app
@@ -124,7 +126,7 @@ After publishing the function, you are ready to subscribe to the Capture event!
124126
![Create subscription](./media/store-captured-data-data-warehouse/set-subscription-values.png)
125127

126128
## Generate sample data
127-
You have now set up your Event Hub, SQL data warehouse, Azure Function App, and Event Grid subscription. Upon completing the simple configuration below, you can run WindTurbineDataGenerator.exe to generate data streams to the Event Hub.
129+
You have now set up your Event Hub, SQL data warehouse, Azure Function App, and Event Grid subscription. You can run WindTurbineDataGenerator.exe to generate data streams to the Event Hub after updating connection string and name of your event hub in the source code.
128130

129131
1. In the portal, select your event hub namespace. Select **Connection Strings**.
130132

@@ -140,7 +142,7 @@ You have now set up your Event Hub, SQL data warehouse, Azure Function App, and
140142

141143
4. Go back to your Visual Studio project. In the *WindTurbineDataGenerator* project, open *program.cs*.
142144

143-
5. Replace the two constant values. Use the copied value for **EventHubConnectionString**. Use **hubdatamigration** the event hub name.
145+
5. Update values for **EventHubConnectionString** and **EventHubName** with connection string and name of your event hub.
144146

145147
```cs
146148
private const string EventHubConnectionString = "Endpoint=sb://demomigrationnamespace.servicebus.windows.net/...";
@@ -150,12 +152,12 @@ You have now set up your Event Hub, SQL data warehouse, Azure Function App, and
150152
6. Build the solution, then run the WindTurbineGenerator.exe application.
151153

152154
## Verify captured data in data warehouse
153-
After a couple of minutes, query the table in your data warehouse. You observe that data generated by the WindTurbineDataGenerator has been streamed to your Event Hub, Captured into an Azure Storage container, and then migrated into the SQL data table by Azure Function.
155+
After a couple of minutes, query the table in your SQL data warehouse. You observe that the data generated by WindTurbineDataGenerator has been streamed to your Event Hub, captured into an Azure Storage container, and then migrated into the SQL Data Warehouse table by Azure Function.
154156

155157
## Next steps
156-
You can use powerful data visualization tools with your data warehouse to achieve your Actionable insights.
158+
You can use powerful data visualization tools with your data warehouse to achieve actionable insights.
157159

158160
This article shows how to use [Power BI with SQL Data Warehouse](https://docs.microsoft.com/azure/sql-data-warehouse/sql-data-warehouse-integrate-power-bi)
159161

160-
Now you are all set to plug in the UI you need to get valuable business insights for your management.
162+
161163

0 commit comments

Comments
 (0)