You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
>*[Store your connection string in an Azure Key Vault](/key-vault/secrets/quick-create-net.md#retrieve-a-secret)
62
-
63
-
Microsoft recommends that you use the most secure authentication flow available. The authentication flow described in this procedure requires a very high degree of trust in the application, and carries risks that aren't present in other flows. You should only use this flow when other more secure flows, such as managed identities, aren't viable.
61
+
>*[Store your connection string in an Azure Key Vault](/key-vault/secrets/quick-create-portal) and use [this method](/key-vault/secrets/quick-create-net.md#retrieve-a-secret) to retrieve it in your code.
Copy file name to clipboardExpand all lines: articles/synapse-analytics/data-explorer/ingest-data/data-explorer-ingest-event-hub-portal.md
+47-39Lines changed: 47 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
-
title: Ingest data from Event Hub into Azure Synapse Data Explorer (Preview)
3
-
description: Learn how to ingest (load) data into Azure Synapse Data Explorer from Event Hub.
2
+
title: Ingest data from Event Hubs into Azure Synapse Data Explorer (Preview)
3
+
description: Learn how to ingest (load) data into Azure Synapse Data Explorer from Event Hubs.
4
4
ms.topic: how-to
5
5
ms.date: 11/02/2021
6
6
author: shsagir
@@ -12,7 +12,7 @@ ms.subservice: data-explorer
12
12
# Customer intent: As a database administrator, I want to ingest data into Azure Synapse Data Explorer from an Event Hub, so I can analyze streaming data.
13
13
---
14
14
15
-
# Ingest data from Event Hub into Azure Synapse Data Explorer
15
+
# Ingest data from Event Hubs into Azure Synapse Data Explorer
Azure Synapse Data Explorer offers ingestion (data loading) from Event Hubs, a big data streaming platform and event ingestion service. [Event Hubs](../../../event-hubs/event-hubs-about.md) can process millions of events per second in near real time. In this article, you create an Event Hub, connect to it from Azure Synapse Data Explorer and see data flow through the system.
26
+
Azure Synapse Data Explorer offers ingestion (data loading) from Event Hubs, a big data streaming platform and event ingestion service. [Event Hubs](../../../event-hubs/event-hubs-about.md) can process millions of events per second in near real time. In this article, you create an Event Hubs, connect to it from Azure Synapse Data Explorer and see data flow through the system.
27
27
28
28
## Prerequisites
29
29
@@ -50,24 +50,24 @@ Azure Synapse Data Explorer offers ingestion (data loading) from Event Hubs, a b
50
50
```
51
51
52
52
- We recommend using a [user assigned managed identity](../../../active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm.md#user-assigned-managed-identity) or [system assigned managed identity](../../../active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm.md#system-assigned-managed-identity) for the data connection (optional).
53
-
- [A sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) that generates data and sends it to an Event Hub. Download the sample app to your system.
53
+
- [A sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) that generates data and sends it to an Event Hubs. Download the sample app to your system.
54
54
- [Visual Studio 2019](https://visualstudio.microsoft.com/vs/) to run the sample app.
55
55
56
56
## Sign in to the Azure portal
57
57
58
58
Sign in to the [Azure portal](https://portal.azure.com/).
59
59
60
-
## Create an Event Hub
60
+
## Create an Event Hubs
61
61
62
-
Create an Event Hub by using an Azure Resource Manager template in the Azure portal.
62
+
Create an Event Hubs by using an Azure Resource Manager template in the Azure portal.
63
63
64
-
1. To create an Event Hub, use the following button to start the deployment. Right-click and select **Open in new window**, so you can follow the rest of the steps in this article.
64
+
1. To create an Event Hubs, use the following button to start the deployment. Right-click and select **Open in new window**, so you can follow the rest of the steps in this article.
65
65
66
66
:::image type="content" source="~/reusable-content/ce-skilling/azure/media/template-deployments/deploy-to-azure-button.svg" alt-text="Button to deploy the Resource Manager template to Azure." border="false" link="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.eventhub%2Fevent-hubs-create-event-hub-and-consumer-group%2Fazuredeploy.json":::
67
67
68
68
The **Deploy to Azure** button takes you to the Azure portal.
69
69
70
-
1. Select the subscription where you want to create the Event Hub, and create a resource group named *test-hub-rg*.
70
+
1. Select the subscription where you want to create the Event Hubs, and create a resource group named *test-hub-rg*.
71
71
72
72
73
73
@@ -77,41 +77,41 @@ Create an Event Hub by using an Azure Resource Manager template in the Azure por
| Subscription | Your subscription | Select the Azure subscription that you want to use for your Event Hub.|
80
+
| Subscription | Your subscription | Select the Azure subscription that you want to use for your Event Hubs.|
81
81
| Resource group | *test-hub-rg* | Create a new resource group. |
82
-
| Location | *West US* | Select *West US* for this article. For a production system, select the region that best meets your needs. Create the Event Hub namespace in the same Location as the Azure Synapse Data Explorer cluster for best performance (most important for Event Hub namespaces with high throughput).
82
+
| Location | *West US* | Select *West US* for this article. For a production system, select the region that best meets your needs. Create the Event Hubs namespace in the same Location as the Azure Synapse Data Explorer cluster for best performance (most important for Event Hubs namespaces with high throughput).
83
83
| Namespace name | A unique namespace name | Choose a unique name that identifies your namespace. For example, *mytestnamespace*. The domain name *servicebus.windows.net* is appended to the name you provide. The name can contain only letters, numbers, and hyphens. The name must start with a letter, and it must end with a letter or number. The value must be between 6 and 50 characters long.
84
-
| Event Hub name | *test-hub* | The Event Hub sits under the namespace, which provides a unique scoping container. The Event Hub name must be unique within the namespace. |
84
+
| Event Hubs name | *test-hub* | The Event Hubs sits under the namespace, which provides a unique scoping container. The Event Hubs name must be unique within the namespace. |
85
85
| Consumer group name | *test-group* | Consumer groups enable multiple consuming applications to each have a separate view of the event stream. |
86
86
| | |
87
87
88
88
1. Select **Review + create**.
89
89
90
90
1. Review the **Summary** of resources created. Select **Create**, which acknowledges that you're creating resources in your subscription.
91
91
92
-
:::image type="content" source="../media/ingest-data-event-hub/review-create.png" alt-text="Screen shot of Azure portal for reviewing and creating Event Hub namespace, Event Hub, and consumer group.":::
92
+
:::image type="content" source="../media/ingest-data-event-hub/review-create.png" alt-text="Screen shot of Azure portal for reviewing and creating Event Hubs namespace, Event Hubs, and consumer group.":::
93
93
94
94
1. Select **Notifications** on the toolbar to monitor the provisioning process. It might take several minutes for the deployment to succeed, but you can move on to the next step now.
Depending on the type of identity, you are using to authenticate with the Event Hub, you may need some additional configurations.
100
+
Depending on the type of identity, you're using to authenticate with the Event Hubs, you might need some other configurations.
101
101
102
-
- If you are authenticating with Event Hub using a user assigned managed identity, go to your Event Hub > **Networking**, and then under **Allow access from**, select **All networks** and save the changes.
102
+
- If you're authenticating with Event Hubs using a user assigned managed identity, go to your Event Hubs > **Networking**, and then under **Allow access from**, select **All networks** and save the changes.
103
103
104
-
:::image type="content" source="../media/ingest-data-event-hub/configure-event-hub-all-networks.png" alt-text="Screenshot of the Event Hub networking page, showing the selection of allowing access to all networks.":::
104
+
:::image type="content" source="../media/ingest-data-event-hub/configure-event-hub-all-networks.png" alt-text="Screenshot of the Event Hubs networking page, showing the selection of allowing access to all networks.":::
105
105
106
-
- If you are authenticating with the Event Hub using a system assigned managed identity, go to your Event Hub > **Networking**, and then either allow access from all networks or under **Allow access from**, select **Selected networks**, select **Allow trusted Microsoft services to bypass this firewall** and save the changes.
106
+
- If you're authenticating with the Event Hubs using a system assigned managed identity, go to your Event Hubs > **Networking**, and then either allow access from all networks or under **Allow access from**, select **Selected networks**, select **Allow trusted Microsoft services to bypass this firewall** and save the changes.
107
107
108
-
:::image type="content" source="../media/ingest-data-event-hub/configure-event-hub-trusted-services.png" alt-text="Screenshot of the Event Hub networking page, showing the selection of allowing access to trusted services.":::
108
+
:::image type="content" source="../media/ingest-data-event-hub/configure-event-hub-trusted-services.png" alt-text="Screenshot of the Event Hubs networking page, showing the selection of allowing access to trusted services.":::
109
109
110
-
## Connect to the Event Hub
110
+
## Connect to the Event Hubs
111
111
112
-
Now you connect to the Event Hub from Data Explorer pool. When this connection is in place, data that flows into the Event Hub streams to the test table you created earlier in this article.
112
+
Now you connect to the Event Hubs from Data Explorer pool. When this connection is in place, data that flows into the Event Hubs streams to the test table you created earlier in this article.
113
113
114
-
1. Select **Notifications** on the toolbar to verify that the Event Hub deployment was successful.
114
+
1. Select **Notifications** on the toolbar to verify that the Event Hubs deployment was successful.
115
115
116
116
1. Under the Data Explorer pool you created, select **Databases** > **TestDatabase**.
117
117
@@ -125,27 +125,27 @@ Now you connect to the Event Hub from Data Explorer pool. When this connection i
125
125
126
126
Fill out the form with the following information, and then select **Create**.
| Data connection name | *test-hub-connection* | The name of the connection you want to create in Azure Synapse Data Explorer.|
133
-
| Subscription | | The subscription ID where the Event Hub resource is located. This field is autopopulated. |
134
-
| Event Hub namespace | A unique namespace name | The name you chose earlier that identifies your namespace. |
135
-
| Event Hub | *test-hub* | The Event Hub you created. |
136
-
| Consumer group | *test-group* | The consumer group defined in the Event Hub you created. |
137
-
| Event system properties | Select relevant properties | The [Event Hub system properties](../../../service-bus-messaging/service-bus-amqp-protocol-guide.md#message-annotations). If there are multiple records per event message, the system properties will be added to the first record. When adding system properties, [create](/azure/data-explorer/kusto/management/create-table-command?context=/azure/synapse-analytics/context/context) or [update](/azure/data-explorer/kusto/management/alter-table-command?context=/azure/synapse-analytics/context/context) table schema and [mapping](/azure/data-explorer/kusto/management/mappings?context=/azure/synapse-analytics/context/context) to include the selected properties. |
138
-
| Compression | *None* | The compression type of the Event Hub messages payload. Supported compression types: *None, Gzip*.|
139
-
| Managed Identity | System-assigned | The managed identity used by the Data Explorer cluster for access to read from the Event Hub.<br /><br />**Note**:<br />When the data connection is created:<br/>\- *System-assigned* identities are automatically created if they don't exist<br />\- The managed identity is automatically assigned the *Azure Event Hubs Data Receiver* role and is added to your Data Explorer cluster. We recommend verifying that the role was assigned and that the identity was added to the cluster. |
133
+
| Subscription | | The subscription ID where the Event Hubs resource is located. This field is autopopulated. |
134
+
| Event Hubs namespace | A unique namespace name | The name you chose earlier that identifies your namespace. |
135
+
| Event Hubs | *test-hub* | The Event Hubs you created. |
136
+
| Consumer group | *test-group* | The consumer group defined in the Event Hubs you created. |
137
+
| Event system properties | Select relevant properties | The [Event Hubs system properties](../../../service-bus-messaging/service-bus-amqp-protocol-guide.md#message-annotations). If there are multiple records per event message, the system properties will be added to the first record. When adding system properties, [create](/azure/data-explorer/kusto/management/create-table-command?context=/azure/synapse-analytics/context/context) or [update](/azure/data-explorer/kusto/management/alter-table-command?context=/azure/synapse-analytics/context/context) table schema and [mapping](/azure/data-explorer/kusto/management/mappings?context=/azure/synapse-analytics/context/context) to include the selected properties. |
138
+
| Compression | *None* | The compression type of the Event Hubs messages payload. Supported compression types: *None, Gzip*.|
139
+
| Managed Identity | System-assigned | The managed identity used by the Data Explorer cluster for access to read from the Event Hubs.<br /><br />**Note**:<br />When the data connection is created:<br/>\- *System-assigned* identities are automatically created if they don't exist<br />\- The managed identity is automatically assigned the *Azure Event Hubs Data Receiver* role and is added to your Data Explorer cluster. We recommend verifying that the role was assigned and that the identity was added to the cluster. |
140
140
141
141
#### Target table
142
142
143
143
There are two options for routing the ingested data: *static* and *dynamic*.
144
-
For this article, you use static routing, where you specify the table name, data format, and mapping as default values. If the Event Hub message includes data routing information, this routing information will override the default settings.
144
+
For this article, you use static routing, where you specify the table name, data format, and mapping as default values. If the Event Hubs message includes data routing information, this routing information will override the default settings.
145
145
146
146
1. Fill out the following routing settings:
147
147
148
-
:::image type="content" source="../media/ingest-data-event-hub/default-routing-settings.png" alt-text="Default routing settings for ingesting data to Event Hub - Azure Synapse Data Explorer.":::
148
+
:::image type="content" source="../media/ingest-data-event-hub/default-routing-settings.png" alt-text="Default routing settings for ingesting data to Event Hubs - Azure Synapse Data Explorer.":::
@@ -168,9 +168,9 @@ If you selected **Event system properties** in the **Data Source** section of th
168
168
169
169
## Copy the connection string
170
170
171
-
When you run the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) listed in Prerequisites, you need the connection string for the Event Hub namespace.
171
+
When you run the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) listed in Prerequisites, you need the connection string for the Event Hubs namespace.
172
172
173
-
1. Under the Event Hub namespace you created, select **Shared access policies**, then **RootManageSharedAccessKey**.
173
+
1. Under the Event Hubs namespace you created, select **Shared access policies**, then **RootManageSharedAccessKey**.
@@ -182,23 +182,31 @@ When you run the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet
182
182
183
183
Use the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) you downloaded to generate data.
184
184
185
+
>[!WARNING]
186
+
>This sample uses connection string authentication to connect to Event Hubs for simplicity of the example. However, hard-coding a connection string into your script requires a very high degree of trust in the application, and carries security risks.
187
+
>
188
+
>For long-term, secure solutions, use one of these options:
>* [Store your connection string in an Azure Key Vault](/key-vault/secrets/quick-create-portal) and use [this method](/key-vault/secrets/quick-create-net.md#retrieve-a-secret) to retrieve it in your code.
192
+
185
193
1. Open the sample app solution in Visual Studio.
186
-
1. In the *program.cs* file, update the `eventHubName` constant to the name of your Event Hub and update the `connectionString` constant to the connection string you copied from the Event Hub namespace.
194
+
1. In the *program.cs* file, update the `eventHubName` constant to the name of your Event Hubs and update the `connectionString` constant to the connection string you copied from the Event Hubs namespace.
187
195
188
196
```csharp
189
197
const string eventHubName = "test-hub";
190
198
// Copy the connection string ("Connection string-primary key") from your Event Hub namespace.
0 commit comments