You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/data-explorer/ingest-data/data-explorer-ingest-event-hub-overview.md
+32-22Lines changed: 32 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
-
title: Event Hub data connection for Azure Synapse Data Explorer (Preview)
3
-
description: This article provides an overview of how to ingest (load) data into Azure Synapse Data Explorer from Event Hub.
2
+
title: Event Hubs data connection for Azure Synapse Data Explorer (Preview)
3
+
description: This article provides an overview of how to ingest (load) data into Azure Synapse Data Explorer from Event Hubs.
4
4
ms.topic: how-to
5
5
ms.date: 11/02/2021
6
6
author: shsagir
@@ -9,17 +9,17 @@ ms.reviewer: tzgitlin
9
9
ms.service: azure-synapse-analytics
10
10
ms.subservice: data-explorer
11
11
---
12
-
# Event Hub data connection (Preview)
12
+
# Event Hubs data connection (Preview)
13
13
14
14
[Azure Event Hubs](../../../event-hubs/event-hubs-about.md) is a big data streaming platform and event ingestion service. Azure Synapse Data Explorer offers continuous ingestion from customer-managed Event Hubs.
15
15
16
-
The Event Hub ingestion pipeline transfers events to Azure Synapse Data Explorer in several steps. You first create an Event Hub in the Azure portal. You then create a target table in Azure Synapse Data Explorer into which the [data in a particular format](#data-format), will be ingested using the given [ingestion properties](#ingestion-properties). The Event Hub connection needs to know [events routing](#events-routing). Data is embedded with selected properties according to the [event system properties mapping](#event-system-properties-mapping). [Create a connection](#event-hub-connection) to Event Hub to [create an Event Hub](#create-an-event-hub) and [send events](#send-events). This process can be managed through the [Azure portal](data-explorer-ingest-event-hub-portal.md), programmatically with [C#](data-explorer-ingest-event-hub-csharp.md) or [Python](data-explorer-ingest-event-hub-python.md), or with the [Azure Resource Manager template](data-explorer-ingest-event-hub-resource-manager.md).
16
+
The Event Hubs ingestion pipeline transfers events to Azure Synapse Data Explorer in several steps. You first create an Event Hubs in the Azure portal. You then create a target table in Azure Synapse Data Explorer into which the [data in a particular format](#data-format), will be ingested using the given [ingestion properties](#ingestion-properties). The Event Hubs connection needs to know [events routing](#events-routing). Data is embedded with selected properties according to the [event system properties mapping](#event-system-properties-mapping). [Create a connection](#event-hub-connection) to Event Hubs to [create an Event Hubs](#create-an-event-hub) and [send events](#send-events). This process can be managed through the [Azure portal](data-explorer-ingest-event-hub-portal.md), programmatically with [C#](data-explorer-ingest-event-hub-csharp.md) or [Python](data-explorer-ingest-event-hub-python.md), or with the [Azure Resource Manager template](data-explorer-ingest-event-hub-resource-manager.md).
17
17
18
18
For general information about data ingestion in Azure Synapse Data Explorer, see [Azure Synapse Data Explorer data ingestion overview](data-explorer-ingest-data-overview.md).
19
19
20
20
## Data format
21
21
22
-
* Data is read from the Event Hub in form of [EventData](/dotnet/api/microsoft.servicebus.messaging.eventdata) objects.
22
+
* Data is read from the Event Hubs in form of [EventData](/dotnet/api/microsoft.servicebus.messaging.eventdata) objects.
23
23
* See [supported formats](data-explorer-ingest-data-supported-formats.md).
24
24
> [!NOTE]
25
25
> Event Hub doesn't support the .raw format.
@@ -46,12 +46,22 @@ Ingestion properties instruct the ingestion process, where to route the data, an
46
46
47
47
## Events routing
48
48
49
-
When you set up an Event Hub connection to Azure Synapse Data Explorer cluster, you specify target table properties (table name, data format, compression, and mapping). The default routing for your data is also referred to as `static routing`.
49
+
When you set up an Event Hubs connection to Azure Synapse Data Explorer cluster, you specify target table properties (table name, data format, compression, and mapping). The default routing for your data is also referred to as `static routing`.
50
50
You can also specify target table properties for each event, using event properties. The connection will dynamically route the data as specified in the [EventData.Properties](/dotnet/api/microsoft.servicebus.messaging.eventdata.properties#Microsoft_ServiceBus_Messaging_EventData_Properties), overriding the static properties for this event.
51
51
52
-
In the following example, set Event Hub details and send weather metric data to table `WeatherMetrics`.
52
+
In the following example, set Event Hubs details and send weather metric data to table `WeatherMetrics`.
53
53
Data is in `json` format. `mapping1` is pre-defined on the table `WeatherMetrics`.
54
54
55
+
>[!WARNING]
56
+
>This example uses connection string authentication to connect to Event Hubs for simplicity of the example. However, hard-coding a connection string into your script requires a very high degree of trust in the application, and carries security risks.
57
+
>
58
+
>For long-term, secure solutions, use one of these options:
>*[Store your connection string in an Azure Key Vault](/key-vault/secrets/quick-create-net.md#retrieve-a-secret)
62
+
63
+
Microsoft recommends that you use the most secure authentication flow available. The authentication flow described in this procedure requires a very high degree of trust in the application, and carries risks that aren't present in other flows. You should only use this flow when other more secure flows, such as managed identities, aren't viable.
System properties store properties that are set by the Event Hubs service, at the time the event is enqueued. The Azure Synapse Data Explorer Event Hub connection will embed the selected properties into the data landing in your table.
88
+
System properties store properties that are set by the Event Hubs service, at the time, the event is enqueued. The Azure Synapse Data Explorer Event Hubs connection will embed the selected properties into the data landing in your table.
Event Hub exposes the following system properties:
94
+
Event Hubs exposes the following system properties:
85
95
86
96
|Property |Data Type |Description|
87
97
|---|---|---|
88
98
| x-opt-enqueued-time |datetime | UTC time when the event was enqueued |
89
-
| x-opt-sequence-number |long | The logical sequence number of the event within the partition stream of the Event Hub
90
-
| x-opt-offset |string | The offset of the event from the Event Hub partition stream. The offset identifier is unique within a partition of the Event Hub stream |
99
+
| x-opt-sequence-number |long | The logical sequence number of the event within the partition stream of the Event Hubs
100
+
| x-opt-offset |string | The offset of the event from the Event Hubs partition stream. The offset identifier is unique within a partition of the Event Hubs stream |
91
101
| x-opt-publisher |string | The publisher name, if the message was sent to a publisher endpoint |
92
102
| x-opt-partition-key |string |The partition key of the corresponding partition that stored the event |
93
103
@@ -97,36 +107,36 @@ If you selected **Event system properties** in the **Data Source** section of th
> For best performance, create all resources in the same region as the Azure Synapse Data Explorer cluster.
104
114
105
-
### Create an Event Hub
115
+
### Create an Event Hubs
106
116
107
-
If you don't already have one, [Create an Event Hub](../../../event-hubs/event-hubs-create.md). Connecting to Event Hub can be managed through the [Azure portal](data-explorer-ingest-event-hub-portal.md), programmatically with [C#](data-explorer-ingest-event-hub-csharp.md) or [Python](data-explorer-ingest-event-hub-python.md), or with the [Azure Resource Manager template](data-explorer-ingest-event-hub-resource-manager.md).
117
+
If you don't already have one, [Create an Event Hubs](../../../event-hubs/event-hubs-create.md). Connecting to Event Hubs can be managed through the [Azure portal](data-explorer-ingest-event-hub-portal.md), programmatically with [C#](data-explorer-ingest-event-hub-csharp.md) or [Python](data-explorer-ingest-event-hub-python.md), or with the [Azure Resource Manager template](data-explorer-ingest-event-hub-resource-manager.md).
108
118
109
119
> [!Note]
110
120
> * The partition count isn't changeable, so you should consider long-term scale when setting partition count.
111
121
> * Consumer group *must* be unique per consumer. Create a consumer group dedicated to Azure Synapse Data Explorer connection.
112
122
113
123
## Send events
114
124
115
-
See the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) that generates data and sends it to an Event Hub.
125
+
See the [sample app](https://github.com/Azure-Samples/event-hubs-dotnet-ingest) that generates data and sends it to an Event Hubs.
116
126
117
-
For an example of how to generate sample data, see [Ingest data from Event Hub into Azure Synapse Data Explorer](data-explorer-ingest-event-hub-portal.md#generate-sample-data)
127
+
For an example of how to generate sample data, see [Ingest data from Event Hubs into Azure Synapse Data Explorer](data-explorer-ingest-event-hub-portal.md#generate-sample-data)
118
128
119
129
## Set up Geo-disaster recovery solution
120
130
121
-
Event Hub offers a [Geo-disaster recovery](../../../event-hubs/event-hubs-geo-dr.md) solution.
122
-
Azure Synapse Data Explorer doesn't support `Alias` Event Hub namespaces. To implement the Geo-disaster recovery in your solution, create two Event Hub data connections: one for the primary namespace and one for the secondary namespace. Azure Synapse Data Explorer will listen to both Event Hub connections.
131
+
Event Hubs offers a [Geo-disaster recovery](../../../event-hubs/event-hubs-geo-dr.md) solution.
132
+
Azure Synapse Data Explorer doesn't support `Alias` Event Hubs namespaces. To implement the Geo-disaster recovery in your solution, create two Event Hubs data connections: one for the primary namespace and one for the secondary namespace. Azure Synapse Data Explorer will listen to both Event Hubs connections.
123
133
124
134
> [!NOTE]
125
135
> It's the user's responsibility to implement a failover from the primary namespace to the secondary namespace.
126
136
127
137
## Next steps
128
138
129
-
-[Ingest data from Event Hub into Azure Synapse Data Explorer](data-explorer-ingest-event-hub-portal.md)
130
-
-[Create an Event Hub data connection for Azure Synapse Data Explorer using C#](data-explorer-ingest-event-hub-csharp.md)
131
-
-[Create an Event Hub data connection for Azure Synapse Data Explorer using Python](data-explorer-ingest-event-hub-python.md)
132
-
-[Create an Event Hub data connection for Azure Synapse Data Explorer using Azure Resource Manager template](data-explorer-ingest-event-hub-resource-manager.md)
139
+
-[Ingest data from Event Hubs into Azure Synapse Data Explorer](data-explorer-ingest-event-hub-portal.md)
140
+
-[Create an Event Hubs data connection for Azure Synapse Data Explorer using C#](data-explorer-ingest-event-hub-csharp.md)
141
+
-[Create an Event Hubs data connection for Azure Synapse Data Explorer using Python](data-explorer-ingest-event-hub-python.md)
142
+
-[Create an Event Hubs data connection for Azure Synapse Data Explorer using Azure Resource Manager template](data-explorer-ingest-event-hub-resource-manager.md)
0 commit comments