You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/storm/apache-storm-develop-csharp-event-hub-topology.md
+12-22Lines changed: 12 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,14 +2,13 @@
2
2
title: Process events from Event Hubs with Storm - Azure HDInsight
3
3
description: Learn how to process data from Azure Event Hubs with a C# Storm topology created in Visual Studio, by using the HDInsight tools for Visual Studio.
4
4
author: hrasheed-msft
5
+
ms.author: hrasheed
5
6
ms.reviewer: jasonh
6
-
7
7
ms.service: hdinsight
8
8
ms.topic: conceptual
9
-
ms.date: 11/27/2017
10
-
ms.author: hrasheed
11
-
ROBOTS: NOINDEX
9
+
ms.date: 01/14/2020
12
10
---
11
+
13
12
# Process events from Azure Event Hubs with Apache Storm on HDInsight (C#)
14
13
15
14
Learn how to work with Azure Event Hubs from [Apache Storm](https://storm.apache.org/) on HDInsight. This document uses a C# Storm topology to read and write data from Event Hubs
@@ -21,20 +20,12 @@ Learn how to work with Azure Event Hubs from [Apache Storm](https://storm.apache
21
20
22
21
The steps in this document use SCP.NET, a NuGet package that makes it easy to create C# topologies and components for use with Storm on HDInsight.
23
22
24
-
> [!IMPORTANT]
25
-
> While the steps in this document rely on a Windows development environment with Visual Studio, the compiled project can be submitted to a Storm on HDInsight cluster that uses Linux. Only Linux-based clusters created after October 28, 2016, support SCP.NET topologies.
26
-
27
23
HDInsight 3.4 and greater use Mono to run C# topologies. The example used in this document works with HDInsight 3.6. If you plan on creating your own .NET solutions for HDInsight, check the [Mono compatibility](https://www.mono-project.com/docs/about-mono/compatibility/) document for potential incompatibilities.
28
24
29
25
### Cluster versioning
30
26
31
27
The Microsoft.SCP.Net.SDK NuGet package you use for your project must match the major version of Storm installed on HDInsight. HDInsight versions 3.5 and 3.6 use Storm 1.x, so you must use SCP.NET version 1.0.x.x with these clusters.
32
28
33
-
> [!IMPORTANT]
34
-
> The example in this document expects an HDInsight 3.5 or 3.6 cluster.
35
-
>
36
-
> Linux is the only operating system used on HDInsight version 3.4 or greater.
37
-
38
29
C# topologies must also target .NET 4.5.
39
30
40
31
## How to work with Event Hubs
@@ -104,9 +95,6 @@ You can download a complete version of the project created in this article from
104
95
105
96
* An Apache Storm cluster on HDInsight. See [Create Apache Hadoop clusters using the Azure portal](../hdinsight-hadoop-create-linux-clusters-portal.md) and select **Storm** for **Cluster type**.
106
97
107
-
> [!WARNING]
108
-
> The example used in this document requires Storm on HDInsight version 3.5 or 3.6. This does not work with older versions of HDInsight, due to breaking class name changes. For a version of this example that works with older clusters, see [GitHub](https://github.com/Azure-Samples/hdinsight-dotnet-java-storm-eventhub/releases).
109
-
110
98
* An [Azure event hub](../../event-hubs/event-hubs-create.md).
111
99
112
100
* The [Azure .NET SDK](https://azure.microsoft.com/downloads/).
@@ -128,7 +116,7 @@ Create a directory named `eventhubspout`, and save the file into the directory.
128
116
129
117
Event Hubs is the data source for this example. Use the information in the "Create an event hub" section of [Get started with Event Hubs](../../event-hubs/event-hubs-create.md).
130
118
131
-
1. After the event hub has been created, view the **EventHub** settings in the Azure portal, and select **Shared access policies**. Select **+ Add** to add the following policies:
119
+
1. After the event hub has been created, view the **EventHub** settings in the Azure portal, and select **Shared access policies**. Select **+ Add** to create the following policies:
132
120
133
121
| Name | Permissions |
134
122
| --- | --- |
@@ -141,11 +129,11 @@ Event Hubs is the data source for this example. Use the information in the "Crea
141
129
142
130
## Configure the EventHubWriter
143
131
144
-
1. If you have not already installed the latest version of the HDInsight tools for Visual Studio, see [Get started using HDInsight tools for Visual Studio](../hadoop/apache-hadoop-visual-studio-tools-get-started.md).
132
+
1. If you haven't already installed the latest version of the HDInsight tools for Visual Studio, see [Get started using HDInsight tools for Visual Studio](../hadoop/apache-hadoop-visual-studio-tools-get-started.md).
145
133
146
134
2. Download the solution from [eventhub-storm-hybrid](https://github.com/Azure-Samples/hdinsight-dotnet-java-storm-eventhub).
147
135
148
-
3. In the **EventHubWriter** project, open the **App.config** file. Use the information from the event hub that you configured earlier to fill in the value for the following keys:
136
+
3.Open **EventHubExample.sln**. In the **EventHubWriter** project, open the **App.config** file. Use the information from the event hub that you configured earlier to fill in the value for the following keys:
149
137
150
138
| Key | Value |
151
139
| --- | --- |
@@ -199,9 +187,11 @@ Event Hubs is the data source for this example. Use the information in the "Crea
199
187
200
188
9. In the **Executors** section, select one of the links in the **Port** column. This displays information logged by the component. The logged information is similar to the following text:
201
189
202
-
2017-03-02 14:51:29.255 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,255 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1830978598,"deviceId":"8566ccbc-034d-45db-883d-d8a31f34068e"}
203
-
2017-03-02 14:51:29.283 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,283 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1756413275,"deviceId":"647a5eff-823d-482f-a8b4-b95b35ae570b"}
204
-
2017-03-02 14:51:29.313 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,312 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1108478910,"deviceId":"206a68fa-8264-4d61-9100-bfdb68ee8f0a"}
190
+
```output
191
+
2017-03-02 14:51:29.255 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,255 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1830978598,"deviceId":"8566ccbc-034d-45db-883d-d8a31f34068e"}
192
+
2017-03-02 14:51:29.283 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,283 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1756413275,"deviceId":"647a5eff-823d-482f-a8b4-b95b35ae570b"}
193
+
2017-03-02 14:51:29.313 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,312 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1108478910,"deviceId":"206a68fa-8264-4d61-9100-bfdb68ee8f0a"}
194
+
```
205
195
206
196
## Stop the topologies
207
197
@@ -215,7 +205,7 @@ To stop the topologies, select each topology in the **Storm Topology Viewer**, t
215
205
216
206
## Next steps
217
207
218
-
In this document, you have learned how to use the Java Event Hubs spout and bolt from a C# topology to work with data in Azure Event Hubs. To learn more about creating C# topologies, see the following:
208
+
In this document, you've learned how to use the Java Event Hubs spout and bolt from a C# topology to work with data in Azure Event Hubs. To learn more about creating C# topologies, see the following:
219
209
220
210
* [Develop C# topologies for Apache Storm on HDInsight using Visual Studio](apache-storm-develop-csharp-visual-studio-topology.md)
0 commit comments