Skip to content

Commit 16b5ec5

Browse files
v-albemiv-albemi
authored
Merge pull request #94423 from dominicbetts/central-custom-pnp
Update custom articles for preview template
2 parents 1f8e06f + f7500b1 commit 16b5ec5

29 files changed

+606
-20
lines changed

‎articles/iot-central-pnp/toc.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,9 +88,9 @@
8888
- name: Connect Azure Monitor action groups on rules
8989
href: ../iot-central/core/howto-use-action-groups-pnp.md?toc=/azure/iot-central-pnp/toc.json&bc=/azure/iot-central-pnp/breadcrumb/toc.json
9090
- name: Create custom rules
91-
href: ../iot-central/core/howto-create-custom-rules.md?toc=/azure/iot-central-pnp/toc.json&bc=/azure/iot-central-pnp/breadcrumb/toc.json
91+
href: ../iot-central/core/howto-create-custom-rules-pnp.md?toc=/azure/iot-central-pnp/toc.json&bc=/azure/iot-central-pnp/breadcrumb/toc.json
9292
- name: Create custom analytics with Databricks
93-
href: ../iot-central/core/howto-create-custom-analytics.md?toc=/azure/iot-central-pnp/toc.json&bc=/azure/iot-central-pnp/breadcrumb/toc.json
93+
href: ../iot-central/core/howto-create-custom-analytics-pnp.md?toc=/azure/iot-central-pnp/toc.json&bc=/azure/iot-central-pnp/breadcrumb/toc.json
9494
- name: Administration
9595
items:
9696
- name: Manage your application

‎articles/iot-central/core/howto-create-custom-analytics-pnp.md

Lines changed: 231 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
---
2+
title: Extend Azure IoT Central with custom analytics | Microsoft Docs
3+
description: As a solution developer, configure an IoT Central application to do custom analytics and visualizations. This solution uses Azure Databricks.
4+
author: dominicbetts
5+
ms.author: dobett
6+
ms.date: 11/01/2019
7+
ms.topic: conceptual
8+
ms.service: iot-central
9+
services: iot-central
10+
ms.custom: mvc
11+
manager: philmea
12+
---
13+
14+
# Extend Azure IoT Central with custom analytics using Azure Databricks (preview features)
15+
16+
[!INCLUDE [iot-central-pnp-original](../../../includes/iot-central-pnp-original-note.md)]
17+
18+
This how-to guide shows you, as a solution developer, how to extend your IoT Central application with custom analytics and visualizations. The example uses an [Azure Databricks](https://docs.microsoft.com/azure/azure-databricks/) workspace to analyze the IoT Central telemetry stream and to generate visualizations such as [box plots](https://wikipedia.org/wiki/Box_plot).
19+
20+
This how-to guide shows you how to extend IoT Central beyond what it can already do with the [built-in analytics tools](./howto-create-custom-analytics-pnp.md?toc=/azure/iot-central-pnp/toc.json&bc=/azure/iot-central-pnp/breadcrumb/toc.json).
21+
22+
In this how-to guide, you learn how to:
23+
24+
* Stream telemetry from an IoT Central application using *continuous data export*.
25+
* Create an Azure Databricks environment to analyze and plot device telemetry.
26+
27+
## Prerequisites
28+
29+
To complete the steps in this how-to guide, you need an active Azure subscription.
30+
31+
If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.
32+
33+
### IoT Central application
34+
35+
Create an IoT Central application on the [Azure IoT Central application manager](https://aka.ms/iotcentral) website with the following settings:
36+
37+
| Setting | Value |
38+
| ------- | ----- |
39+
| Payment plan | Pay-As-You-Go |
40+
| Application template | In-store analytics – condition monitoring |
41+
| Application name | Accept the default or choose your own name |
42+
| URL | Accept the default or choose your own unique URL prefix |
43+
| Directory | Your Azure Active Directory tenant |
44+
| Azure subscription | Your Azure subscription |
45+
| Region | Your nearest region |
46+
47+
The examples and screenshots in this article use the **United States** region. Choose a location close to you and make sure you create all your resources in the same region.
48+
49+
This application template includes two simulated thermostat devices that send telemetry.
50+
51+
### Resource group
52+
53+
Use the [Azure portal to create a resource group](https://portal.azure.com/#create/Microsoft.ResourceGroup) called **IoTCentralAnalysis** to contain the other resources you create. Create your Azure resources in the same location as your IoT Central application.
54+
55+
### Event Hubs namespace
56+
57+
Use the [Azure portal to create an Event Hubs namespace](https://portal.azure.com/#create/Microsoft.EventHub) with the following settings:
58+
59+
| Setting | Value |
60+
| ------- | ----- |
61+
| Name | Choose your namespace name |
62+
| Pricing tier | Basic |
63+
| Subscription | Your subscription |
64+
| Resource group | IoTCentralAnalysis |
65+
| Location | East US |
66+
| Throughput Units | 1 |
67+
68+
### Azure Databricks workspace
69+
70+
Use the [Azure portal to create an Azure Databricks Service](https://portal.azure.com/#create/Microsoft.Databricks) with the following settings:
71+
72+
| Setting | Value |
73+
| ------- | ----- |
74+
| Workspace name | Choose your workspace name |
75+
| Subscription | Your subscription |
76+
| Resource group | IoTCentralAnalysis |
77+
| Location | East US |
78+
| Pricing Tier | Standard |
79+
80+
When you've created the required resources, your **IoTCentralAnalysis** resource group looks like the following screenshot:
81+
82+
![IoT Central analysis resource group](media/howto-create-custom-analytics/resource-group.png)
83+
84+
## Create an event hub
85+
86+
You can configure an IoT Central application to continuously export telemetry to an event hub. In this section, you create an event hub to receive telemetry from your IoT Central application. The event hub delivers the telemetry to your Stream Analytics job for processing.
87+
88+
1. In the Azure portal, navigate to your Event Hubs namespace and select **+ Event Hub**.
89+
1. Name your event hub **centralexport**, and select **Create**.
90+
1. In the list of event hubs in your namespace, select **centralexport**. Then choose **Shared access policies**.
91+
1. Select **+ Add**. Create a policy named **Listen** with the **Listen** claim.
92+
1. When the policy is ready, select it in the list, and then copy the **Connection string-primary key** value.
93+
1. Make a note of this connection string, you use it later when you configure your Databricks notebook to read from the event hub.
94+
95+
Your Event Hubs namespace looks like the following screenshot:
96+
97+
![Event Hubs namespace](media/howto-create-custom-analytics/event-hubs-namespace.png)
98+
99+
## Configure export in IoT Central
100+
101+
On the [Azure IoT Central application manager](https://aka.ms/iotcentral) website, navigate to the IoT Central application you created from the Contoso template. In this section, you configure the application to stream the telemetry from its simulated devices to your event hub. To configure the export:
102+
103+
1. Navigate to the **Data Export** page, select **+ New**, and then **Azure Event Hubs**.
104+
1. Use the following settings to configure the export, then select **Save**:
105+
106+
| Setting | Value |
107+
| ------- | ----- |
108+
| Display Name | Export to Event Hubs |
109+
| Enabled | On |
110+
| Event Hubs namespace | Your Event Hubs namespace name |
111+
| Event hub | centralexport |
112+
| Measurements | On |
113+
| Devices | Off |
114+
| Device Templates | Off |
115+
116+
![Data export configuration](media/howto-create-custom-analytics/cde-configuration.png)
117+
118+
Wait until the export status is **Running** before you continue.
119+
120+
## Configure Databricks workspace
121+
122+
In the Azure portal, navigate to your Azure Databricks service and select **Launch Workspace**. A new tab opens in your browser and signs you in to your workspace.
123+
124+
### Create a cluster
125+
126+
On the **Azure Databricks** page, under the list of common tasks, select **New Cluster**.
127+
128+
Use the information in the following table to create your cluster:
129+
130+
| Setting | Value |
131+
| ------- | ----- |
132+
| Cluster Name | centralanalysis |
133+
| Cluster Mode | Standard |
134+
| Databricks Runtime Version | 5.5 LTS (Scala 2.11, Spark 2.4.3) |
135+
| Python Version | 3 |
136+
| Enable Autoscaling | No |
137+
| Terminate after minutes of inactivity | 30 |
138+
| Worker Type | Standard_DS3_v2 |
139+
| Workers | 1 |
140+
| Driver Type | Same as worker |
141+
142+
Creating a cluster may take several minutes, wait for the cluster creation to complete before you continue.
143+
144+
### Install libraries
145+
146+
On the **Clusters** page, wait until the cluster state is **Running**.
147+
148+
The following steps show you how to import the library your sample needs into the cluster:
149+
150+
1. On the **Clusters** page, wait until the state of the **centralanalysis** interactive cluster is **Running**.
151+
152+
1. Select the cluster and then choose the **Libraries** tab.
153+
154+
1. On the **Libraries** tab, choose **Install New**.
155+
156+
1. On the **Install Library** page, choose **Maven** as the library source.
157+
158+
1. In the **Coordinates** textbox, enter the following value: `com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.10`
159+
160+
1. Choose **Install** to install the library on the cluster.
161+
162+
1. The library status is now **Installed**:
163+
164+
![Library installed](media/howto-create-custom-analytics/cluster-libraries.png)
165+
166+
### Import a Databricks notebook
167+
168+
Use the following steps to import a Databricks notebook that contains the Python code to analyze and visualize your IoT Central telemetry:
169+
170+
1. Navigate to the **Workspace** page in your Databricks environment. Select the dropdown next to your account name and then choose **Import**.
171+
172+
1. Choose to import from a URL and enter the following address: [https://github.com/Azure-Samples/iot-central-docs-samples/blob/master/databricks/IoT%20Central%20Analysis.dbc?raw=true](https://github.com/Azure-Samples/iot-central-docs-samples/blob/master/databricks/IoT%20Central%20Analysis.dbc?raw=true)
173+
174+
1. To import the notebook, choose **Import**.
175+
176+
1. Select the **Workspace** to view the imported notebook:
177+
178+
![Imported notebook](media/howto-create-custom-analytics/import-notebook.png)
179+
180+
1. Edit the code in the first Python cell to add the Event Hubs connection string you saved previously:
181+
182+
```python
183+
from pyspark.sql.functions import *
184+
from pyspark.sql.types import *
185+
186+
###### Event Hub Connection strings ######
187+
telementryEventHubConfig = {
188+
'eventhubs.connectionString' : '{your Event Hubs connection string}'
189+
}
190+
```
191+
192+
## Run analysis
193+
194+
To run the analysis, you must attach the notebook to the cluster:
195+
196+
1. Select **Detached** and then select the **centralanalysis** cluster.
197+
1. If the cluster isn't running, start it.
198+
1. To start the notebook, select the run button.
199+
200+
You may see an error in the last cell. If so, check the previous cells are running, wait a minute for some data to be written to storage, and then run the last cell again.
201+
202+
### View smoothed data
203+
204+
In the notebook, scroll down to cell 14 to see a plot of the rolling average humidity by device type. This plot continuously updates as streaming telemetry arrives:
205+
206+
![Smoothed telemetry plot](media/howto-create-custom-analytics/telemetry-plot.png)
207+
208+
You can resize the chart in the notebook.
209+
210+
### View box plots
211+
212+
In the notebook, scroll down to cell 20 to see the [box plots](https://en.wikipedia.org/wiki/Box_plot). The box plots are based on static data so to update them you must rerun the cell:
213+
214+
![Box plots](media/howto-create-custom-analytics/box-plots.png)
215+
216+
You can resize the plots in the notebook.
217+
218+
## Tidy up
219+
220+
To tidy up after this how-to and avoid unnecessary costs, delete the **IoTCentralAnalysis** resource group in the Azure portal.
221+
222+
You can delete the IoT Central application from the **Management** page within the application.
223+
224+
## Next steps
225+
226+
In this how-to guide, you learned how to:
227+
228+
* Stream telemetry from an IoT Central application using *continuous data export*.
229+
* Create an Azure Databricks environment to analyze and plot telemetry data.
230+
231+
Now that you know how to create custom analytics, the suggested next step is to learn how to [Visualize and analyze your Azure IoT Central data in a Power BI dashboard](./howto-connect-powerbi.md?toc=/azure/iot-central-pnp/toc.json&bc=/azure/iot-central-pnp/breadcrumb/toc.json).

0 commit comments

Comments
 (0)