Skip to content

Commit 5acd4d4

Browse files
ktoliverktoliver
authored
Merge pull request #46660 from Fleid/patch-2
Add more info to existing samples
2 parents 2ef5cb1 + 5682c17 commit 5acd4d4

File tree

1 file changed

+163
-25
lines changed

1 file changed

+163
-25
lines changed

‎articles/stream-analytics/stream-analytics-parsing-json.md

Lines changed: 163 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,6 @@ ms.date: 06/21/2019
1212
Azure Stream Analytics support processing events in CSV, JSON, and Avro data formats. Both JSON and Avro data can be structured and contain some complex types such as nested objects (records) and arrays.
1313

1414

15-
16-
1715
## Record data types
1816
Record data types are used to represent JSON and Avro arrays when corresponding formats are used in the input data streams. These examples demonstrate a sample sensor, which is reading input events in JSON format. Here is example of a single event:
1917

@@ -40,7 +38,6 @@ Record data types are used to represent JSON and Avro arrays when corresponding
4038
}
4139
```
4240

43-
4441
### Access nested fields in known schema
4542
Use dot notation (.) to easily access nested fields directly from your query. For example, this query selects the Latitude and Longitude coordinates under the Location property in the preceding JSON data. The dot notation can be used to navigate multiple levels as shown below.
4643

@@ -49,57 +46,82 @@ SELECT
4946
DeviceID,
5047
Location.Lat,
5148
Location.Long,
49+
SensorReadings.Temperature,
5250
SensorReadings.SensorMetadata.Version
5351
FROM input
5452
```
5553

54+
The result is:
55+
56+
|DeviceID|Lat|Long|Temperature|Version|
57+
|-|-|-|-|-|
58+
|12345|47|122|80|1.2.45|
59+
60+
5661
### Select all properties
5762
You can select all the properties of a nested record using '*' wildcard. Consider the following example:
5863

5964
```SQL
60-
SELECT input.Location.*
65+
SELECT
66+
DeviceID,
67+
Location.*
6168
FROM input
6269
```
6370

6471
The result is:
6572

66-
```json
67-
{
68-
"Lat" : 47,
69-
"Long" : 122
70-
}
71-
```
73+
|DeviceID|Lat|Long|
74+
|-|-|-|
75+
|12345|47|122|
7276

7377

7478
### Access nested fields when property name is a variable
75-
Use the [GetRecordPropertyValue](https://docs.microsoft.com/stream-analytics-query/getrecordpropertyvalue-azure-stream-analytics) function if the property name is a variable.
7679

77-
For example, imagine a sample data stream needs to be joined with reference data containing thresholds for each device sensor. A snippet of such reference data is shown below.
80+
Use the [GetRecordPropertyValue](https://docs.microsoft.com/stream-analytics-query/getrecordpropertyvalue-azure-stream-analytics) function if the property name is a variable. This allows for building dynamic queries without hardcoding property names.
81+
82+
For example, imagine the sample data stream needs **to be joined with reference data** containing thresholds for each device sensor. A snippet of such reference data is shown below.
7883

7984
```json
8085
{
8186
"DeviceId" : "12345",
8287
"SensorName" : "Temperature",
83-
"Value" : 75
88+
"Value" : 85
89+
},
90+
{
91+
"DeviceId" : "12345",
92+
"SensorName" : "Humidity",
93+
"Value" : 65
8494
}
8595
```
8696

97+
The goal here is to join our sample dataset from the top of the article to that reference data, and output one event for each sensor measure above its threshold. That means our single event above can generate multiple output events if multiple sensors are above their respective thresholds, thanks to the join. To achieve similar results without a join, see the section below.
98+
8799
```SQL
88100
SELECT
89101
input.DeviceID,
90-
thresholds.SensorName
102+
thresholds.SensorName,
103+
"Alert : Sensor above threshold" AS AlertMessage
91104
FROM input -- stream input
92105
JOIN thresholds -- reference data input
93106
ON
94107
input.DeviceId = thresholds.DeviceId
95108
WHERE
96109
GetRecordPropertyValue(input.SensorReadings, thresholds.SensorName) > thresholds.Value
97-
-- the where statement selects the property value coming from the reference data
98110
```
99111

112+
**GetRecordPropertyValue** selects the property in *SensorReadings*, which name matches the property name coming from the reference data. Then the associated value from *SensorReadings* is extracted.
113+
114+
The result is:
115+
116+
|DeviceID|SensorName|AlertMessage|
117+
|-|-|-|
118+
|12345|Humidity|Alert : Sensor above threshold|
119+
100120
### Convert record fields into separate events
101-
To convert record fields into separate events, use the [APPLY](https://docs.microsoft.com/stream-analytics-query/apply-azure-stream-analytics) operator together with the [GetRecordProperties](https://docs.microsoft.com/stream-analytics-query/getrecordproperties-azure-stream-analytics) function.
102-
For example, if the previous example had several records for SensorReading, the following query could be used to extract them into different events:
121+
122+
To convert record fields into separate events, use the [APPLY](https://docs.microsoft.com/stream-analytics-query/apply-azure-stream-analytics) operator together with the [GetRecordProperties](https://docs.microsoft.com/stream-analytics-query/getrecordproperties-azure-stream-analytics) function.
123+
124+
With the original sample data, the following query could be used to extract properties into different events.
103125

104126
```SQL
105127
SELECT
@@ -110,42 +132,158 @@ FROM input as event
110132
CROSS APPLY GetRecordProperties(event.SensorReadings) AS sensorReading
111133
```
112134

135+
The result is:
136+
137+
|DeviceID|SensorName|AlertMessage|
138+
|-|-|-|
139+
|12345|Temperature|80|
140+
|12345|Humidity|70|
141+
|12345|CustomSensor01|5|
142+
|12345|CustomSensor02|99|
143+
|12345|SensorMetadata|[object Object]|
113144

145+
Using [WITH](https://docs.microsoft.com/stream-analytics-query/with-azure-stream-analytics), it's then possible to route those events to different destinations:
146+
147+
```SQL
148+
WITH Stage0 AS
149+
(
150+
SELECT
151+
event.DeviceID,
152+
sensorReading.PropertyName,
153+
sensorReading.PropertyValue
154+
FROM input as event
155+
CROSS APPLY GetRecordProperties(event.SensorReadings) AS sensorReading
156+
)
157+
158+
SELECT DeviceID, PropertyValue AS Temperature INTO TemperatureOutput FROM Stage0 WHERE PropertyName = 'Temperature'
159+
SELECT DeviceID, PropertyValue AS Humidity INTO HumidityOutput FROM Stage0 WHERE PropertyName = 'Humidity'
160+
```
114161

115162
## Array data types
116163

117-
Array data types are an ordered collection of values. Some typical operations on array values are detailed below. These examples assume the input events have a property named "arrayField" that is an array datatype.
164+
Array data types are an ordered collection of values. Some typical operations on array values are detailed below. These examples use the functions [GetArrayElement](https://docs.microsoft.com/stream-analytics-query/getarrayelement-azure-stream-analytics), [GetArrayElements](https://docs.microsoft.com/stream-analytics-query/getarrayelements-azure-stream-analytics), [GetArrayLength](https://docs.microsoft.com/stream-analytics-query/getarraylength-azure-stream-analytics), and the [APPLY](https://docs.microsoft.com/stream-analytics-query/apply-azure-stream-analytics) operator.
118165

119-
These examples use the functions [GetArrayElement](https://docs.microsoft.com/stream-analytics-query/getarrayelement-azure-stream-analytics), [GetArrayElements](https://docs.microsoft.com/stream-analytics-query/getarrayelements-azure-stream-analytics), [GetArrayLength](https://docs.microsoft.com/stream-analytics-query/getarraylength-azure-stream-analytics), and the [APPLY](https://docs.microsoft.com/stream-analytics-query/apply-azure-stream-analytics) operator.
166+
Here is an example of a single event. Both `CustomSensor03` and `SensorMetadata` are of type **array**:
167+
168+
```json
169+
{
170+
"DeviceId" : "12345",
171+
"SensorReadings" :
172+
{
173+
"Temperature" : 80,
174+
"Humidity" : 70,
175+
"CustomSensor01" : 5,
176+
"CustomSensor02" : 99,
177+
"CustomSensor03": [12,-5,0]
178+
},
179+
"SensorMetadata":[
180+
{
181+
"smKey":"Manufacturer",
182+
"smValue":"ABC"
183+
},
184+
{
185+
"smKey":"Version",
186+
"smValue":"1.2.45"
187+
}
188+
]
189+
}
190+
```
120191

121192
### Working with a specific array element
193+
122194
Select array element at a specified index (selecting the first array element):
123195

124196
```SQL
125197
SELECT
126-
GetArrayElement(arrayField, 0) AS firstElement
198+
GetArrayElement(SensorReadings.CustomSensor03, 0) AS firstElement
127199
FROM input
128200
```
129201

202+
The result is:
203+
204+
|firstElement|
205+
|-|
206+
|12|
207+
130208
### Select array length
131209

132210
```SQL
133211
SELECT
134-
GetArrayLength(arrayField) AS arrayLength
212+
GetArrayLength(SensorReadings.CustomSensor03) AS arrayLength
135213
FROM input
136214
```
137215

216+
The result is:
217+
218+
|arrayLength|
219+
|-|
220+
|3|
221+
138222
### Convert array elements into separate events
223+
139224
Select all array element as individual events. The [APPLY](https://docs.microsoft.com/stream-analytics-query/apply-azure-stream-analytics) operator together with the [GetArrayElements](https://docs.microsoft.com/stream-analytics-query/getarrayelements-azure-stream-analytics) built-in function extracts all array elements as individual events:
140225

141226
```SQL
142227
SELECT
143-
arrayElement.ArrayIndex,
144-
arrayElement.ArrayValue
145-
FROM input as event
146-
CROSS APPLY GetArrayElements(event.arrayField) AS arrayElement
228+
DeviceId,
229+
CustomSensor03Record.ArrayIndex,
230+
CustomSensor03Record.ArrayValue
231+
FROM input
232+
CROSS APPLY GetArrayElements(SensorReadings.CustomSensor03) AS CustomSensor03Record
233+
147234
```
148235

236+
The result is:
237+
238+
|DeviceId|ArrayIndex|ArrayValue|
239+
|-|-|-|
240+
|12345|0|12|
241+
|12345|1|-5|
242+
|12345|2|0|
243+
244+
```SQL
245+
SELECT
246+
i.DeviceId,
247+
SensorMetadataRecords.ArrayValue.smKey as smKey,
248+
SensorMetadataRecords.ArrayValue.smValue as smValue
249+
FROM input i
250+
CROSS APPLY GetArrayElements(SensorMetadata) AS SensorMetadataRecords
251+
```
252+
253+
The result is:
254+
255+
|DeviceId|smKey|smValue|
256+
|-|-|-|
257+
|12345|Manufacturer|ABC|
258+
|12345|Version|1.2.45|
259+
260+
If the extracted fields need to appear in columns, it is possible to pivot the dataset using the [WITH](https://docs.microsoft.com/stream-analytics-query/with-azure-stream-analytics) syntax in addition to the [JOIN](https://docs.microsoft.com/stream-analytics-query/join-azure-stream-analytics) operation. That join will require a [time boundary](https://docs.microsoft.com/stream-analytics-query/join-azure-stream-analytics#BKMK_DateDiff) condition that prevents duplication:
261+
262+
```SQL
263+
WITH DynamicCTE AS (
264+
SELECT
265+
i.DeviceId,
266+
SensorMetadataRecords.ArrayValue.smKey as smKey,
267+
SensorMetadataRecords.ArrayValue.smValue as smValue
268+
FROM input i
269+
CROSS APPLY GetArrayElements(SensorMetadata) AS SensorMetadataRecords
270+
)
271+
272+
SELECT
273+
i.DeviceId,
274+
i.Location.*,
275+
V.smValue AS 'smVersion',
276+
M.smValue AS 'smManufacturer'
277+
FROM input i
278+
LEFT JOIN DynamicCTE V ON V.smKey = 'Version' and V.DeviceId = i.DeviceId AND DATEDIFF(minute,i,V) BETWEEN 0 AND 0
279+
LEFT JOIN DynamicCTE M ON M.smKey = 'Manufacturer' and M.DeviceId = i.DeviceId AND DATEDIFF(minute,i,M) BETWEEN 0 AND 0
280+
```
281+
282+
The result is:
283+
284+
|DeviceId|Lat|Long|smVersion|smManufacturer|
285+
|-|-|-|-|-|
286+
|12345|47|122|1.2.45|ABC|
149287

150288
## See Also
151289
[Data Types in Azure Stream Analytics](https://docs.microsoft.com/stream-analytics-query/data-types-azure-stream-analytics)

0 commit comments

Comments
 (0)