You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/stream-analytics/stream-analytics-use-reference-data.md
+7-9Lines changed: 7 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,14 +6,14 @@ ms.author: jeanb
6
6
ms.reviewer: mamccrea
7
7
ms.service: stream-analytics
8
8
ms.topic: conceptual
9
-
ms.date: 10/8/2019
9
+
ms.date: 5/11/2020
10
10
---
11
11
# Using reference data for lookups in Stream Analytics
12
12
13
13
Reference data (also known as a lookup table) is a finite data set that is static or slowly changing in nature, used to perform a lookup or to augment your data streams. For example, in an IoT scenario, you could store metadata about sensors (which don’t change often) in reference data and join it with real time IoT data streams. Azure Stream Analytics loads reference data in memory to achieve low latency stream processing. To make use of reference data in your Azure Stream Analytics job, you will generally use a [Reference Data Join](https://docs.microsoft.com/stream-analytics-query/reference-data-join-azure-stream-analytics) in your query.
14
14
15
15
## Example
16
-
If a commercial vehicle is registered with the Toll Company, they can pass through the toll booth without being stopped for inspection. We will use a commercial vehicle registration lookup table to identify all commercial vehicles with expired registration.
16
+
You can have a real time stream of events generated when cars pass a toll booth. The toll booth can capture the license plate in real time and join with a static dataset that has registration details to identify license plates that have expired.
@@ -105,15 +105,13 @@ You can use [Azure SQL Database Managed Instance](https://docs.microsoft.com/azu
105
105
106
106
## Size limitation
107
107
108
-
Stream Analytics supports reference data with **maximum size of 300 MB**. The 300 MB limit of maximum size of reference data is achievable only with simple queries. As the complexity of query increases to include stateful processing, such as windowed aggregates, temporal joins and temporal analytic functions, it is expected that the maximum supported size of reference data decreases. If Azure Stream Analytics cannot load the reference data and perform complex operations, the job will run out of memory and fail. In such cases, SU % Utilization metric will reach 100%.
108
+
It is recommended to use reference datasets which are less than 300 MB for best performance. Usage of reference data greater than 300 MB is supported in jobs with 6 SUs or more. This functionality is in preview and must not be used in production. Using a very large reference data may impact performance of your job. As the complexity of query increases to include stateful processing, such as windowed aggregates, temporal joins and temporal analytic functions, it is expected that the maximum supported size of reference data decreases. If Azure Stream Analytics cannot load the reference data and perform complex operations, the job will run out of memory and fail. In such cases, SU % Utilization metric will reach 100%.
109
109
110
-
|**Number of Streaming Units**|**Approx. Max Size Supported (in MB)**|
110
+
|**Number of Streaming Units**|**Recommended Size**|
111
111
|---------|---------|
112
-
|1 |50 |
113
-
|3 |150 |
114
-
|6 and beyond |300 |
115
-
116
-
Increasing number of Streaming Units of a job beyond 6 does not increase the maximum supported size of reference data.
112
+
|1 |50 MB or lower |
113
+
|3 |150 MB or lower |
114
+
|6 and beyond |300 MB or lower. Using reference data greater than 300 MB is supported in preview and could impact performance of your job. |
117
115
118
116
Support for compression is not available for reference data.
0 commit comments