You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-sql-edge/stream-data.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ ms.topic: conceptual
8
8
author: rothja
9
9
ms.author: jroth
10
10
ms.reviewer: jroth
11
-
ms.date: 05/19/2020
11
+
ms.date: 07/08/2022
12
12
---
13
13
14
14
# Data streaming in Azure SQL Edge
@@ -26,7 +26,7 @@ T-SQL streaming can help you:
26
26
27
27
## How does T-SQL streaming work?
28
28
29
-
T-SQL streaming works in exactly the same manner as [Azure Stream Analytics](../stream-analytics/stream-analytics-introduction.md#how-does-stream-analytics-work). For example, it uses the concept of streaming *jobs* for processing of real-time data streaming.
29
+
T-SQL streaming works in exactly the same manner as [Azure Stream Analytics](../stream-analytics/stream-analytics-introduction.md). For example, it uses the concept of streaming *jobs* for processing of real-time data streaming.
30
30
31
31
A stream analytics job consists of:
32
32
@@ -59,4 +59,4 @@ The following limitations and restrictions apply to T-SQL streaming.
59
59
60
60
-[Create a Stream Analytics job in Azure SQL Edge ](create-stream-analytics-job.md)
61
61
-[Viewing metadata associated with stream jobs in Azure SQL Edge ](streaming-catalog-views.md)
Copy file name to clipboardExpand all lines: articles/stream-analytics/stream-analytics-introduction.md
+11-23Lines changed: 11 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,66 +4,54 @@ description: Learn about Azure Stream Analytics, a managed service that helps yo
4
4
ms.service: stream-analytics
5
5
ms.topic: overview
6
6
ms.custom: mvc, contperf-fy21q2
7
-
ms.date: 8/20/2021
7
+
ms.date: 07/08/2022
8
8
#Customer intent: What is Azure Stream Analytics and why should I care? As an IT Pro or developer, how do I use Stream Analytics to perform analytics on data streams?
9
9
---
10
10
11
11
# Welcome to Azure Stream Analytics
12
12
13
-
Azure Stream Analytics is a real-time analytics and complex event-processing engine that is designed to analyze and process high volumes of fast streaming data from multiple sources simultaneously. Patterns and relationships can be identified in information extracted from a number of input sources including devices, sensors, clickstreams, social media feeds, and applications. These patterns can be used to trigger actions and initiate workflows such as creating alerts, feeding information to a reporting tool, or storing transformed data for later use. Also, Stream Analytics is available on Azure IoT Edge runtime, enabling to process data on IoT devices.
13
+
Azure Stream Analytics is a fully managed stream processing engine that is designed to analyze and process large volumes of streaming data with sub-millisecond latencies. Patterns and relationships can be identified in data that originates from a variety of input sources including applications, devices, sensors, clickstreams, and social media feeds. These patterns can be used to trigger actions and initiate workflows such as creating alerts, feeding information to a reporting tool, or storing transformed data for later use. Stream Analytics is also available on the Azure IoT Edge runtime, enabling to process data directly on IoT devices.
14
14
15
15
The following scenarios are examples of when you can use Azure Stream Analytics:
16
16
17
-
* Analyze real-time telemetry streams from IoT devices
18
-
* Web logs/clickstream analytics
17
+
* Streaming ETL pipeline to Azure Storage in Parquet format
18
+
* Event driven applications with Azure SQL Database and Azure Cosmos DB
19
+
* Analyze real-time telemetry streams and logs from applications and IoT devices
20
+
* Real-time dashboarding with Power BI
21
+
* Anomaly detection to detect spikes, dips, and slow positive and negative changes in sensor values
19
22
* Geospatial analytics for fleet management and driverless vehicles
20
23
* Remote monitoring and predictive maintenance of high value assets
21
-
*Real-time analytics on Point of Sale data for inventory control and anomaly detection
24
+
*Clickstream analytics to determine customer behavior
22
25
23
26
You can try Azure Stream Analytics with a free Azure subscription.
An Azure Stream Analytics job consists of an input, query, and an output. Stream Analytics ingests data from Azure Event Hubs (including Azure Event Hubs from Apache Kafka), Azure IoT Hub, or Azure Blob Storage. The query, which is based on SQL query language, can be used to easily filter, sort, aggregate, and join streaming data over a period of time. You can also extend this SQL language with JavaScript and C# user-defined functions (UDFs). You can easily adjust the event ordering options and duration of time windows when performing aggregation operations through simple language constructs and/or configurations.
31
-
32
-
Each job has one or several outputs for the transformed data, and you can control what happens in response to the information you've analyzed. For example, you can:
33
-
34
-
* Send data to services such as Azure Functions, Service Bus Topics or Queues to trigger communications or custom workflows downstream.
35
-
* Send data to a Power BI dashboard for real-time dashboarding.
36
-
* Store data in other Azure storage services (for example, Azure Data Lake, Azure Synapse Analytics, etc.) to train a machine learning model based on historical data or perform batch analytics.
37
-
38
-
The following image shows how data is sent to Stream Analytics, analyzed, and sent for other actions like storage or presentation:
Azure Stream Analytics is designed to be easy to use, flexible, reliable, and scalable to any job size. It is available across multiple Azure regions, and runs on IoT Edge or Azure Stack.
45
-
46
-
## Ease of getting started
35
+
## Ease of use
47
36
48
37
Azure Stream Analytics is easy to start. It only takes a few clicks to connect to multiple sources and sinks, creating an end-to-end pipeline. Stream Analytics can connect to Azure Event Hubs and Azure IoT Hub for streaming data ingestion, as well as Azure Blob storage to ingest historical data. Job input can also include static or slow-changing reference data from Azure Blob storage or SQL Database that you can join to streaming data to perform lookup operations.
49
38
50
-
Stream Analytics can route job output to many storage systems such as Azure Blob storage, Azure SQL Database, Azure Data Lake Store, and Azure CosmosDB. You can also run batch analytics on stream outputs with Azure Synapse Analytics or HDInsight, or you can send the output to another service, like Event Hubs for consumption or Power BI for real-time visualization.
39
+
Stream Analytics can route job output to many storage systems such as Azure Blob storage, Azure SQL Database, Azure Data Lake Store, and Azure Cosmos DB. You can also run batch analytics on stream outputs with Azure Synapse Analytics or HDInsight, or you can send the output to another service, like Event Hubs for consumption or Power BI for real-time visualization.
51
40
52
41
For the entire list of Stream Analytics outputs, see [Understand outputs from Azure Stream Analytics](stream-analytics-define-outputs.md).
53
42
54
43
## Programmer productivity
55
44
56
45
Azure Stream Analytics uses a SQL query language that has been augmented with powerful temporal constraints to analyze data in motion. You can also create jobs by using developer tools like Azure PowerShell, Azure CLI, Stream Analytics Visual Studio tools, the [Stream Analytics Visual Studio Code extension](quick-create-visual-studio-code.md), or Azure Resource Manager templates. Using developer tools allows you to develop transformation queries offline and use the CI/CD pipeline to submit jobs to Azure.
57
46
58
-
The Stream Analytics query language allows to perform CEP (Complex Event Processing) by offering a wide array of functions for analyzing streaming data. This query language supports simple data manipulation, aggregation and analytics functions, geospatial functions, pattern matching and anomaly detection. You can edit queries in the portal or using our development tools, and test them using sample data that is extracted from a live stream.
47
+
The Stream Analytics query language allows you to perform CEP (Complex Event Processing) by offering a wide array of functions for analyzing streaming data. This query language supports simple data manipulation, aggregation and analytics functions, geospatial functions, pattern matching and anomaly detection. You can edit queries in the portal or using our development tools, and test them using sample data that is extracted from a live stream.
59
48
60
49
You can extend the capabilities of the query language by defining and invoking additional functions. You can define function calls in the Azure Machine Learning to take advantage of Azure Machine Learning solutions, and integrate JavaScript or C# user-defined functions (UDFs) or user-defined aggregates to perform complex calculations as part a Stream Analytics query.
61
50
62
51
## Fully managed
63
52
64
53
Azure Stream Analytics is a fully managed (PaaS) offering on Azure. You don't have to provision any hardware or infrastructure, update OS or software. Azure Stream Analytics fully manages your job, so you can focus on your business logic and not on the infrastructure.
65
54
66
-
67
55
## Run in the cloud or on the intelligent edge
68
56
69
57
Azure Stream Analytics can run in the cloud, for large-scale analytics, or run on IoT Edge or Azure Stack for ultra-low latency analytics. Azure Stream Analytics uses the same tools and query language on both cloud and the edge, enabling developers to build truly hybrid architectures for stream processing.
description: This article describes Azure Stream Analytics resource model which includes the Azure Stream Analytics input, output, and query.
4
+
author: sidramadoss
5
+
ms.author: sidram
6
+
ms.service: stream-analytics
7
+
ms.topic: conceptual
8
+
ms.date: 07/08/2022
9
+
---
10
+
11
+
# Azure Stream Analytics resource model
12
+
13
+
Azure Stream Analytics is a fully managed platform-as-a-service (PaaS) for stream processing. This article describes the resource model for Stream Analytics by introducing the concept of a Stream Analytics cluster, job and the components of a job.
14
+
15
+
## Stream Analytics job
16
+
A Stream Analytics job is the fundamental unit in Stream Analytics that allows you to define and run your stream processing logic. A job consists of 3 main components:
17
+
* Input
18
+
* Output
19
+
* Query
20
+
21
+
### Input
22
+
A job can have one or more inputs to continuously read data from. These streaming input data sources could be an Azure Event Hubs, Azure IoT Hub or Azure Storage. Stream Analytics also supports reading static or slow changing input data (called reference data) which is often used to enrich streaming data. Adding these inputs to your job is a zero-code operation.
23
+
24
+
### Output
25
+
A job can have one or more outputs to continuously write data to. Stream Analytics supports 12 different output sinks including Azure SQL Database, Azure Data Lake Storage, Azure Cosmos DB, Power BI and more. Adding these outputs to your job is also a zero-code operation.
26
+
27
+
### Query
28
+
You can implement your stream processing logic by writing a SQL query in your job. The rich SQL language support allows you to tackle scenarios such as parsing complex JSON, filtering values, computing aggregates, performing joins, and even more advanced use cases such as geospatial analytics and anomaly detection. You can also extend this SQL language with JavaScript user-defined-functions (UDF) and user-defined-aggregates (UDA). Stream Analytics also allows you to easily adjust for late and out-of-order events through simple configurations in your job's settings. You can also choose to execute your query based on input event's arrival time at the input source or when the event was generated at the event source.
29
+
30
+
## Running a job
31
+
Once you have developed your job by configuring inputs, output and a query, you can start your job by specifying number of Streaming Units. Once your job has started, it goes into a **Running** state and will stay in that state until explicitly stopped or it runs into an unrecoverable failure. When the job is in a running state, it continuously pulls data from your input sources, executes the query logic which produces results that get written to your output sinks with milliseconds end-to-end latency.
32
+
33
+
When your job is started, the Stream Analytics service takes care of compiling your query and assigns certain amount of compute and memory based on the number of Streaming Units configured in your job. You don't have to worry about any underlying infrastructure as cluster maintenance, security patches as that is automatically taken care by the platform. When running jobs in the Standard SKU, you are charged for the Streaming Units only when the job runs.
34
+
35
+
## Stream Analytics cluster
36
+
By default, Stream Analytics jobs run in the Standard multi-tenant environment which forms the Standard SKU. Stream Analytics also provides a Dedicated SKU where you can provision an entire Stream Analytics cluster that belongs to you. Doing so gives you full control of which jobs run on your cluster. The minimum size of a Stream Analytics cluster is 36 Streaming Units and you are charged for the entire cluster capacity from when it gets provisioned. You can learn more about the [benefits of Stream Analytics clusters and when to use it](cluster-overview.md).
37
+
38
+
:::image type="content" source="./media/stream-analytics-resource-model/stream-analytics-standard-sku.png" alt-text="Diagram that shows Standard multi-tenant environment in Stream Analytics." border="false":::
39
+
40
+
:::image type="content" source="./media/stream-analytics-resource-model/stream-analytics-dedicated-sku.png" alt-text="Diagram that shows Dedicated environment in Stream Analytics." border="false":::
41
+
42
+
## Next steps
43
+
44
+
Learn how to manage your Azure Stream Analytics and other concepts:
45
+
46
+
*[Build a fraud detection solution using Stream Analytics](stream-analytics-real-time-fraud-detection.md)
0 commit comments