You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/stream-analytics/stream-analytics-introduction.md
+37-27Lines changed: 37 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,81 +10,91 @@ ms.date: 07/08/2022
10
10
11
11
# Welcome to Azure Stream Analytics
12
12
13
-
Azure Stream Analytics is a fully managed stream processing engine that is designed to analyze and process large volumes of streaming data with sub-millisecond latencies. Patterns and relationships can be identified in data that originates from a variety of input sources including applications, devices, sensors, clickstreams, and social media feeds. These patterns can be used to trigger actions and initiate workflows such as creating alerts, feeding information to a reporting tool, or storing transformed data for later use. Stream Analytics is also available on the Azure IoT Edge runtime, enabling to process data directly on IoT devices.
13
+
Azure Stream Analytics is a fully managed stream processing engine that is designed to analyze and process large volumes of streaming data with sub-millisecond latencies. You can build a streaming data pipeline using Stream Analytics to identity patterns and relationships in data that originates from various input sources including applications, devices, sensors, clickstreams, and social media feeds. Then, you can use these patterns to trigger actions and initiate workflows such as raising alerts, feeding information to a reporting tool, or storing transformed data for later use. Stream Analytics is also available on the Azure IoT Edge runtime, which enables you to process data directly from IoT devices.
14
14
15
-
The following scenarios are examples of when you can use Azure Stream Analytics:
15
+
Here are a few example scenarios where you can use Azure Stream Analytics:
16
16
17
-
* Streaming ETL pipeline to Azure Storage in Parquet format
18
-
* Event driven applications with Azure SQL Database and Azure Cosmos DB
19
-
* Analyze real-time telemetry streams and logs from applications and IoT devices
20
-
* Real-time dashboarding with Power BI
21
-
* Anomaly detection to detect spikes, dips, and slow positive and negative changes in sensor values
22
-
* Geospatial analytics for fleet management and driverless vehicles
17
+
* Anomaly detection in sensor data to detect spikes, dips, and slow positive and negative changes
18
+
* Geo-spatial analytics for fleet management and driverless vehicles
23
19
* Remote monitoring and predictive maintenance of high value assets
24
20
* Clickstream analytics to determine customer behavior
21
+
* Analyze real-time telemetry streams and logs from applications and IoT devices
25
22
26
-
You can try Azure Stream Analytics with a free Azure subscription.
23
+
:::image type="content" source="./media/stream-analytics-introduction/stream-analytics-e2e-pipeline.png" alt-text="Diagram that shows the stages Ingest, Analyze, and Deliver stages of a streaming pipeline." lightbox="./media/stream-analytics-introduction/stream-analytics-e2e-pipeline.png":::
Azure Stream Analytics is a fully managed (PaaS) offering on Azure. You don't have to provision any hardware or infrastructure, update OS, or software. Azure Stream Analytics fully manages your job, so you can focus on your business logic and not on the infrastructure.
34
30
35
-
## Ease of use
31
+
## Ease of use
36
32
37
-
Azure Stream Analytics is easy to start. It only takes a few clicks to connect to multiple sources and sinks, creating an end-to-end pipeline. Stream Analytics can connect to Azure Event Hubs and Azure IoT Hub for streaming data ingestion, as well as Azure Blob storage to ingest historical data. Job input can also include static or slow-changing reference data from Azure Blob storage or SQL Database that you can join to streaming data to perform lookup operations.
33
+
Azure Stream Analytics is easy to start. It takes only a few clicks to create an end-to-end streaming data pipeline that connects to multiple sources and sinks.
38
34
39
-
Stream Analytics can route job output to many storage systems such as Azure Blob storage, Azure SQL Database, Azure Data Lake Store, and Azure Cosmos DB. You can also run batch analytics on stream outputs with Azure Synapse Analytics or HDInsight, or you can send the output to another service, like Event Hubs for consumption or Power BI for real-time visualization. For the entire list of Stream Analytics outputs, see [Understand outputs from Azure Stream Analytics](stream-analytics-define-outputs.md).
35
+
You can create a Stream Analytics job that can connect to Azure Event Hubs and Azure IoT Hub for streaming data ingestion, and Azure Blob storage or Azure Data Lake Storage Gen2 to ingest historical data. The input for the Stream Analytics job can also include static or slow-changing reference data from Azure Blob storage or SQL Database that you can join with streaming data to perform lookup operations. For more information on Stream Analytics **inputs**, see [Stream data as input into Stream Analytics](stream-analytics-define-inputs.md).
40
36
41
-
The Azure Stream Analytics no-code editor offers a no-code experience that enables you to develop Stream Analytics jobs effortlessly, using drag-and-drop functionality, without having to write any code. It further simplifies Stream Analytics job development experience. To learn more about the no-code editor, see [No-code stream processing in Azure Stream Analytics](./no-code-stream-processing.md)
37
+
You can route output from a Stream Analytics job to many storage systems such as Azure Blob storage, Azure SQL Database, Azure Data Lake Store, and Azure Cosmos DB. You can also run batch analytics on stream outputs with Azure Synapse Analytics or HDInsight, or you can send the output to another service, like Event Hubs for consumption or Power BI for real-time visualization. For the entire list of Stream Analytics **outputs**, see [Understand outputs from Azure Stream Analytics](stream-analytics-define-outputs.md).
38
+
39
+
The Azure Stream Analytics no-code editor offers a no-code experience that enables you to develop Stream Analytics jobs effortlessly, using drag-and-drop functionality, without having to write any code. It further simplifies Stream Analytics job development experience. To learn more about the no-code editor, see [No-code stream processing in Azure Stream Analytics](./no-code-stream-processing.md).
42
40
43
41
## Programmer productivity
44
42
45
-
Azure Stream Analytics uses a SQL query language that has been augmented with powerful temporal constraints to analyze data in motion. You can also create jobs by using developer tools like Azure PowerShell, Azure CLI, Stream Analytics Visual Studio tools, the [Stream Analytics Visual Studio Code extension](quick-create-visual-studio-code.md), or Azure Resource Manager templates. Using developer tools allows you to develop transformation queries offline and use the CI/CD pipeline to submit jobs to Azure.
43
+
Azure Stream Analytics uses a SQL query language that's augmented with powerful temporal constraints to analyze data in motion. You can [create a Stream Analytics job using the Azure portal](stream-analytics-quick-create-portal.md). You can also create jobs by using developer tools such as the following ones:
46
44
47
-
The Stream Analytics query language allows you to perform CEP (Complex Event Processing) by offering a wide array of functions for analyzing streaming data. This query language supports simple data manipulation, aggregation and analytics functions, geospatial functions, pattern matching and anomaly detection. You can edit queries in the portal or using our development tools, and test them using sample data that is extracted from a live stream.
45
+
-[Visual Studio Code](quick-create-visual-studio-code.md)
You can extend the capabilities of the query language by defining and invoking additional functions. You can define function calls in the Azure Machine Learning to take advantage of Azure Machine Learning solutions, and integrate JavaScript or C# user-defined functions (UDFs) or user-defined aggregates to perform complex calculations as part a Stream Analytics query.
53
+
Developer tools allow you to develop transformation queries offline and use the CI/CD pipeline to submit jobs to Azure.
50
54
51
-
## Fully managed
55
+
The Stream Analytics query language allows you to perform Complex Event Processing (CEP) by offering a wide array of functions for analyzing streaming data. This query language supports simple data manipulation, aggregation and analytics functions, geospatial functions, pattern matching, and anomaly detection. You can edit queries in the portal or using development tools, and test them using sample data that is extracted from a live stream.
56
+
57
+
You can extend the capabilities of the query language by defining and invoking additional functions. You can define function calls in the Azure Machine Learning to take advantage of Azure Machine Learning solutions, and integrate JavaScript or C# user-defined functions (UDFs) or user-defined aggregates to perform complex calculations as part a Stream Analytics query.
52
58
53
-
Azure Stream Analytics is a fully managed (PaaS) offering on Azure. You don't have to provision any hardware or infrastructure, update OS or software. Azure Stream Analytics fully manages your job, so you can focus on your business logic and not on the infrastructure.
54
59
55
60
## Run in the cloud or on the intelligent edge
56
61
57
62
Azure Stream Analytics can run in the cloud, for large-scale analytics, or run on IoT Edge or Azure Stack for ultra-low latency analytics. Azure Stream Analytics uses the same tools and query language on both cloud and the edge, enabling developers to build truly hybrid architectures for stream processing.
58
63
59
64
## Low total cost of ownership
60
65
61
-
As a cloud service, Stream Analytics is optimized for cost. There are no upfront costs involved - you only pay for the [streaming units you consume](stream-analytics-streaming-unit-consumption.md). There is no commitment or cluster provisioning required, and you can scale the job up or down based on your business needs.
66
+
As a cloud service, Stream Analytics is optimized for cost. There are no upfront costs involved - you only pay for the [streaming units you consume](stream-analytics-streaming-unit-consumption.md). There's no commitment or cluster provisioning required, and you can scale the job up or down based on your business needs.
62
67
63
68
## Mission-critical ready
64
69
65
70
Azure Stream Analytics is available across multiple regions worldwide and is designed to run mission-critical workloads by supporting reliability, security, and compliance requirements.
66
71
67
72
### Reliability
68
73
69
-
Azure Stream Analytics guarantees exactly once event processing and at-least-once delivery of events, so events are never lost. Exactly once processing is guaranteed with selected output as described in Event Delivery Guarantees.
74
+
Azure Stream Analytics guarantees exactly once event processing and at-least-once delivery of events, so events are never lost. Exactly once processing is guaranteed with selected output as described in [Event Delivery Guarantees](/stream-analytics-query/event-delivery-guarantees-azure-stream-analytics).
70
75
71
76
Azure Stream Analytics has built-in recovery capabilities in case the delivery of an event fails. Stream Analytics also provides built-in checkpoints to maintain the state of your job and provides repeatable results.
72
77
73
78
As a managed service, Stream Analytics guarantees event processing with a 99.9% availability at a minute level of granularity.
74
79
75
80
### Security
76
81
77
-
In terms of security, Azure Stream Analytics encrypts all incoming and outgoing communications and supports TLS 1.2. Built-in checkpoints are also encrypted. Stream Analytics doesn't store the incoming data since all processing is done in-memory.
78
-
Stream Analytics also supports Azure Virtual Networks (VNET) when running a job in a [Stream Analytics Cluster](./cluster-overview.md).
82
+
In terms of security, Azure Stream Analytics encrypts all incoming and outgoing communications and supports Transport Layer Security (TLS) 1.2. Built-in checkpoints are also encrypted. Stream Analytics doesn't store the incoming data since all processing is done in-memory. Stream Analytics also supports Azure Virtual Networks when running a job in a [Stream Analytics Cluster](./cluster-overview.md).
79
83
80
84
81
85
## Performance
82
86
83
87
Stream Analytics can process millions of events every second and it can deliver results with ultra low latencies. It allows you to [scale out](stream-analytics-autoscale.md) to adjust to your workloads. Stream Analytics supports higher performance by partitioning, allowing complex queries to be parallelized and executed on multiple streaming nodes. Azure Stream Analytics is built on [Trill](https://github.com/Microsoft/Trill), a high-performance in-memory streaming analytics engine developed in collaboration with Microsoft Research.
84
88
89
+
85
90
## Next steps
86
91
87
-
You now have an overview of Azure Stream Analytics. Next, you can dive deep and create your first Stream Analytics job:
92
+
You can try Azure Stream Analytics with a free Azure subscription.
0 commit comments