Merge pull request #107916 from sidramadoss/patch-63

Azure ML UDFs

Author: megvanhuygen

articles/stream-analytics/TOC.yml

@@ -61,6 +61,12 @@
       href: stream-analytics-sql-output-perf.md
     - name: Blob custom path patterns
       href: stream-analytics-custom-path-patterns-blob-storage-output.md
+  - name: User-defined functions
+    items:
+    - name: Machine learning UDF
+      href: machine-learning-udf.md
+    - name: C# UDF
+      href: stream-analytics-edge-csharp-udf-methods.md
   - name: Optimize your Stream Analytics job
     items:
     - name: Understand and adjust Streaming Units
@@ -187,8 +193,6 @@
       href: stream-analytics-tools-for-visual-studio-edge-jobs.md
     - name: Set up CI/CD pipeline
       href: stream-analytics-tools-for-visual-studio-cicd.md
-    - name: Write .NET UDF
-      href: stream-analytics-edge-csharp-udf-methods.md
   - name: Visual Studio Code
     items:
     - name: Test locally with sample data

articles/stream-analytics/machine-learning-udf.md

@@ -0,0 +1,167 @@
+---
+title: Integrate Azure Stream Analytics with Azure Machine Learning
+description: This article describes how to integrate an Azure Stream Analytics job with Azure Machine Learning models.
+author: sidram
+ms.author: sidram
+ms.reviewer: mamccrea
+ms.service: stream-analytics
+ms.topic: conceptual
+ms.date: 03/19/2020
+---
+# Integrate Azure Stream Analytics with Azure Machine Learning (Preview)
+
+You can implement machine learning models as a user-defined function (UDF) in your Azure Stream Analytics jobs to do real-time scoring and predictions on your streaming input data. [Azure Machine Learning](../machine-learning/overview-what-is-azure-ml.md) allows you to use any popular open-source tool, such as Tensorflow, scikit-learn, or PyTorch, to prep, train, and deploy models.
+
+> [!NOTE]
+> This functionality is in public preview. You can access this feature on the Azure portal only by using the [Stream Analytics portal preview link](https://aka.ms/asaportalpreview). This functionality is also available in the latest version of [Stream Analytics tools for Visual Studio](https://docs.microsoft.com/azure/stream-analytics/stream-analytics-tools-for-visual-studio-install).
+
+## Prerequisites
+
+Complete the following steps before you add a machine learning model as a function to your Stream Analytics job:
+
+1. Use Azure Machine Learning to [deploy your model as a web service](https://docs.microsoft.com/azure/machine-learning/how-to-deploy-and-where).
+
+2. Your scoring script should have [sample inputs and outputs](../machine-learning/how-to-deploy-and-where.md#example-entry-script) which is used by Azure Machine Learning to generate a schema specification. Stream Analytics uses the schema to understand the function signature of your web service.
+
+3. Make sure your web service accepts and returns JSON serialized data.
+
+4. Deploy your model on [Azure Kubernetes Service](../machine-learning/how-to-deploy-and-where.md#choose-a-compute-target) for high-scale production deployments. If the web service is not able to handle the number of requests coming from your job, the performance of your Stream Analytics job will be degraded, which impacts latency.
+
+## Add a machine learning model to your job
+
+You can add Azure Machine Learning functions to your Stream Analytics job directly from the Azure portal.
+
+1. Navigate to your Stream Analytics job in the Azure portal, and select **Functions** under **Job topology**. Then, select **Azure ML Service** from the **+ Add** dropdown menu.
+
+   ![Add Azure ML UDF](./media/machine-learning-udf/add-azureml-udf.png)
+
+2. Fill in the **Azure Machine Learning Service function** form with the following property values:
+
+   ![Configure Azure ML UDF](./media/machine-learning-udf/configure-azureml-udf.png)
+
+The following table describes each property of Azure ML Service functions in Stream Analytics.
+
+|Property|Description|
+|--------|-----------|
+|Function alias|Enter a name to invoke the function in your query.|
+|Subscription|Your Azure subscription..|
+|Azure ML workspace|The Azure Machine Learning workspace you used to deploy your model as a web service.|
+|Deployments|The web service hosting your model.|
+|Function signature|The signature of your web service inferred from the API's schema specification. If your signature fails to load, check that you have provided sample input and output in your scoring script to automatically generate the schema.|
+|Number of parallel requests per partition|This is an advanced configuration to optimize high-scale throughput. This number represents the concurrent requests sent from each partition of your job to the web service. Jobs with six streaming units (SU) and lower have one partition. Jobs with 12 SUs have two partitions, 18 SUs have three partitions and so on.<br><br> For example, if your job has two partitions and you set this parameter to four, there will be eight concurrent requests from your job to your web service.|
+|Max batch count|This is an advanced configuration for optimizing high-scale throughput. This number represents the maximum number of events be batched together in a single request sent to your web service.|
+
+## Supported input parameters
+
+When your Stream Analytics query invokes an Azure Machine Learning UDF, the job creates a JSON serialized request to the web service. The request is based on a model-specific schema. You have to provide a sample input and output in your scoring script to [automatically generate a schema](../machine-learning/how-to-deploy-and-where.md#optional-automatic-schema-generation). The schema allows Stream Analytics to construct the JSON serialized request for any of the supported data types such as numpy, pandas and PySpark. Multiple input events can be batched together in a single request.
+
+The following Stream Analytics query is an example of how to invoke an Azure Machine Learning UDF:
+
+```SQL
+SELECT udf.score(<model-specific-data-structure>)
+INTO output
+FROM input
+```
+
+Stream Analytics only supports passing one parameter for Azure Machine Learning functions. You may need to prepare your data before passing it as an input to machine learning UDF.
+
+## Pass multiple input parameters to the UDF
+
+Most common examples of inputs to machine learning models are numpy arrays and DataFrames. You can create an array using a JavaScript UDF, and create a JSON-serialized DataFrame using the `WITH` clause.
+
+### Create an input array
+
+You can create a JavaScript UDF which accepts *N* number of inputs and creates an array that can be used as input to your Azure Machine Learning UDF.
+
+```javascript
+function createArray(vendorid, weekday, pickuphour, passenger, distance) {
+    'use strict';
+    var array = [vendorid, weekday, pickuphour, passenger, distance]
+    return array;
+}
+```
+
+Once you have added the JavaScript UDF to your job, you can invoke your Azure Machine Learning UDF using the following query:
+
+```SQL
+SELECT udf.score(
+udf.createArray(vendorid, weekday, pickuphour, passenger, distance)
+)
+INTO output
+FROM input
+```
+
+The following JSON is an example request:
+
+```JSON
+{
+    "data": [
+        ["1","Mon","12","1","5.8"],
+        ["2","Wed","10","2","10"]
+    ]
+}
+```
+
+### Create a pandas or PySpark DataFrame
+
+You can use the `WITH` clause to create a JSON serialized DataFrame that can be passed as input to your Azure Machine Learning UDF as shown below.
+
+The following query creates a DataFrame by selecting the necessary fields and uses the DataFrame as input to the Azure Machine Learning UDF.
+
+```SQL
+WITH 
+Dataframe AS (
+SELECT vendorid, weekday, pickuphour, passenger, distance
+FROM input
+)
+
+SELECT udf.score(Dataframe)
+INTO output
+FROM input
+```
+
+The following JSON is an example request from the previous query:
+
+```JSON
+{
+    "data": [{
+            "vendorid": "1",
+            "weekday": "Mon",
+            "pickuphour": "12",
+            "passenger": "1",
+            "distance": "5.8"
+        }, {
+            "vendorid": "2",
+            "weekday": "Tue",
+            "pickuphour": "10",
+            "passenger": "2",
+            "distance": "10"
+        }
+    ]
+}
+```
+
+## Optimize the performance for Azure Machine Learning UDFs
+
+When you deploy your model to Azure Kubernetes Service, you can [profile your model to determine resource utilization](../machine-learning/how-to-deploy-and-where.md#profilemodel). You can also [enable App Insights for your deployments](../machine-learning/how-to-enable-app-insights.md) to understand request rates, response times, and failure rates.
+
+If you have a scenario with high event throughput, you may need to change the following parameters in Stream Analytics to achieve optimal performance with low end-to-end latencies:
+
+1. Max batch count.
+2. Number of parallel requests per partition.
+
+### Determine the right batch size
+
+After you have deployed your web service, you send sample request with varying batch sizes starting from 50 and increasing it in order of hundreds. For example, 200, 500, 1000, 2000 and so on. You'll notice that after a certain batch size, the latency of the response increases. The point after which latency of response increases should be the max batch count for your job.
+
+### Determine the number of parallel requests per partition
+
+At optimal scaling, your Stream Analytics job should be able to send multiple parallel requests to your web service and get a response within few milliseconds. The latency of the web service's response can directly impact the latency and performance of your Stream Analytics job. If the call from your job to the web service takes a long time, you will likely see an increase in watermark delay and may also see an increase in the number of backlogged input events.
+
+To prevent such latency, ensure that your Azure Kubernetes Service (AKS) cluster has been provisioned with the [right number of nodes and replicas](../machine-learning/how-to-deploy-azure-kubernetes-service.md#using-the-cli). It's critical that your web service is highly available and returns successful responses. If your job receives a service unavailable response (503) from your web service, it will continuously retry with exponential back off. Any response other than success (200) and service unavailable (503) will cause your job to go to a failed state.
+
+## Next steps
+
+* [Tutorial: Azure Stream Analytics JavaScript user-defined functions](stream-analytics-javascript-user-defined-functions.md)
+* [Scale your Stream Analytics job with Azure Machine Learning Studio (classic) function](stream-analytics-scale-with-machine-learning-functions.md)
+

articles/stream-analytics/media/machine-learning-udf/add-azureml-udf.png

@@ -6,13 +6,17 @@ ms.author: mamccrea
 ms.reviewer: mamccrea
 ms.service: stream-analytics
 ms.topic: conceptual
-ms.date: 06/11/2019
+ms.date: 03/19/2020
 ms.custom: seodec18
 ---
 
-# Perform sentiment analysis with Azure Stream Analytics and Azure Machine Learning Studio (classic) (Preview)
+# Perform sentiment analysis with Azure Stream Analytics and Azure Machine Learning Studio (classic)
+
 This article describes how to quickly set up a simple Azure Stream Analytics job that integrates Azure Machine Learning Studio (classic). You use a Machine Learning sentiment analytics model from the Cortana Intelligence Gallery to analyze streaming text data and determine the sentiment score in real time. Using the Cortana Intelligence Suite lets you accomplish this task without worrying about the intricacies of building a sentiment analytics model.
 
+> [!TIP]
+> It is highly recommended to use [Azure Machine Learning UDFs](machine-learning-udf.md) instead of Azure Machine Learning Studio (classic) UDF for improved performance and reliability.
+
 You can apply what you learn from this article to scenarios such as these:
 
 * Analyzing real-time sentiment on streaming Twitter data.

articles/stream-analytics/media/machine-learning-udf/configure-azureml-udf.png

@@ -6,10 +6,13 @@ ms.author: jeanb
 ms.reviewer: mamccrea
 ms.service: stream-analytics
 ms.topic: conceptual
-ms.date: 06/21/2019
+ms.date: 03/16/2020
 ---
 # Scale your Stream Analytics job with Azure Machine Learning Studio (classic) functions
 
+> [!TIP]
+> It is highly recommended to use [Azure Machine Learning UDFs](machine-learning-udf.md) instead of Azure Machine Learning Studio (classic) UDF for improved performance and reliability.
+
 This article discusses how to efficiently scale Azure Stream Analytics jobs that use Azure Machine Learning functions. For information on how to scale Stream Analytics jobs in general see the article [Scaling jobs](stream-analytics-scale-jobs.md).
 
 ## What is an Azure Machine Learning function in Stream Analytics?
@@ -48,7 +51,7 @@ In general, ***B*** for batch size, ***L*** for the web service latency at batch
 
 ![Scale Stream Analytics with Machine Learning Functions Formula](./media/stream-analytics-scale-with-ml-functions/stream-analytics-scale-with-ml-functions-02.png "Scale Stream Analytics with Machine Learning Functions Formula")
 
-You can also configure the 'max concurrent calls' on the Machine Learning web service. It’s recommended to set this parameter to the maximum value (200 currently).
+You can also configure the 'max concurrent calls' on the Machine Learning web service. It's recommended to set this parameter to the maximum value (200 currently).
 
 For more information on this setting, review the [Scaling article for Machine Learning Web Services](../machine-learning/studio/scaling-webservice.md).
 
@@ -71,7 +74,7 @@ Let's examine the configuration necessary to create a Stream Analytics job, whic
 
 Using 1 SU, could this Stream Analytics job handle the traffic? The job can keep up with the input using the default batch size of 1000. The default latency of the sentiment analysis Machine Learning web service (with a default batch size of 1000) creates no more than a second of latency.
 
-The Stream Analytics job’s **overall** or end-to-end latency would typically be a few seconds. Take a more detailed look into this Stream Analytics job, *especially* the Machine Learning function calls. With a batch size of 1000, a throughput of 10,000 events takes about 10 requests to the web service. Even with one SU, there are enough concurrent connections to accommodate this input traffic.
+The Stream Analytics job's **overall** or end-to-end latency would typically be a few seconds. Take a more detailed look into this Stream Analytics job, *especially* the Machine Learning function calls. With a batch size of 1000, a throughput of 10,000 events takes about 10 requests to the web service. Even with one SU, there are enough concurrent connections to accommodate this input traffic.
 
 If the input event rate increases by 100x, then the Stream Analytics job needs to process 1,000,000 tweets per second. There are two options to accomplish the increased scale:
 
@@ -109,7 +112,7 @@ Below is a table for the throughput of the Stream Analytics job for different SU
 
 By now, you should already have a good understanding of how Machine Learning functions in Stream Analytics work. You likely also understand that Stream Analytics jobs "pull" data from data sources and each "pull" returns a batch of events for the Stream Analytics job to process. How does this pull model impact the Machine Learning web service requests?
 
-Normally, the batch size we set for Machine Learning functions won’t exactly be divisible by the number of events returned by each Stream Analytics job "pull". When this occurs, the Machine Learning web service is called with "partial" batches. Using partial batches avoids incurring additional job latency overhead in coalescing events from pull to pull.
+Normally, the batch size we set for Machine Learning functions won't exactly be divisible by the number of events returned by each Stream Analytics job "pull". When this occurs, the Machine Learning web service is called with "partial" batches. Using partial batches avoids incurring additional job latency overhead in coalescing events from pull to pull.
 
 ## New function-related monitoring metrics
 In the Monitor area of a Stream Analytics job, three additional function-related metrics have been added. They are **FUNCTION REQUESTS**, **FUNCTION EVENTS** and **FAILED FUNCTION REQUESTS**, as shown in the graphic below.