momo - adding section on model performance monitoring

Author: msakande

articles/machine-learning/how-to-monitor-model-performance.md

@@ -478,6 +478,223 @@ created_monitor = poller.result()
 
 ---
 
+## Set up model performance monitoring
+
+Azure Machine Learning model monitoring enables you to track the objective performance of your models in production by calculating model performance metrics. These model performance metrics include accuracy (for classification models) and root mean squared error (RMSE) for (regression models).
+
+Before you can configure your model performance signal, you need to satisfy the following requirements:
+
+* Have production model output data (your model's predictions) with a unique ID for each row.
+* Have ground truth data (or actuals) with a unique ID for each row. This data will be joined with production data.
+* (Optional) Have a prejoined dataset with model outputs and ground truth data.
+
+The key requirement for enabling model performance monitoring is that you already collected ground truth data. Since ground truth data is encountered at the application level, it's your responsibility to collect it as it becomes available. You should also maintain a data asset in Azure Machine Learning with this ground truth data.
+
+To illustrate, suppose you have a deployed model to predict if a credit card transaction is fraudulent or not fraudulent. As you use this model in production, you can collect the model output data with the [model data collector](how-to-collect-production-data.md). Ground truth data becomes available when a credit card holder specifies whether or not the transaction was fraudulent or not. This `is_fraud` ground truth should be collected at the application level and maintained within an Azure Machine Learning data asset that the model performance monitoring signal can use.
+
+# [Azure CLI](#tab/azure-cli)
+
+Once you've satisfied the previous requirements, you can set up model monitoring with the following CLI command and YAML definition:
+
+```azurecli
+az ml schedule create -f ./model-performance-monitoring.yaml
+```
+
+The following YAML contains the definition for model monitoring with production inference data that you've collected.
+
+```YAML 
+$schema:  http://azureml/sdk-2-0/Schedule.json
+name: model_performance_monitoring
+display_name: Credit card fraud model performance
+description: Credit card fraud model performance
+
+trigger:
+  type: recurrence
+  frequency: day
+  interval: 7 
+  schedule: 
+    hours: 10
+    minutes: 15
+  
+create_monitor:
+  compute: 
+    instance_type: standard_e8s_v3
+    runtime_version: "3.3"
+  monitoring_target:
+    ml_task: classification
+    endpoint_deployment_id: azureml:loan-approval-endpoint:loan-approval-deployment
+
+  monitoring_signals:
+    fraud_detection_model_performance: 
+      type: model_performance 
+      production_data:
+        data_column_names:
+          prediction: is_fraud
+          correlation_id: correlation_id
+      reference_data:
+        input_data:
+          path: azureml:my_model_ground_truth_data:1
+          type: mltable
+        data_column_names:
+          actual: is_fraud
+          correlation_id: correlation_id
+        data_context: actuals
+      alert_enabled: true
+      metric_thresholds: 
+        tabular_classification:
+          accuracy: 0.95
+          precision: 0.8
+  alert_notification: 
+      emails: 
+        - [email protected]
+```
+
+# [Python SDK](#tab/python)
+
+Once you've satisfied the previous requirements, you can set up model monitoring using the following Python code:
+
+```python
+from azure.identity import InteractiveBrowserCredential
+from azure.ai.ml import Input, MLClient
+from azure.ai.ml.constants import (
+    MonitorFeatureType,
+    MonitorMetricName,
+    MonitorDatasetContext
+)
+from azure.ai.ml.entities import (
+    AlertNotification,
+    DataDriftSignal,
+    DataQualitySignal,
+    DataDriftMetricThreshold,
+    DataQualityMetricThreshold,
+    NumericalDriftMetrics,
+    CategoricalDriftMetrics,
+    DataQualityMetricsNumerical,
+    DataQualityMetricsCategorical,
+    MonitorFeatureFilter,
+    MonitorInputData,
+    MonitoringTarget,
+    MonitorDefinition,
+    MonitorSchedule,
+    RecurrencePattern,
+    RecurrenceTrigger,
+    ServerlessSparkCompute,
+    ReferenceData,
+    ProductionData
+)
+
+# get a handle to the workspace
+ml_client = MLClient(
+   InteractiveBrowserCredential(),
+   subscription_id,
+   resource_group,
+   workspace
+)
+
+spark_compute = ServerlessSparkCompute(
+    instance_type="standard_e4s_v3",
+    runtime_version="3.2"
+)
+
+#define target dataset (production dataset)
+production_data = ProductionData(
+    input_data=Input(
+        type="uri_folder",
+        path="azureml:my_model_production_data:1"
+    ),
+    data_context=MonitorDatasetContext.MODEL_INPUTS,
+    pre_processing_component="azureml:production_data_preprocessing:1"
+)
+
+
+# training data to be used as baseline dataset
+reference_data_training = ReferenceData(
+    input_data=Input(
+        type="mltable",
+        path="azureml:my_model_training_data:1"
+    ),
+    data_context=MonitorDatasetContext.TRAINING
+)
+
+# create an advanced data drift signal
+features = MonitorFeatureFilter(top_n_feature_importance=20)
+metric_thresholds = DataDriftMetricThreshold(
+    numerical=NumericalDriftMetrics(
+        jensen_shannon_distance=0.01
+    ),
+    categorical=CategoricalDriftMetrics(
+        pearsons_chi_squared_test=0.02
+    )
+)
+
+advanced_data_drift = DataDriftSignal(
+    production_data=production_data,
+    reference_data=reference_data_training,
+    features=features,
+    metric_thresholds=metric_thresholds
+)
+
+
+# create an advanced data quality signal
+features = ['feature_A', 'feature_B', 'feature_C']
+metric_thresholds = DataQualityMetricThreshold(
+    numerical=DataQualityMetricsNumerical(
+        null_value_rate=0.01
+    ),
+    categorical=DataQualityMetricsCategorical(
+        out_of_bounds_rate=0.02
+    )
+)
+
+advanced_data_quality = DataQualitySignal(
+    production_data=production_data,
+    reference_data=reference_data_training,
+    features=features,
+    metric_thresholds=metric_thresholds,
+    alert_enabled="False"
+)
+
+# put all monitoring signals in a dictionary
+monitoring_signals = {
+    'data_drift_advanced': advanced_data_drift,
+    'data_quality_advanced': advanced_data_quality
+}
+
+# create alert notification object
+alert_notification = AlertNotification(
+    emails=['[email protected]', '[email protected]']
+)
+
+# Finally monitor definition
+monitor_definition = MonitorDefinition(
+    compute=spark_compute,
+    monitoring_signals=monitoring_signals,
+    alert_notification=alert_notification
+)
+
+recurrence_trigger = RecurrenceTrigger(
+    frequency="day",
+    interval=1,
+    schedule=RecurrencePattern(hours=3, minutes=15)
+)
+
+model_monitor = MonitorSchedule(
+    name="fraud_detection_model_monitoring_advanced",
+    trigger=recurrence_trigger,
+    create_monitor=monitor_definition
+)
+
+poller = ml_client.schedules.begin_create_or_update(model_monitor)
+created_monitor = poller.result()
+
+```
+
+# [Studio](#tab/azure-studio)
+
+The studio currently doesn't support model performance monitoring.
+
+---
+
 ## Set up model monitoring by bringing your own production data to Azure Machine Learning
 
 You can also set up model monitoring for models deployed to Azure Machine Learning batch endpoints or deployed outside of Azure Machine Learning. If you have production data but no deployment, you can use the data to perform continuous model monitoring. To monitor these models, you must meet the following requirements: