Merge pull request #111112 from buchananwp/patch-2

Monitoring - Refine Feature Attribution Drift setup

Author: Court72

articles/machine-learning/concept-model-monitoring.md

@@ -49,8 +49,11 @@ Azure Machine Learning model monitoring (preview) supports the following list of
 | Data drift | Data drift tracks changes in the distribution of a model's input data by comparing it to the model's training data or recent past production data. | Jensen-Shannon Distance, Population Stability Index, Normalized Wasserstein Distance, Two-Sample Kolmogorov-Smirnov Test, Pearson's Chi-Squared Test | Classification (tabular data), Regression (tabular data) | Production data - model inputs | Recent past production data or training data |
 | Prediction drift | Prediction drift tracks changes in the distribution of a model's prediction outputs by comparing it to validation or test labeled data or recent past production data. | Jensen-Shannon Distance, Population Stability Index, Normalized Wasserstein Distance, Chebyshev Distance, Two-Sample Kolmogorov-Smirnov Test, Pearson's Chi-Squared Test | Classification (tabular data), Regression (tabular data) | Production data - model outputs | Recent past production data or validation data |
 | Data quality | Data quality tracks the data integrity of a model's input by comparing it to the model's training data or recent past production data. The data quality checks include checking for null values, type mismatch, or out-of-bounds of values. | Null value rate, data type error rate, out-of-bounds rate | Classification (tabular data), Regression (tabular data) | production data - model inputs | Recent past production data or training data |
-| Feature attribution drift | Feature attribution drift tracks the importance or contributions of features to prediction outputs in production by comparing it to feature importance at training time | Normalized discounted cumulative gain | Classification (tabular data), Regression (tabular data) | Production data | Training data |
+| Feature attribution drift | Feature attribution drift tracks the importance or contributions of features to prediction outputs in production by comparing it to feature importance at training time | Normalized discounted cumulative gain | Classification (tabular data), Regression (tabular data) | Production data - model inputs & outputs (*see the following note*) | Training data (required) |
 
+> [!NOTE]
+> For 'feature attribution drift' signal (during Preview), the user must create a custom data asset of type 'uri_folder' that contains joined inputs and outputs (Model Data Collector can be leveraged). Additionally, 'target_column_name' is also a required field, which specifies the prediction column in your training dataset. 
+  
 ## How model monitoring works in Azure Machine Learning
 
 Azure Machine Learning acquires monitoring signals by performing statistical computations on production inference data and reference data. This reference data can include the model's training data or validation data, while the production inference data refers to the model's input and output data collected in production.
@@ -84,4 +87,4 @@ Each machine learning model and its use cases are unique. Therefore, model monit
 
 - [Perform continuous model monitoring in Azure Machine Learning](how-to-monitor-model-performance.md)
 - [Model data collection](concept-data-collection.md)
-- [Collect production inference data](how-to-collect-production-data.md)
+- [Collect production inference data](how-to-collect-production-data.md)

articles/machine-learning/how-to-collect-production-data.md

@@ -74,6 +74,7 @@ First, you'll need to add custom logging code to your scoring script (`score.py`
     global inputs_collector, outputs_collector
     inputs_collector = Collector(name='model_inputs')          
     outputs_collector = Collector(name='model_outputs')
+    inputs_outputs_collector = Collector(name='model_inputs_outputs')
     ```
 
     By default, Azure Machine Learning raises an exception if there's a failure during data collection. Optionally, you can use the `on_error` parameter to specify a function to run if logging failure happens. For instance, using the `on_error` parameter in the following code, Azure Machine Learning logs the error rather than throwing an exception:
@@ -106,6 +107,7 @@ def init():
   # instantiate collectors with appropriate names, make sure align with deployment spec
   inputs_collector = Collector(name='model_inputs')                    
   outputs_collector = Collector(name='model_outputs')
+  inputs_outputs_collector = Collector(name='model_inputs_outputs') #note: this is used to enable Feature Attribution Drift
 
 def run(data): 
   # json data: { "data" : {  "col1": [1,2,3], "col2": [2,3,4] } }
@@ -122,6 +124,13 @@ def run(data):
 
   # collect outputs data, pass in correlation_context so inputs and outputs data can be correlated later
   outputs_collector.collect(output_df, context)
+
+  # create a dataframe with inputs/outputs joined - this creates a URI folder (not mltable) 
+  # input_output_df = input_df.merge(output_df, context)
+  input_output_df = input_df.join(output_df)
+
+  # collect both your inputs and output  
+  inputs_outputs_collector.collect(input_output_df, context)
 
   return output_df.to_dict()
 

articles/machine-learning/how-to-monitor-model-performance.md

@@ -66,7 +66,7 @@ Before following the steps in this article, make sure you have the following pre
 >
 > Model monitoring jobs are scheduled to run on serverless Spark compute pool with `Standard_E4s_v3` VM instance type support only. More VM instance type support will come in the future roadmap.
 
-## Set up out-of-box model monitoring
+## Set up out-of-the-box model monitoring
 
 If you deploy your model to production in an Azure Machine Learning online endpoint, Azure Machine Learning collects production inference data automatically and uses it for continuous monitoring.
 
@@ -79,6 +79,9 @@ You can use Azure CLI, the Python SDK, or Azure Machine Learning studio for out-
   * smart defaults for metrics and thresholds.
 * A monitoring job is scheduled to run daily at 3:15am (for this example) to acquire monitoring signals and evaluate each metric result against its corresponding threshold. By default, when any threshold is exceeded, an alert email is sent to the user who set up the monitoring.
 
+## Configure feature importance
+
+For feature importance to be enabled with any of your signals (such as data drift or data quality,) you need to provide both the 'baseline_dataset' (typically training) dataset and 'target_column_name' fields. 
 
 # [Azure CLI](#tab/azure-cli)
 
@@ -88,7 +91,7 @@ Azure Machine Learning model monitoring uses `az ml schedule` for model monitori
 az ml schedule create -f ./out-of-box-monitoring.yaml
 ```
 
-The following YAML contains the definition for out-of-box model monitoring.
+The following YAML contains the definition for out-of-the-box model monitoring.
 
 ```yaml
 # out-of-box-monitoring.yaml
@@ -117,7 +120,7 @@ create_monitor:
 
 # [Python](#tab/python)
 
-You can use the following code to set up out-of-box model monitoring:
+You can use the following code to set up out-of-the-box model monitoring:
 
 ```python
 
@@ -269,18 +272,18 @@ create_monitor:
          dataset:
             input_dataset:
                path: azureml:my_model_production_data:1
-               type: mltable
-            dataset_context: model_inputs
+               type: uri_folder
+            dataset_context: model_inputs_outputs
       baseline_dataset:
         input_dataset:
           path: azureml:my_model_training_data:1
           type: mltable
-        dataset_context: model_inputs
+        dataset_context: training
         target_column_name: fraud_detected
       model_type: classification
       # if no metric_thresholds defined, use the default metric_thresholds
       metric_thresholds:
-         threshold: 0.05
+         threshold: 0.9
 
   alert_notification:
     emails:
@@ -384,10 +387,10 @@ advanced_data_quality = DataQualitySignal(
 monitor_target_data = TargetDataset(
     dataset=MonitorInputData(
         input_dataset=Input(
-            type="mltable",
-            path="azureml:my_model_production_data:1"
+            type="uri_folder",
+            path="azureml:endpoint_name-deployment_name-model_inputs_outputs:1"
         ),
-        dataset_context=MonitorDatasetContext.MODEL_INPUTS,
+        dataset_context=MonitorDatasetContext.MODEL_INPUTS_OUTPUTS,
     )
 )
 monitor_baseline_data = MonitorInputData(
@@ -398,7 +401,7 @@ monitor_baseline_data = MonitorInputData(
     target_column_name="fraud_detected",
     dataset_context=MonitorDatasetContext.TRAINING,
 )
-metric_thresholds = FeatureAttributionDriftMetricThreshold(threshold=0.05)
+metric_thresholds = FeatureAttributionDriftMetricThreshold(threshold=0.9)
 
 feature_attribution_drift = FeatureAttributionDriftSignal(
     target_dataset=monitor_target_data,
@@ -447,7 +450,7 @@ created_monitor = poller.result()
 
 # [Studio](#tab/azure-studio)
 
-1. Complete the entires on the basic settings page as described in the [Set up out-of-box model monitoring](#set-up-out-of-box-model-monitoring) section.
+1. Complete the entires on the basic settings page as described in the [Set up out-of-box model monitoring](#set-up-out-of-the-box-model-monitoring) section.
 1. Select **More options** to open the advanced setup wizard.
 
 1. In the "Configure dataset" section, add a dataset to be used as the comparison baseline. We recommend using the model training data as the comparison baseline for data drift and data quality, and using the model validation data as the comparison baseline for prediction drift.
@@ -471,18 +474,22 @@ created_monitor = poller.result()
 
 1. Select **Add** to add another signal.
 1. In the "Add Signal" screen, select the **Feature Attribution Drift** panel.
-1. Enter a name for Feature Attribution Drift signal.
+1. Enter a name for Feature Attribution Drift signal. Feature attribution drift currently requires a few additional steps:
+1. Configure your data assets for Feature Attribution Drift
+   1. In your model creation wizard, add your custom data asset from your [custom data collection](how-to-collect-production-data.md) called 'model inputs and outputs' which combines your joined model inputs and data assets as a separate data context. 
+   
+      :::image type="content" source="media/how-to-monitor-models/feature-attribution-drift-inputs-outputs.png" alt-text="Screenshot showing how to configure a custom data asset with inputs and outputs joined." lightbox="media/how-to-monitor-models/feature-attribution-drift-inputs-outputs.png":::
+      
+   1. Specify your training reference dataset that will be used in the feature attribution drift component, and select your 'target column name' field, which is required to enable feature importance. 
+   1. Confirm your parameters are correct
 1. Adjust the data window size according to your business case.
-1. Select the training data as the baseline dataset. 
-1. Select the target column name.
 1. Adjust the threshold according to your need.
 1. Select **Save** to return to the "Select monitoring signals" section.
 1. If you're done with editing or adding signals, select **Next**.
 
    :::image type="content" source="media/how-to-monitor-models/model-monitoring-advanced-config-add-signal.png" alt-text="Screenshot showing settings for adding signals." lightbox="media/how-to-monitor-models/model-monitoring-advanced-config-add-signal.png":::
 
 1. In the "Notification" screen, enable alert notification for each signal.
-1. (Optional) Enable "Azure Monitor" for all metrics to be sent to Azure Monitor.
 1. Select **Next**.
 
    :::image type="content" source="media/how-to-monitor-models/model-monitoring-advanced-config-notification.png" alt-text="Screenshot of settings on the notification screen." lightbox="media/how-to-monitor-models/model-monitoring-advanced-config-notification.png":::