changes to how to docs

Author: ahughes-msft

articles/machine-learning/how-to-collect-production-data.md

@@ -89,7 +89,7 @@ To begin, add custom logging code to your scoring script (`score.py`). For custo
     > [!NOTE]
     > Currently, the `collect()` API logs only pandas DataFrames. If the data is not in a DataFrame when passed to `collect()`, it won't get logged to storage and an error will be reported.
 
-The following code is an example of a full scoring script (`score.py`) that uses the custom logging Python SDK. In this example, a third `Collector` called `inputs_outputs_collector` logs a joined DataFrame of the `model_inputs` and the `model_outputs`. This joined DataFrame enables more monitoring signals such as feature attribution drift. If you're not interested in these monitoring signals, you can remove this `Collector`.
+The following code is an example of a full scoring script (`score.py`) that uses the custom logging Python SDK.
 
 ```python
 import pandas as pd
@@ -132,6 +132,86 @@ def predict(input_df):
   return output_df
 ```
 
+#### Update your scoring script to log custom unique IDs
+
+In addition to logging pandas DataFrames directly within your scoring script, you can log data with unique IDs of your choice. These IDs can come from your application, an external system, or can be generated by you. If you do not provide a custom ID, as detailed in this section, the Data collector will autogenerate a unique `correlationid` to help you correlate your model's inputs and outputs later. If you supply a custom ID, the `correlationid` field in the logged data will contain the value of your supplied custom ID.
+
+1. In addition to the steps above, import the `azureml.ai.monitoring.context` package by adding the following line to your scoring script:
+
+```python
+from azureml.ai.monitoring.context import BasicCorrelationContext
+```
+
+1. In your scoring script, instantiate a `BasicCorrelationContext` object and pass in the `id` you wish to log for that row. We recommend that this `id` be a unique ID from your system, so you can uniquely identify each logged row from your Blob storage. Pass this object into your `collect()` API call as a parameter:
+
+```python
+  # create a context with a custom unique id
+  artificial_context = BasicCorrelationContext(id='test')
+
+  # collect inputs data, store correlation_context
+  context = inputs_collector.collect(input_df, artificial_context)
+```
+
+1. Ensure that you pass into the context into your `outputs_collector` so that your model inputs and outputs have the same unique ID logged with them, and they can be easily correlated later:
+
+```python
+  # collect outputs data, pass in context so inputs and outputs data can be correlated later
+  outputs_collector.collect(output_df, context)
+```
+
+A comprehensive example is detailed below:
+
+```python
+import pandas as pd
+import json
+from azureml.ai.monitoring import Collector
+from azureml.ai.monitoring.context import BasicCorrelationContext
+
+def init():
+  global inputs_collector, outputs_collector, inputs_outputs_collector
+
+  # instantiate collectors with appropriate names, make sure align with deployment spec
+  inputs_collector = Collector(name='model_inputs')                    
+  outputs_collector = Collector(name='model_outputs')
+
+def run(data): 
+  # json data: { "data" : {  "col1": [1,2,3], "col2": [2,3,4] } }
+  pdf_data = preprocess(json.loads(data))
+  
+  # tabular data: {  "col1": [1,2,3], "col2": [2,3,4] }
+  input_df = pd.DataFrame(pdf_data)
+
+  # create a context with a custom unique id
+  artificial_context = BasicCorrelationContext(id='test')
+
+  # collect inputs data, store correlation_context
+  context = inputs_collector.collect(input_df, artificial_context)
+
+  # perform scoring with pandas Dataframe, return value is also pandas Dataframe
+  output_df = predict(input_df) 
+
+  # collect outputs data, pass in context so inputs and outputs data can be correlated later
+  outputs_collector.collect(output_df, context)
+  
+  return output_df.to_dict()
+  
+def preprocess(json_data):
+  # preprocess the payload to ensure it can be converted to pandas DataFrame
+  return json_data["data"]
+
+def predict(input_df):
+  # process input and return with outputs
+  ...
+  
+  return output_df
+```
+
+#### Collect data for model performance monitoring
+
+If you are interested in using your collected data for model performance monitoring, it is important that each logged row has a unique `correlationid` which can be used to correlate the data with ground truth data, when it becomes available. The data collector will autogenerate a unique `correlationid` for each logged row, and include it in the `correlationid` field in the JSON object. Please see [store collected data in a blob](#store-collected-data-in-a-blob) for comprehensive details on the JSON schema.
+
+If you are interested in using your own unique ID to log with your production data, it is recommended that you log it as a separate column in your `pandas DataFrame`. The reason for this is because [the data collector will batch requests](#data-collector-batching) which fall within close proximity of one another. If you need the `correlationid` to be readily available downstream for integration with ground truth data, having it logged as a separate column is recommended.
+
 ### Update your dependencies
 
 Before you can create your deployment with the updated scoring script, you need to create your environment with the base image `mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04` and the appropriate conda dependencies. Thereafter, you can build the environment using the specification in the following YAML.
@@ -301,6 +381,28 @@ With collected binary data, we show the raw file directly, with `instance_id` as
 }
 ```
 
+#### Data collector batching
+
+The data collector will batch requests together into the same JSON object if they are sent within a short duration of each other. For example, if you run a script to send sample data to your endpoint, and the deployment has data collection enabled, some of the requests may get batched together, depending on the interval of time between them. If you are using data collection to use with [Azure Machine Learning model monitoring](#concept-model-monitoring.md) this behavior is handled appropriately and each request is handled as independent by the model monitoring service. However, if you expect each logged row of data have its own unique `correlationid`, you can include the `correlationid` as a column in the `pandas DataFrame` you are logging with the data collector. Information on how to do this can be found in [data collection for model performance monitoring][#collect-data-for-model-performance-monitoring].
+
+Here is an example of two logged requests being batched together:
+
+```json
+{"specversion":"1.0",
+"id":"720b8867-54a2-4876-80eb-1fd6a8975770",
+"source":"/subscriptions/79a1ba0c-35bb-436b-bff2-3074d5ff1f89/resourceGroups/rg-bozhlinmomoignite/providers/Microsoft.MachineLearningServices/workspaces/momo-demo-ws/onlineEndpoints/credit-default-mdc-testing-4/deployments/main2",
+"type":"azureml.inference.model_inputs",
+"datacontenttype":"application/json",
+"time":"2024-03-05T18:16:25Z",
+"data":[{"LIMIT_BAL":502970,"AGE":54,"BILL_AMT1":308068,"BILL_AMT2":381402,"BILL_AMT3":442625,"BILL_AMT4":320399,"BILL_AMT5":322616,"BILL_AMT6":397534,"PAY_AMT1":17987,"PAY_AMT2":78764,"PAY_AMT3":26067,"PAY_AMT4":24102,"PAY_AMT5":-1155,"PAY_AMT6":2154,"SEX":2,"EDUCATION":2,"MARRIAGE":2,"PAY_0":0,"PAY_2":0,"PAY_3":0,"PAY_4":0,"PAY_5":0,"PAY_6":0},{"LIMIT_BAL":293458,"AGE":35,"BILL_AMT1":74131,"BILL_AMT2":-71014,"BILL_AMT3":59284,"BILL_AMT4":98926,"BILL_AMT5":110,"BILL_AMT6":1033,"PAY_AMT1":-3926,"PAY_AMT2":-12729,"PAY_AMT3":17405,"PAY_AMT4":25110,"PAY_AMT5":7051,"PAY_AMT6":1623,"SEX":1,"EDUCATION":3,"MARRIAGE":2,"PAY_0":-2,"PAY_2":-2,"PAY_3":-2,"PAY_4":-2,"PAY_5":-1,"PAY_6":-1}],
+"contentrange":"bytes 0-6794/6795",
+"correlationid":"test",
+"xrequestid":"test",
+"modelversion":"default",
+"collectdatatype":"pandas.core.frame.DataFrame",
+"agent":"azureml-ai-monitoring/0.1.0b4"}
+```
+
 #### View the data in the studio UI
 
 To view the collected data in Blob Storage from the studio UI:

articles/machine-learning/how-to-monitor-model-performance.md

@@ -451,14 +451,22 @@ You must satisfy the following requirements for you to configure your model perf
 
     > [!NOTE]
     >
-    > For Azure Machine Learning model performance monitoring, we recommend that you log your unique ID in its own column, using the [Azure Machine Learning data collector](how-to-collect-production-data.md). This practice will ensure that each collected row is guaranteed to have a unique ID.
+    > For Azure Machine Learning model performance monitoring, we recommend that you log your unique ID in its own column, using the [Azure Machine Learning data collector](how-to-collect-production-data.md).
 
 * Have ground truth data (actuals) with a unique ID for each row. The unique ID for a given row should match the unique ID for the model outputs for that particular inference request. This unique ID is used to join your ground truth dataset with the model outputs.
 
   Without having ground truth data, you can't perform model performance monitoring. Since ground truth data is encountered at the application level, it's your responsibility to collect it as it becomes available. You should also maintain a data asset in Azure Machine Learning that contains this ground truth data.
 
 * (Optional) Have a pre-joined tabular dataset with model outputs and ground truth data already joined together.
 
+### Monitoring model performance requirements when using data collector
+
+If you use the [Azure Machine Learning data collector](concept-data-collection.md) to collect production inference data and do not supply your own unique ID for each row as a separate column, a `correlationid` will be autogenerated for you and included in the logged JSON object. However, the data collector will [batch rows](how-to-collect-production-data.md#data-collector-batching) which are sent within close proximity to each other. Batched rows will fall within the same JSON object and will thus have the same `correlationid`.
+
+In order to differentiate between the rows in the same JSON object, Azure Machine Learning model performance monitoring uses indexing to determine the first, second, third, and so on, row in the JSON object. For example, if three rows are batched together, and the `correlationid` is `test`, row 1 will have an id of `test_0`, row 2 will have an id of `test_1`, and row 3 will have an id `test_2`. To ensure that your ground truth dataset contains unique IDs which match to the collected production inference model outputs, ensure that you index each `correlationid` appropriately. If your logged JSON object only has one row, then the `correlationid` would be `correlationid_0`.
+
+To avoid using this indexing, we recommend that you log your unique ID in its own column within the `pandas DataFrame`, using the [Azure Machine Learning data collector](how-to-collect-production-data.md). Then, in your model monitoring configuration, you specify the name of this column to join your model output data with your ground data. As long as the IDs for each row in both datasets are the same, Azure Machine Learning model monitoring can perform model performance monitoring.
+
 ### Example workflow for monitoring model performance
 
 To understand the concepts associated with model performance monitoring, consider this example workflow. Suppose you're deploying a model to predict whether credit card transactions are fraudulent or not, you can follow these steps to monitor the model's performance:
@@ -532,25 +540,17 @@ create_monitor:
 Once you've satisfied the [prerequisites for model performance monitoring](#more-prerequisites-for-model-performance-monitoring), you can set up model monitoring with the following Python code:
 
 ```python
-from azure.identity import InteractiveBrowserCredential
+from azure.identity import DefaultAzureCredential
 from azure.ai.ml import Input, MLClient
 from azure.ai.ml.constants import (
-    MonitorFeatureType,
-    MonitorMetricName,
-    MonitorDatasetContext
+    MonitorDatasetContext,
 )
 from azure.ai.ml.entities import (
     AlertNotification,
-    DataDriftSignal,
-    DataQualitySignal,
-    DataDriftMetricThreshold,
-    DataQualityMetricThreshold,
-    NumericalDriftMetrics,
-    CategoricalDriftMetrics,
-    DataQualityMetricsNumerical,
-    DataQualityMetricsCategorical,
-    MonitorFeatureFilter,
-    MonitorInputData,
+    BaselineDataRange,
+    ModelPerformanceMetricThreshold,
+    ModelPerformanceSignal,
+    ModelPerformanceClassificationThresholds,
     MonitoringTarget,
     MonitorDefinition,
     MonitorSchedule,
@@ -563,79 +563,70 @@ from azure.ai.ml.entities import (
 
 # get a handle to the workspace
 ml_client = MLClient(
-   InteractiveBrowserCredential(),
-   subscription_id,
-   resource_group,
-   workspace
+    DefaultAzureCredential(),
+    subscription_id="79a1ba0c-35bb-436b-bff2-3074d5ff1f89",
+    resource_group_name="rg-bozhlinmomoignite",
+    workspace_name="momo-demo-ws",
 )
 
+# create your compute
 spark_compute = ServerlessSparkCompute(
     instance_type="standard_e4s_v3",
-    runtime_version="3.2"
+    runtime_version="3.3"
 )
 
-#define target dataset (production dataset)
+# reference your azureml endpoint and deployment
+monitoring_target = MonitoringTarget(
+    ml_task="classification",
+)
+
+# MDC-generated production data with data column names 
 production_data = ProductionData(
     input_data=Input(
         type="uri_folder",
-        path="azureml:my_model_production_data:1"
+        path="azureml:credit-default-main-model_outputs:1"
     ),
-    data_context=MonitorDatasetContext.MODEL_INPUTS,
-    pre_processing_component="azureml:production_data_preprocessing:1"
+    data_column_names={
+        "target_column": "DEFAULT_NEXT_MONTH",
+        "join_column": "correlationid"
+    },
+    data_window=BaselineDataRange(
+        lookback_window_offset="P0D",
+        lookback_window_size="P10D",
+    )
 )
 
-
-# training data to be used as baseline dataset
-reference_data_training = ReferenceData(
+# ground truth reference data 
+reference_data_ground_truth = ReferenceData(
     input_data=Input(
         type="mltable",
-        path="azureml:my_model_training_data:1"
+        path="azureml:credit-ground-truth:1"
     ),
-    data_context=MonitorDatasetContext.TRAINING
+    data_column_names={
+        "target_column": "ground_truth",
+        "join_column": "correlationid"
+    },
+    data_context=MonitorDatasetContext.GROUND_TRUTH_DATA,
 )
 
-# create an advanced data drift signal
-features = MonitorFeatureFilter(top_n_feature_importance=20)
-metric_thresholds = DataDriftMetricThreshold(
-    numerical=NumericalDriftMetrics(
-        jensen_shannon_distance=0.01
+# create the model performance signal
+metric_thresholds = ModelPerformanceMetricThreshold(
+    classification=ModelPerformanceClassificationThresholds(
+        accuracy=0.50,
+        precision=0.50,
+        recall=0.50
     ),
-    categorical=CategoricalDriftMetrics(
-        pearsons_chi_squared_test=0.02
-    )
 )
 
-advanced_data_drift = DataDriftSignal(
+model_performance = ModelPerformanceSignal(
     production_data=production_data,
-    reference_data=reference_data_training,
-    features=features,
+    reference_data=reference_data_ground_truth,
     metric_thresholds=metric_thresholds
 )
 
-
-# create an advanced data quality signal
-features = ['feature_A', 'feature_B', 'feature_C']
-metric_thresholds = DataQualityMetricThreshold(
-    numerical=DataQualityMetricsNumerical(
-        null_value_rate=0.01
-    ),
-    categorical=DataQualityMetricsCategorical(
-        out_of_bounds_rate=0.02
-    )
-)
-
-advanced_data_quality = DataQualitySignal(
-    production_data=production_data,
-    reference_data=reference_data_training,
-    features=features,
-    metric_thresholds=metric_thresholds,
-    alert_enabled="False"
-)
-
 # put all monitoring signals in a dictionary
 monitoring_signals = {
-    'data_drift_advanced': advanced_data_drift,
-    'data_quality_advanced': advanced_data_quality
+    'model_performance':model_performance,
 }
 
 # create alert notification object
@@ -646,6 +637,7 @@ alert_notification = AlertNotification(
 # Finally monitor definition
 monitor_definition = MonitorDefinition(
     compute=spark_compute,
+    monitoring_target=monitoring_target,
     monitoring_signals=monitoring_signals,
     alert_notification=alert_notification
 )
@@ -657,14 +649,13 @@ recurrence_trigger = RecurrenceTrigger(
 )
 
 model_monitor = MonitorSchedule(
-    name="fraud_detection_model_monitoring_advanced",
+    name="credit_default_model_performance",
     trigger=recurrence_trigger,
     create_monitor=monitor_definition
 )
 
 poller = ml_client.schedules.begin_create_or_update(model_monitor)
 created_monitor = poller.result()
-
 ```
 
 # [Studio](#tab/azure-studio)