Skip to content

Commit 837f016

Browse files
committed
Edit content
1 parent aa69b13 commit 837f016

File tree

1 file changed

+111
-76
lines changed

1 file changed

+111
-76
lines changed

articles/machine-learning/how-to-monitor-model-performance.md

Lines changed: 111 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ In the Azure CLI, you use `az ml schedule` to schedule a monitoring job.
8686

8787
1. Create a monitoring definition in a YAML file. For a sample out-of-box definition, see the following YAML code, which is also available in the [azureml-examples repository](https://github.com/Azure/azureml-examples/blob/main/cli/monitoring/out-of-box-monitoring.yaml).
8888

89-
Before you use this definition, adjust the values to fit your environment. For `endpoint_deployment_id`, use a value in the format `azureml:<endpoint-name>:<model-name>`.
89+
Before you use this definition, adjust the values to fit your environment. For `endpoint_deployment_id`, use a value in the format `azureml:<endpoint-name>:<deployment-name>`.
9090

9191
:::code language="yaml" source="~/azureml-examples-main/cli/monitoring/out-of-box-monitoring.yaml":::
9292

@@ -98,7 +98,20 @@ In the Azure CLI, you use `az ml schedule` to schedule a monitoring job.
9898

9999
# [Python SDK](#tab/python)
100100

101-
To set up the out-of-box model monitoring, use code that's similar to the following sample. For `endpoint_deployment_id`, use a value in the format `azureml:<endpoint-name>:<model-name>`.
101+
To set up out-of-box model monitoring, use code that's similar to the following sample. Replace the following placeholders with appropriate values:
102+
103+
| Placeholder | Description | Example |
104+
| --- | --- | --- |
105+
| <subscription_ID> | The ID of your subscription | aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e |
106+
| <resource-group-name> | The name of the resource group that contains your workspace | my-resource-group |
107+
| <workspace_name> | The name of your workspace | my-workspace |
108+
| <endpoint-name> | The name of the endpoint to monitor | credit-default |
109+
| <deployment-name> | The name of the deployment to monitor | main |
110+
| <email-address1> and <email-address2> | Email addresses to use for notifications | `[email protected]` |
111+
| <frequency-unit> | The monitoring frequency unit, such as "minute," "hour," "day," "week," or "month" | day |
112+
| <interval> | The interval between jobs, such as 1 or 2 days or weeks | 1 |
113+
| <start-hour> | The hour to start monitoring, on a 24-hour clock | 3 |
114+
| <start-minutes> | The minutes after the specified hour to start monitoring | 15 |
102115

103116
```python
104117
from azure.identity import DefaultAzureCredential
@@ -116,9 +129,9 @@ from azure.ai.ml.entities import (
116129
# Get a handle to the workspace.
117130
ml_client = MLClient(
118131
DefaultAzureCredential(),
119-
subscription_id="subscription_id",
120-
resource_group_name="resource_group_name",
121-
workspace_name="workspace_name",
132+
subscription_id="<subscription_ID>",
133+
resource_group_name="<resource-group-name>",
134+
workspace_name="<workspace_name>",
122135
)
123136

124137
# Create the compute instance.
@@ -130,13 +143,12 @@ spark_compute = ServerlessSparkCompute(
130143
# Specify your online endpoint deployment.
131144
monitoring_target = MonitoringTarget(
132145
ml_task="classification",
133-
endpoint_deployment_id="azureml:credit-default:main"
146+
endpoint_deployment_id="azureml:<endpoint-name>:<deployment-name>"
134147
)
135148

136-
137149
# Create an alert notification object.
138150
alert_notification = AlertNotification(
139-
151+
emails=['<email-address1>', '<email-address2>']
140152
)
141153

142154
# Create the monitor definition.
@@ -148,9 +160,9 @@ monitor_definition = MonitorDefinition(
148160

149161
# Specify the schedule frequency.
150162
recurrence_trigger = RecurrenceTrigger(
151-
frequency="day",
152-
interval=1,
153-
schedule=RecurrencePattern(hours=3, minutes=15)
163+
frequency="<frequency-unit>",
164+
interval=<interval>,
165+
schedule=RecurrencePattern(hours=<start-hour>, minutes=<start-minutes>)
154166
)
155167

156168
# Create the monitoring schedule.
@@ -215,20 +227,44 @@ After enabling feature importance, you'll see a feature importance for each feat
215227
You can use Azure CLI, the Python SDK, or the studio for advanced setup of model monitoring.
216228

217229
# [Azure CLI](#tab/azure-cli)
230+
1. Create a monitoring definition in a YAML file. For a sample advanced definition, see the following YAML code, which is also available in the [azureml-examples repository](https://github.com/Azure/azureml-examples/blob/main/cli/monitoring/advanced-model-monitoring.yaml).
218231

219-
Create advanced model monitoring setup with the following CLI command and YAML definition:
232+
Before you use this definition, adjust the following values and any others you need to fit your environment:
220233

221-
```azurecli
222-
az ml schedule create -f ./advanced-model-monitoring.yaml
223-
```
234+
- For `endpoint_deployment_id`, use a value in the format `azureml:<endpoint-name>:<deployment-name>`.
235+
- For `path` in reference input data sections, use a value in the format `azureml:<reference-data-asset-name>:<version>`.
236+
- For `target_column`, use something.
237+
- For `features`, list something.
238+
- For `emails`, list the email addresses that you want to use for notifications.
224239

225-
The following YAML contains the definition for advanced model monitoring.
240+
:::code language="yaml" source="~/azureml-examples-main/cli/monitoring/advanced-model-monitoring.yaml":::
226241

227-
:::code language="yaml" source="~/azureml-examples-main/cli/monitoring/advanced-model-monitoring.yaml":::
242+
1. Run the following command to create the model:
243+
244+
```azurecli
245+
az ml schedule create -f ./advanced-model-monitoring.yaml
246+
```
228247

229248
# [Python SDK](#tab/python)
230249

231-
Use the following code for advanced model monitoring setup:
250+
To set up advanced model monitoring, use code that's similar to the following sample. Replace the following placeholders with appropriate values:
251+
252+
| Placeholder | Description | Example |
253+
| --- | --- | --- |
254+
| <subscription_ID> | The ID of your subscription | aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e |
255+
| <resource-group-name> | The name of the resource group that contains your workspace | my-resource-group |
256+
| <workspace_name> | The name of your workspace | my-workspace |
257+
| <endpoint-name> | The name of the endpoint to monitor | credit-default |
258+
| <deployment-name> | The name of the deployment to monitor | main |
259+
| <production-data-asset-name> | The name of the data asset that contains production data | credit-default-main-model_inputs |
260+
| <reference-data-asset-name> | The name of the data asset that contains reference data | credit-default-reference |
261+
| <target-column> | Fill in | DEFAULT_NEXT_MONTH |
262+
| <feature1>, <feature2>, and <feature3> | Fill in , such as 'SEX', 'EDUCATION', or 'AGE' | AGE |
263+
| <email-address1> and <email-address2> | Email addresses to use for notifications | `[email protected]` |
264+
| <frequency-unit> | The monitoring frequency unit, such as "minute," "hour," "day," "week," or "month" | day |
265+
| <interval> | The interval between jobs, such as 1 or 2 days or weeks | 1 |
266+
| <start-hour> | The hour to start monitoring, on a 24-hour clock | 3 |
267+
| <start-minutes> | The minutes after the specified hour to start monitoring | 15 |
232268

233269
```python
234270
from azure.identity import DefaultAzureCredential
@@ -262,51 +298,52 @@ from azure.ai.ml.entities import (
262298
ProductionData
263299
)
264300

265-
# get a handle to the workspace
301+
# Get a handle to the workspace.
266302
ml_client = MLClient(
267303
DefaultAzureCredential(),
268-
subscription_id="subscription_id",
269-
resource_group_name="resource_group_name",
270-
workspace_name="workspace_name",
304+
subscription_id="<subscription_ID>",
305+
resource_group_name="<resource-group-name>",
306+
workspace_name="<workspace_name>",
271307
)
272308

273-
# create your compute
309+
# Create a compute instance.
274310
spark_compute = ServerlessSparkCompute(
275311
instance_type="standard_e4s_v3",
276312
runtime_version="3.3"
277313
)
278314

279-
# specify the online deployment (if you have one)
315+
# Specify the online deployment if you have one.
280316
monitoring_target = MonitoringTarget(
281317
ml_task="classification",
282-
endpoint_deployment_id="azureml:credit-default:main"
318+
endpoint_deployment_id="azureml:<endpoint-name>:<deployment-name>"
283319
)
284320

285-
# specify a lookback window size and offset, or omit this to use the defaults, which are specified in the documentation
321+
# Specify a look-back window size and offset to use. Omit this line to use the default values, which are listed in the documentation.
286322
data_window = BaselineDataRange(lookback_window_size="P1D", lookback_window_offset="P0D")
287323

324+
# Set up the production data.
288325
production_data = ProductionData(
289326
input_data=Input(
290327
type="uri_folder",
291-
path="azureml:credit-default-main-model_inputs:1"
328+
path="azureml:<production-data-asset-name>:1"
292329
),
293330
data_window=data_window,
294331
data_context=MonitorDatasetContext.MODEL_INPUTS,
295332
)
296333

297-
# training data to be used as reference dataset
334+
# Set up the training data to use as a reference dataset.
298335
reference_data_training = ReferenceData(
299336
input_data=Input(
300337
type="mltable",
301-
path="azureml:credit-default-reference:1"
338+
path="azureml:<reference-data-asset-name>:1"
302339
),
303340
data_column_names={
304-
"target_column":"DEFAULT_NEXT_MONTH"
341+
"target_column":"<target-column>"
305342
},
306343
data_context=MonitorDatasetContext.TRAINING,
307344
)
308345

309-
# create an advanced data drift signal
346+
# Create an advanced data drift signal.
310347
features = MonitorFeatureFilter(top_n_feature_importance=10)
311348

312349
metric_thresholds = DataDriftMetricThreshold(
@@ -324,7 +361,7 @@ advanced_data_drift = DataDriftSignal(
324361
metric_thresholds=metric_thresholds
325362
)
326363

327-
# create an advanced prediction drift signal
364+
# Create an advanced prediction drift signal.
328365
metric_thresholds = PredictionDriftMetricThreshold(
329366
categorical=CategoricalDriftMetrics(
330367
jensen_shannon_distance=0.01
@@ -336,8 +373,8 @@ advanced_prediction_drift = PredictionDriftSignal(
336373
metric_thresholds=metric_thresholds
337374
)
338375

339-
# create an advanced data quality signal
340-
features = ['SEX', 'EDUCATION', 'AGE']
376+
# Create an advanced data quality signal.
377+
features = ['<feature1>', '<feature2>', '<feature3>']
341378

342379
metric_thresholds = DataQualityMetricThreshold(
343380
numerical=DataQualityMetricsNumerical(
@@ -355,7 +392,7 @@ advanced_data_quality = DataQualitySignal(
355392
alert_enabled=False
356393
)
357394

358-
# create feature attribution drift signal
395+
# Create a feature attribution drift signal.
359396
metric_thresholds = FeatureAttributionDriftMetricThreshold(normalized_discounted_cumulative_gain=0.9)
360397

361398
feature_attribution_drift = FeatureAttributionDriftSignal(
@@ -364,94 +401,92 @@ feature_attribution_drift = FeatureAttributionDriftSignal(
364401
alert_enabled=False
365402
)
366403

367-
# put all monitoring signals in a dictionary
404+
# Put all monitoring signals in a dictionary.
368405
monitoring_signals = {
369406
'data_drift_advanced':advanced_data_drift,
370407
'data_quality_advanced':advanced_data_quality,
371408
'feature_attribution_drift':feature_attribution_drift,
372409
}
373410

374-
# create alert notification object
411+
# Create an alert notification object.
375412
alert_notification = AlertNotification(
376-
413+
emails=['<email-address1>', '<email-address2>']
377414
)
378415

379-
# create the monitor definition
416+
# Create the monitor definition.
380417
monitor_definition = MonitorDefinition(
381418
compute=spark_compute,
382419
monitoring_target=monitoring_target,
383420
monitoring_signals=monitoring_signals,
384421
alert_notification=alert_notification
385422
)
386423

387-
# specify the frequency on which to run your monitor
424+
# Specify the schedule frequency.
388425
recurrence_trigger = RecurrenceTrigger(
389-
frequency="day",
390-
interval=1,
391-
schedule=RecurrencePattern(hours=3, minutes=15)
426+
frequency="<frequency-unit>",
427+
interval=<interval>,
428+
schedule=RecurrencePattern(hours=<start-hour>, minutes=<start-minutes>)
392429
)
393430

394-
# create your monitor
431+
# Create the monitoring schedule.
395432
model_monitor = MonitorSchedule(
396433
name="credit_default_monitor_advanced",
397434
trigger=recurrence_trigger,
398435
create_monitor=monitor_definition
399436
)
400437

438+
# Schedule the monitoring job.
401439
poller = ml_client.schedules.begin_create_or_update(model_monitor)
402440
created_monitor = poller.result()
403441
```
404442

405443
# [Studio](#tab/azure-studio)
406444

407-
To set up advanced monitoring:
445+
To set up advanced monitoring, take the following steps:
408446

409-
1. Complete the entires on the **Basic settings** page as described earlier in the [Set up out-of-box model monitoring](#set-up-out-of-box-model-monitoring) section.
410-
1. Select **Next** to open the **Configure data asset** page of the **Advanced settings** section.
411-
1. **Add** a dataset to be used as the reference dataset. We recommend that you use the model training data as the comparison reference dataset for data drift and data quality. Also, use the model validation data as the comparison reference dataset for prediction drift.
447+
1. In [Azure Machine Learning studio](https://ml.azure.com), go to your workspace.
448+
1. Under **Manage**, select **Monitoring**, and then select **Add**.
449+
1. On the Basic settings page, enter information as described earlier in [Set up out-of-box model monitoring](#set-up-out-of-box-model-monitoring).
450+
1. Select **Next** to open the Configure data asset page of the **Advanced settings** section.
451+
1. If you don't see the data asset that you want to use as a reference dataset, select **Add**. We recommend that you use the model training data as the comparison reference dataset for data drift and data quality. Also, use the model validation data as the comparison reference dataset for prediction drift. Add the data assets that you want to use.
412452

413453
:::image type="content" source="media/how-to-monitor-models/model-monitoring-advanced-configuration-data.png" alt-text="Screenshot showing how to add datasets for the monitoring signals to use." lightbox="media/how-to-monitor-models/model-monitoring-advanced-configuration-data.png":::
414454

415-
1. Select **Next** to go to the **Select monitoring signals** page. On this page, you see some monitoring signals already added (if you selected an Azure Machine Learning online deployment earlier). The signals (data drift, prediction drift, and data quality) use recent, past production data as the comparison reference dataset and use smart defaults for metrics and thresholds.
455+
1. Select **Next**. The **Select monitoring signals** page opens. If you selected an Azure Machine Learning online deployment earlier, you see some monitoring signals. The data drift, prediction drift, and data quality signals use recent, past production data as the comparison reference dataset and use smart defaults for metrics and thresholds.
416456

417457
:::image type="content" source="media/how-to-monitor-models/model-monitoring-monitoring-signals.png" alt-text="Screenshot showing default monitoring signals." lightbox="media/how-to-monitor-models/model-monitoring-monitoring-signals.png":::
418458

419-
1. Select **Edit** next to the data drift signal.
420-
1. Configure the data drift in the **Edit signal** window as follows:
421-
422-
1. In step 1, for the production data asset, select your model inputs dataset. Also, make the following selection:
423-
- Select the desired lookback window size.
424-
1. In step 2, for the reference data asset, select your training dataset. Also, make the following selection:
425-
- Select the target (output) column.
426-
1. In step 3, select to monitor drift for the top N most important features, or monitor drift for a specific set of features.
427-
1. In step 4, select your preferred metric and thresholds to use for numerical features.
428-
1. In step 5, select your preferred metric and thresholds to use for categorical features.
459+
1. Next to the data drift signal, select **Edit**.
460+
1. In the **Edit Signal** window, take the following steps to configure the data drift signal:
461+
1. In step 1, for the production data asset, select your model input data asset. Also select the lookback window size that you want to use.
462+
1. In step 2, for the reference data asset, select your training dataset. Also select the target, or output, column.
463+
1. In step 3, select **Top N features** to monitor drift for the *N* most important features. Or select specific features if you want to monitor drift for a specific set.
464+
1. In step 4, select the metric and threshold that you want to use for numerical features.
465+
1. In step 5, select the metric and threshold that you want to use for categorical features.
466+
1. Select **Save**.
429467

430468
:::image type="content" source="media/how-to-monitor-models/model-monitoring-configure-signals.png" alt-text="Screenshot showing how to configure selected monitoring signals." lightbox="media/how-to-monitor-models/model-monitoring-configure-signals.png":::
431469

432-
1. Select **Save** to return to the **Select monitoring signals** page.
433-
1. Select **Add** to open the **Edit Signal** window.
434-
1. Select **Feature attribution drift (preview)** to configure the feature attribution drift signal as follows:
435-
436-
1. In step 1, select the production data asset that has your model inputs
437-
- Also, select the desired lookback window size.
438-
1. In step 2, select the production data asset that has your model outputs.
439-
- Also, select the common column between these data assets to join them on. If the data was collected with the [data collector](how-to-collect-production-data.md), the common column is `correlationid`.
440-
1. (Optional) If you used the data collector to collect data that has your model inputs and outputs already joined, select the joined dataset as your production data asset (in step 1)
441-
- Also, **Remove** step 2 in the configuration panel.
442-
1. In step 3, select your training dataset to use as the reference dataset.
443-
- Also, select the target (output) column for your training dataset.
444-
1. In step 4, select your preferred metric and threshold.
470+
1. On the Select monitoring signals page, select **Add**.
471+
1. On the Edit Signal window, Select **Feature attribution drift (PREVIEW)**, and then take the following steps to configure the feature attribution drift signal:
472+
473+
1. In step 1, select the production data asset that has your model inputs. Also select the lookback window size that you want to use.
474+
1. In step 2, select the production data asset that has your model outputs. Also select the common column to use to join the production data and the output data. If you use the [data collector](how-to-collect-production-data.md) to collect data, select **correlationid**.
475+
1. (Optional) If you use the data collector to collect data that has your model inputs and outputs already joined, take the following steps:
476+
1. In step 1, for the production data asset, select the joined dataset.
477+
1. In step 2, select **Remove** to remove step 2 from the configuration panel.
478+
1. In step 3, for the reference dataset, select your training dataset. Also select the target, or output, column for your training dataset.
479+
1. In step 4, select the metric and threshold that you want to use.
480+
1. Select **Save**.
445481

446482
:::image type="content" source="media/how-to-monitor-models/model-monitoring-configure-feature-attribution-drift.png" alt-text="Screenshot showing how to configure feature attribution drift signal." lightbox="media/how-to-monitor-models/model-monitoring-configure-feature-attribution-drift.png":::
447483

448-
1. Select **Save** to return to the **Select monitoring signals** page.
484+
1. On the Select monitoring signals page, finish configuring your monitoring signals, and then select **Next**.
449485

450486
:::image type="content" source="media/how-to-monitor-models/model-monitoring-configured-signals.png" alt-text="Screenshot showing the configured signals." lightbox="media/how-to-monitor-models/model-monitoring-configured-signals.png":::
451487

452-
1. When you're finished with your monitoring signals configuration, select **Next** to go to the **Notifications** page.
453-
1. On the **Notifications** page, enable alert notifications for each signal and select **Next**.
454-
1. Review your settings on the **Review monitoring settings** page.
488+
1. On the Notifications page, turn on notifications for each signal, and then select **Next**.
489+
1. On the Review monitoring details page, review your settings.
455490

456491
:::image type="content" source="media/how-to-monitor-models/model-monitoring-advanced-configuration-review.png" alt-text="Screenshot showing review page of the advanced configuration for model monitoring." lightbox="media/how-to-monitor-models/model-monitoring-advanced-configuration-review.png":::
457492

0 commit comments

Comments
 (0)