You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/machine-learning/how-to-monitor-model-performance.md
+23-17Lines changed: 23 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -425,44 +425,50 @@ To set up advanced monitoring:
425
425
426
426
## Set up model performance monitoring
427
427
428
-
Azure Machine Learning model monitoring enables you to track the performance of your models in
429
-
production by calculating model performance metrics. The following set of model performance metrics are currently supported:
428
+
Azure Machine Learning model monitoring enables you to track the performance of your models in production by calculating their performance metrics. The following model performance metrics are currently supported:
429
+
430
+
For classification models:
430
431
431
-
Classification models:
432
432
- Precision
433
433
- Accuracy
434
434
- Recall
435
435
436
-
Regression models:
436
+
For regression models:
437
+
437
438
- Mean Absolute Error (MAE)
438
439
- Mean Squared Error (MSE)
439
440
- Root Mean Squared Error (RMSE)
440
441
441
-
Before you can configure your model performance signal, you need to satisfy the following requirements:
442
+
### More prerequisites for model performance monitoring
442
443
443
-
* Have production model output data (your model's predictions) with a unique ID for each row. If you collect your data with the [Azure Machine Learning data collector](how-to-collect-production-data.md), a `correlation_id` is provided for each inference request for you. With the data collector, you also have the option to log your own unique ID from your application.
444
+
You must satisfy the following requirements for you to configure your model performance signal:
445
+
446
+
* Have output data for the production model (the model's predictions) with a unique ID for each row. If you collect production data with the [Azure Machine Learning data collector](how-to-collect-production-data.md), a `correlation_id` is provided for each inference request for you. With the data collector, you also have the option to log your own unique ID from your application.
444
447
445
448
> [!NOTE]
446
449
>
447
-
> For Azure Machine Learning model performance monitoring, we recommend logging your unique ID as it's own column using the [Azure Machine Learning data collector](how-to-collect-production-data.md). This will ensure that each collected row is guaranteed to have a unique ID.
450
+
> For Azure Machine Learning model performance monitoring, we recommend that you log your unique ID in its own column, using the [Azure Machine Learning data collector](how-to-collect-production-data.md). This practice will ensure that each collected row is guaranteed to have a unique ID.
448
451
449
-
* Have ground truth data (actuals) with a unique ID for each row. The unique ID for a given row should match the unique ID for the model outputs for that particular inference request. This unique ID will be used to join your ground truth dataset with the model outputs.
452
+
* Have ground truth data (actuals) with a unique ID for each row. The unique ID for a given row should match the unique ID for the model outputs for that particular inference request. This unique ID is used to join your ground truth dataset with the model outputs.
450
453
451
-
* (Optional) Have a pre-joined tabular dataset with model outputs and ground truth data already joined together.
454
+
Without having ground truth data, you can't perform model performance monitoring. Since ground truth data is encountered at the application level, it's your responsibility to collect it as it becomes available. You should also maintain a data asset in Azure Machine Learning that contains this ground truth data.
452
455
453
-
The key requirement for enabling model performance monitoring is that you already collected ground truth data. Since ground truth data is encountered at the application level, it's your responsibility to collect it as it becomes available. You should also maintain a data asset in Azure Machine Learning with this ground truth data.
454
-
455
-
### Example scenario
456
+
* (Optional) Have a pre-joined tabular dataset with model outputs and ground truth data already joined together.
456
457
457
-
Here is an example scenario to help you understand the concepts associated with model performance monitoring:
458
+
### When should you monitor model performance?
458
459
459
-
Support you have deployed a model to predict if credit card transactions are fraudulent or not. As part of your deployment, you use the [data collector](how-to-collect-production-data.md) to collect production model input and model output data. You log a unique ID for each row (this unique ID can come from your application, or you can use our generated `correlationid`, which is unique for each logged JSON object). When ground truth data becomes available, it is logged and mapped to the same unique ID which was logged with the model outputs. This `is_fraud` data is collected, maintained, and registered to Azure Machine Learning as a data asset. Then, a model performance monitoring signal can be created to join the model outputs and ground truth datasets on the unique ID which was logged. Lastly, model performance metrics are computed.
460
+
The following example describes model performance monitoring for a scenario when you deploy a model to predict whether credit card transactions are fraudulent or not.
460
461
461
-
Alternatively,
462
+
1. Suppose your deployment uses the data collector to collect the model's production inference data (input and output data), and the output data is stored in a column `is_fraud`.
463
+
1. For each row of the collected inference data, you log a unique ID. The unique ID can come from your application, or you can use the `correlationid` that Azure Machine Learning uniquely generates for each logged JSON object.
464
+
1. Later, when the ground truth (or actual) `is_fraud` data becomes available, it also gets logged and mapped to the same unique ID that was logged with the model's outputs.
465
+
1. This ground truth `is_fraud` data is also collected, maintained, and registered to Azure Machine Learning as a data asset.
466
+
1. You can then create a model performance monitoring signal that joins the model's production inference and ground truth data assets, using the unique ID columns.
467
+
1. Finally, you can compute the model performance metrics.
462
468
463
469
# [Azure CLI](#tab/azure-cli)
464
470
465
-
Once you've satisfied the previous requirements, you can set up model monitoring with the following CLI command and YAML definition:
471
+
Once you've satisfied the [prerequisites for model performance monitoring](#more-prerequisites-for-model-performance-monitoring), you can set up model monitoring with the following CLI command and YAML definition:
466
472
467
473
```azurecli
468
474
az ml schedule create -f ./model-performance-monitoring.yaml
@@ -519,7 +525,7 @@ create_monitor:
519
525
520
526
# [Python SDK](#tab/python)
521
527
522
-
Once you've satisfied the previous requirements, you can set up model monitoring using the following Python code:
528
+
Once you've satisfied the [prerequisites for model performance monitoring](#more-prerequisites-for-model-performance-monitoring), you can set up model monitoring with the following Python code:
523
529
524
530
```python
525
531
from azure.identity import InteractiveBrowserCredential
0 commit comments