Merge pull request #98058 from RachelKellam/master

Understand charts and metrics updates

Author: tiburd

articles/machine-learning/service/how-to-understand-automated-ml.md

@@ -3,13 +3,13 @@ title: Understand automated ML results
 titleSuffix: Azure Machine Learning
 description: Learn how to view and understand charts and metrics for each of your automated machine learning runs. 
 services: machine-learning
-author: cartacioS
-ms.author: sacartac
+author: RachelKellam
+ms.author: rakellam
 ms.reviewer: sgilley
 ms.service: machine-learning
 ms.subservice: core
 ms.topic: conceptual
-ms.date: 11/04/2019
+ms.date: 12/05/2019
 ---
 
 # Understand automated machine learning results
@@ -93,76 +93,108 @@ recall_score_macro|Recall is the percent of correctly labeled elements of a cert
 recall_score_micro|Recall is the percent of correctly labeled elements of a certain class. Micro is computed globally by counting the total true positives, false negatives and false positives|[Calculation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html)|average="micro"|
 recall_score_weighted|Recall is the percent of correctly labeled elements of a certain class. Weighted is the arithmetic mean of recall for each class, weighted by number of true instances in each class.|[Calculation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html)|average="weighted"|
 weighted_accuracy|Weighted accuracy is accuracy where the weight given to each example is equal to the proportion of true instances in that example's true class.|[Calculation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)|sample_weight is a vector equal to the proportion of that class for each element in the target|
-
+<a name="confusion-matrix"></a>
 ### Confusion matrix
+#### What is a confusion matrix?
+A confusion matrix is used to describe the performance of a classification model. Each row displays the instances of the true, or actual class in your dataset, and each column represents the instances of the class that was predicted by the model. 
 
-A confusion matrix is used to describe the performance of a classification model. Each row displays the instances of the true class, and each column represents the instances of the predicted class. The confusion matrix shows the correctly classified labels and the incorrectly classified labels for a given model.
+#### What does automated ML do with the confusion matrix?
+For classification problems, Azure Machine Learning automatically provides a confusion matrix for each model that is built. For each confusion matrix, automated ML will show the frequency of each predicted label (column) compared against the true label (row). The darker the color, the higher the count in that particular part of the matrix. 
 
-For classification problems, Azure Machine Learning automatically provides a confusion matrix for each model that is built. For each confusion matrix, automated ML will show the frequency of each predicted label and each true label intersection. The darker the color, the higher the count in that particular part of the matrix. Ideally, the darkest colors would be along the diagonal of the matrix. 
+#### What does a good model look like?
+We are comparing the actual value of the dataset against the predicted values that the model gave. Because of this, machine learning models have higher accuracy if the model has most of its values along the diagonal, meaning the model predicted the correct value. If a model has class imbalance, the confusion matrix will help to detect a biased model.
 
-Example 1: A classification model with poor accuracy
+##### Example 1: A classification model with poor accuracy
 ![A classification model with poor accuracy](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-confusion-matrix1.png)
 
-Example 2: A classification model with high accuracy (ideal)
+##### Example 2: A classification model with high accuracy 
 ![A classification model with high accuracy](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-confusion-matrix2.png)
 
+##### Example 3: A classification model with high accuracy and high bias in model predictions
+![A classification model with high accuracy and high bias in model predictions](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-biased-model.png)
 
+<a name="precision-recall-chart"></a>
 ### Precision-recall chart
+#### What is a precision-recall chart?
+The precision-recall curve shows the relationship between precision and recall from a model. The term precision represents that ability for a model to label all instances correctly. Recall represents the ability for a classifier to find all instances of a particular label.
+
+#### What does automated ML do with the precision-recall chart?
+
+With this chart, you can compare the precision-recall curves for each model to determine which model has an acceptable relationship between precision and recall for your particular business problem. This chart shows Macro Average Precision-Recall, Micro Average Precision-Recall, and the precision-recall associated with all classes for a model. 
 
-With this chart, you can compare the precision-recall curves for each model to determine which model has an acceptable relationship between precision and recall for your particular business problem. This chart shows Macro Average Precision-Recall, Micro Average Precision-Recall, and the precision-recall associated with all classes for a model.
+Macro-average will compute the metric independently of each class and then take the average, treating all classes equally. However, micro-average will aggregate the contributions of all the classes to compute the average. Micro-average is preferable if there is class imbalance present in the dataset.
 
-The term Precision represents that ability for a classifier to label all instances correctly. Recall represents the ability for a classifier to find all instances of a particular label. The precision-recall curve shows the relationship between these two concepts. Ideally, the model would have 100% precision and 100% accuracy.
+#### What does a good model look like?
+Depending on the goal of the business problem, the ideal precision-recall curve could differ. Some examples are given below
 
-Example 1: A classification model with low precision and low recall
+##### Example 1: A classification model with low precision and low recall
 ![A classification model with low precision and low recall](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-precision-recall1.png)
 
-Example 2: A classification model with ~100% precision and ~100% recall (ideal)
+##### Example 2: A classification model with ~100% precision and ~100% recall 
 ![A classification model high precision and recall](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-precision-recall2.png)
+<a name="roc"></a>
+### ROC chart
 
-### ROC
-
+#### What is a ROC chart?
 Receiver operating characteristic (or ROC) is a plot of the correctly classified labels vs. the incorrectly classified labels for a particular model. The ROC curve can be less informative when training models on datasets with high bias, as it will not show the false positive labels.
 
-Example 1: A classification model with low true labels and high false labels
-![Classification model with low true labels and high false labels](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-roc-1.png)
+#### What does automated ML do with the ROC chart?
+Automated ML generates Macro Average Precision-Recall, Micro Average Precision-Recall, and the precision-recall associated with all classes for a model. 
 
-Example 2: A classification model with high true labels and low false labels
-![a classification model with high true labels and low false labels](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-roc-2.png)
+Macro-average will compute the metric independently of each class and then take the average, treating all classes equally. However, micro-average will aggregate the contributions of all the classes to compute the average. Micro-average is preferable if there is class imbalance present in the dataset.
 
-### Lift curve
+#### What does a good model look like?
+Ideally, the model will have closer to 100% true positive rate and closer to 0% false positive rate. 
 
-You can compare the lift of the model built automatically with Azure Machine Learning to the baseline in order to view the value gain of that particular model.
+##### Example 1: A classification model with low true labels and high false labels
+![Classification model with low true labels and high false labels](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-roc-1.png)
 
-Lift charts are used to evaluate the performance of a classification model. It shows how much better you can expect to do with a model compared to without a model. 
+##### Example 2: A classification model with high true labels and low false labels
+![a classification model with high true labels and low false labels](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-roc-2.png)
+<a name="lift-curve"></a>
+### Lift chart
+#### What is a lift chart?
+Lift charts are used to evaluate the performance of a classification model. It shows how much better you can expect to do with the generated model compared to without a model in terms of accuracy.
+#### What does automated ML do with the lift chart?
+You can compare the lift of the model built automatically with Azure Machine Learning to the baseline in order to view the value gain of that particular model.
+#### What does a good model look like?
 
-Example 1: Model performs worse than a random selection model
+##### Example 1: A classification model that does worse than a random selection model
 ![A classification model that does worse than a random selection model](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-lift-curve1.png)
-
-Example 2: Model performs better than a random selection model
+##### Example 2: A classification model that performs better than a random selection model
 ![A classification model that performs better](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-lift-curve2.png)
-
-### Gains curve
+<a name="gains-curve"></a>
+### Gains chart
+#### What is a gains chart?
 
 A gains chart evaluates the performance of a classification model by each portion of the data. It shows for each percentile of the data set, how much better you can expect to perform compared against a random selection model.
 
+#### What does automated ML do with the gains chart?
 Use the cumulative gains chart to help you choose the classification cutoff using a percentage that corresponds to a desired gain from the model. This information provides another way of looking at the results in the accompanying lift chart.
 
-Example 1: A classification model with minimal gain
+#### What does a good model look like?
+##### Example 1: A classification model with minimal gain
 ![a classification model with minimal gain](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-gains-curve1.png)
 
-Example 2: A classification model with significant gain
+##### Example 2: A classification model with significant gain
 ![A classification model with significant gain](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-gains-curve2.png)
+<a name="calibration-plot"></a>
+### Calibration chart
 
-### Calibration plot
+#### What is a calibration chart?
+A calibration plot is used to display the confidence of a predictive model. It does this by showing the relationship between the predicted probability and the actual probability, where “probability” represents the likelihood that a particular instance belongs under some label.
+#### What does automated ML do with the calibration chart?
+For all classification problems, you can review the calibration line for micro-average, macro-average, and each class in a given predictive model.
 
-For all classification problems, you can review the calibration line for micro-average, macro-average, and each class in a given predictive model. 
+Macro-average will compute the metric independently of each class and then take the average, treating all classes equally. However, micro-average will aggregate the contributions of all the classes to compute the average. 
+#### What does a good model look like?
+ A well-calibrated model  aligns with the y=x line, where it is reasonably confident in its predictions. An over-confident model  aligns with the y=0 line, where the predicted probability is present but there is no actual probability. 
 
-A calibration plot is used to display the confidence of a predictive model. It does this by showing the relationship between the predicted probability and the actual probability, where “probability” represents the likelihood that a particular instance belongs under some label. A well calibrated model  aligns with the y=x line, where it is reasonably confident in its predictions. An over-confident model  aligns with the y=0 line, where the predicted probability is present but there is no actual probability.
 
-Example 1: A more well-calibrated model
+##### Example 1: A well-calibrated model
 ![ more well-calibrated model](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-calib-curve1.png)
 
-Example 2: An over-confident model
+##### Example 2: An over-confident model
 ![An over-confident model](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-calib-curve2.png)
 
 ## <a name="regression"></a> Regression results
@@ -192,38 +224,38 @@ normalized_root_mean_squared_error|Normalized root mean squared error is root me
 root_mean_squared_log_error|Root mean squared log error is the square root of the expected squared logarithmic error|[Calculation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html)|None|
 normalized_root_mean_squared_log_error|Normalized Root mean squared log error is root mean squared log error divided by the range of the data|[Calculation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html)|Divide by range of the data|
 
-### <a name="pvt"></a> Predicted vs. True
-
+### <a name="pvt"></a> Predicted vs. True chart
+#### What is a Predicted vs. True chart?
 Predicted vs. True shows the relationship between a predicted value and its correlating true value for a regression problem. This graph can be used to measure performance of a model as the closer to the y=x line the predicted values are, the better the accuracy of a predictive model.
 
+#### What does automated ML do with the Predicted vs. True chart?
 After each run, you can see a predicted vs. true graph for each regression model. To protect data privacy, values are binned together and the size of each bin is shown as a bar graph on the bottom portion of the chart area. You can compare the predictive model, with the lighter shade area showing error margins, against the ideal value of where the model should be.
 
-Example 1: A regression model with low accuracy in predictions
+#### What does a good model look like?
+##### Example 1: A classification model with low accuracy
 ![A regression model with low accuracy in predictions](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-regression1.png)
 
-Example 2: A regression model with high accuracy in its predictions
+##### Example 2: A regression model with high accuracy 
 [![A regression model with high accuracy in its predictions](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-regression2.png)](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-regression2-expanded.png)
 
 
 
-### <a name="histo"></a> Histogram of residuals
-
+### <a name="histo"></a> Histogram of residuals chart
+#### What is a residuals chart?
 A residual represents an observed y – the predicted y. To show a margin of error with low bias, the histogram of residuals should be shaped as a bell curve, centered around 0. 
+#### What does automated ML do with the residuals chart?
+Automated ML automatically provides a residuals chart to show the distribution of errors in the predictions.
+#### What does a good model look like?
+A good model will typically have a bell curve or errors around zero.
 
-Example 1: A regression model with bias in its errors
+##### Example 1: A regression model with bias in its errors
 ![SA regression model with bias in its errors](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-regression3.png)
 
-Example 2: A regression model with more even distribution of errors
+##### Example 2: A regression model with more even distribution of errors
 ![A regression model with more even distribution of errors](./media/how-to-understand-automated-ml/azure-machine-learning-auto-ml-regression4.png)
 
 ## <a name="explain-model"></a> Model interpretability and feature importance
-
-Feature importance allows you to see how valuable each feature was in the construction of a model. This calculation is turned off by default, as it can significantly increase run time.   You can enable model explanation for all models or explain only the best fit model.
-
-You can review the feature importance score for the model overall as well as per class on a predictive model. You can see per feature how the importance compares against each class and overall.
-
-![Feature Explain ability](./media/how-to-understand-automated-ml/feature-importance.gif)
-
+Automated ML provides a machine learning interpretability dashboard for your runs.
 For more information on enabling interpretability features, see the [how-to](how-to-machine-learning-interpretability-automl.md) on enabling interpretability in automated ML experiments.
 
 ## Next steps