Merge pull request #2169 from lorentzenchr:improve_imbalanced_classification

PiperOrigin-RevId: 515322926

Author: copybara-github

site/en/tutorials/structured_data/imbalanced_data.ipynb

@@ -76,9 +76,8 @@
         "* Create train, validation, and test sets.\n",
         "* Define and train a model using Keras (including setting class weights).\n",
         "* Evaluate the model using various metrics (including precision and recall).\n",
-        "* Try common techniques for dealing with imbalanced data like:\n",
-        "    * Class weighting \n",
-        "    * Oversampling\n"
+        "* Select a threshold for a probabilistic classifier to get a deterministic classifier.\n",
+        "* Try and compare with class weighted modelling and oversampling."
       ]
     },
     {
@@ -275,6 +274,31 @@
         "id": "8a_Z_kBmr7Oh"
       },
       "source": [
+        "We check whether the distribution of the classes in the three sets is about the same or not."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "96520cffee66"
+      },
+      "outputs": [],
+      "source": [
+        "print(f'Average class probability in training set:   {train_labels.mean():.4f}')\n",
+        "print(f'Average class probability in validation set: {val_labels.mean():.4f}')\n",
+        "print(f'Average class probability in test set:       {test_labels.mean():.4f}')"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8a_Z_kBmr7Oh"
+      },
+      "source": [
+        "Given the small number of positive labels, this seems about right.\n",
+        "\n",
         "Normalize the input features using the sklearn StandardScaler.\n",
         "This will set the mean to 0 and standard deviation to 1.\n",
         "\n",
@@ -374,6 +398,8 @@
       "outputs": [],
       "source": [
         "METRICS = [\n",
+        "      keras.metrics.BinaryCrossentropy(name='cross entropy'),  # same as model's loss\n",
+        "      keras.metrics.MeanSquaredError(name='Brier score'),\n",
         "      keras.metrics.TruePositives(name='tp'),\n",
         "      keras.metrics.FalsePositives(name='fp'),\n",
         "      keras.metrics.TrueNegatives(name='tn'),\n",
@@ -406,6 +432,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "SU0GX6E6mieP"
@@ -414,8 +441,22 @@
         "### Understanding useful metrics\n",
         "\n",
         "Notice that there are a few metrics defined above that can be computed by the model that will be helpful when evaluating the performance.\n",
+        "These can be divided into three groups.\n",
+        "\n",
+        "#### Metrics for probability predictions\n",
+        "\n",
+        "As we train our network with the cross entropy as a loss function, it is fully capable of predicting class probabilities, i.e. it is a probabilistic classifier.\n",
+        "Good metrics to assess probabilistic predictions are, in fact, **proper scoring rules**. Their key property is that predicting the true probability is optimal. We give two well-known examples:\n",
         "\n",
+        "*   **cross entropy** also known as log loss\n",
+        "*   **Mean squared error** also known as the Brier score\n",
         "\n",
+        "#### Metrics for deterministic 0/1 predictions\n",
+        "\n",
+        "In the end, one often wants to predict a class label, 0 or 1, *no fraud* or *fraud*.\n",
+        "This is called a deterministic classifier.\n",
+        "To get a label prediction from our probabilistic classifier, one needs to choose a probability threshold $t$.\n",
+        "The default is to predict label 1 (fraud) if the predicted probability is larger than $t=50\\%$ and all the following metrics implicitly use this default. \n",
         "\n",
         "*   **False** negatives and **false** positives are samples that were **incorrectly** classified\n",
         "*   **True** negatives and **true** positives are samples that were **correctly** classified\n",
@@ -425,14 +466,21 @@
         ">   $\\frac{\\text{true positives}}{\\text{true positives + false positives}}$\n",
         "*   **Recall** is the percentage of **actual** positives that were correctly classified\n",
         ">   $\\frac{\\text{true positives}}{\\text{true positives + false negatives}}$\n",
+        "\n",
+        "**Note:** Accuracy is not a helpful metric for this task. You can have 99.8%+ accuracy on this task by predicting False all the time.  \n",
+        "\n",
+        "#### Other metrices\n",
+        "\n",
+        "The following metrics take into account all possible choices of thresholds $t$.\n",
+        "\n",
         "*   **AUC** refers to the Area Under the Curve of a Receiver Operating Characteristic curve (ROC-AUC). This metric is equal to the probability that a classifier will rank a random positive sample higher than a random negative sample.\n",
         "*   **AUPRC** refers to Area Under the Curve of the Precision-Recall Curve. This metric computes precision-recall pairs for different probability thresholds. \n",
         "\n",
-        "Note: Accuracy is not a helpful metric for this task. You can have 99.8%+ accuracy on this task by predicting False all the time.  \n",
         "\n",
-        "Read more:\n",
-        "*  [True vs. False and Positive vs. Negative](https://developers.google.com/machine-learning/crash-course/classification/true-false-positive-negative)\n",
-        "*  [Accuracy](https://developers.google.com/machine-learning/crash-course/classification/accuracy)\n",
+        "#### Read more:\n",
+        "*   [Strictly Proper Scoring Rules, Prediction, and Estimation](https://www.stat.washington.edu/people/raftery/Research/PDF/Gneiting2007jasa.pdf)\n",
+        "*   [True vs. False and Positive vs. Negative](https://developers.google.com/machine-learning/crash-course/classification/true-false-positive-negative)\n",
+        "*   [Accuracy](https://developers.google.com/machine-learning/crash-course/classification/accuracy)\n",
         "*   [Precision and Recall](https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall)\n",
         "*   [ROC-AUC](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc)\n",
         "*   [Relationship between Precision-Recall and ROC Curves](https://www.biostat.wisc.edu/~page/rocpr.pdf)"
@@ -458,7 +506,7 @@
         "Now create and train your model using the function that was defined earlier. Notice that the model is fit using a larger than default batch size of 2048, this is important to ensure that each batch has a decent chance of containing a few positive samples. If the batch size was too small, they would likely have no fraudulent transactions to learn from.\n",
         "\n",
         "\n",
-        "Note: this model will not handle the class imbalance well. You will improve it later in this tutorial."
+        "Note: Fitting this model will not handle the class imbalance efficiently. You will improve it later in this tutorial."
       ]
     },
     {
@@ -527,7 +575,7 @@
         "id": "qk_3Ry6EoYDq"
       },
       "source": [
-        "These initial guesses are not great. You know the dataset is imbalanced. Set the output layer's bias to reflect that (See: [A Recipe for Training Neural Networks: \"init well\"](http://karpathy.github.io/2019/04/25/recipe/#2-set-up-the-end-to-end-trainingevaluation-skeleton--get-dumb-baselines)). This can help with initial convergence."
+        "These initial guesses are not great. You know the dataset is imbalanced. Set the output layer's bias to reflect that, see [A Recipe for Training Neural Networks: \"init well\"](http://karpathy.github.io/2019/04/25/recipe/#2-set-up-the-end-to-end-trainingevaluation-skeleton--get-dumb-baselines). This can help with initial convergence."
       ]
     },
     {
@@ -628,9 +676,9 @@
         "id": "FrDC8hvNr9yw"
       },
       "source": [
-        "This initial loss is about 50 times less than if would have been with naive initialization.\n",
+        "This initial loss is about 50 times less than it would have been with naive initialization.\n",
         "\n",
-        "This way the model doesn't need to spend the first few epochs just learning that positive examples are unlikely. This also makes it easier to read plots of the loss during training."
+        "This way the model doesn't need to spend the first few epochs just learning that positive examples are unlikely. It also makes it easier to read plots of the loss during training."
       ]
     },
     {
@@ -724,7 +772,8 @@
         "               color=colors[n], label='Val ' + label,\n",
         "               linestyle=\"--\")\n",
         "  plt.xlabel('Epoch')\n",
-        "  plt.ylabel('Loss')"
+        "  plt.ylabel('Loss')\n",
+        "  plt.legend()"
       ]
     },
     {
@@ -868,11 +917,11 @@
       },
       "outputs": [],
       "source": [
-        "def plot_cm(labels, predictions, p=0.5):\n",
-        "  cm = confusion_matrix(labels, predictions > p)\n",
+        "def plot_cm(labels, predictions, threshold=0.5):\n",
+        "  cm = confusion_matrix(labels, predictions > threshold)\n",
         "  plt.figure(figsize=(5,5))\n",
         "  sns.heatmap(cm, annot=True, fmt=\"d\")\n",
-        "  plt.title('Confusion matrix @{:.2f}'.format(p))\n",
+        "  plt.title('Confusion matrix @{:.2f}'.format(threshold))\n",
         "  plt.ylabel('Actual label')\n",
         "  plt.xlabel('Predicted label')\n",
         "\n",
@@ -915,18 +964,47 @@
         "id": "PyZtSr1v6L4t"
       },
       "source": [
-        "If the model had predicted everything perfectly, this would be a [diagonal matrix](https://en.wikipedia.org/wiki/Diagonal_matrix) where values off the main diagonal, indicating incorrect predictions, would be zero. In this case the matrix shows that you have relatively few false positives, meaning that there were relatively few legitimate transactions that were incorrectly flagged. However, you would likely want to have even fewer false negatives despite the cost of increasing the number of false positives. This trade off may be preferable because false negatives would allow fraudulent transactions to go through, whereas false positives may cause an email to be sent to a customer to ask them to verify their card activity."
+        "If the model had predicted everything perfectly (impossible with true randomness), this would be a [diagonal matrix](https://en.wikipedia.org/wiki/Diagonal_matrix) where values off the main diagonal, indicating incorrect predictions, would be zero. In this case, the matrix shows that you have relatively few false positives, meaning that there were relatively few legitimate transactions that were incorrectly flagged."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "P-QpQsip_F2Q"
+      },
+      "source": [
+        "### Changing the threshold\n",
+        "\n",
+        "The default threshold of $t=50\\%$ corresponds to equal costs of false negatives and false positives.\n",
+        "In the case of fraud detection, however, you would likely associate higher costs to false negatives than to false positives.\n",
+        "This trade off may be preferable because false negatives would allow fraudulent transactions to go through, whereas false positives may cause an email to be sent to a customer to ask them to verify their card activity.\n",
+        "\n",
+        "By decreasing the threshold, we attribute higher cost to false negatives, thereby increasing missed transactions at the price of more false positives.\n",
+        "We test thresholds at 10% and at 1%."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "52bd793e04bb"
+      },
+      "outputs": [],
+      "source": [
+        "plot_cm(test_labels, test_predictions_baseline, threshold=0.1)\n",
+        "plot_cm(test_labels, test_predictions_baseline, threshold=0.01)"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "P-QpQsip_F2Q"
       },
       "source": [
         "### Plot the ROC\n",
         "\n",
-        "Now plot the [ROC](https://developers.google.com/machine-learning/glossary#ROC). This plot is useful because it shows, at a glance, the range of performance the model can reach just by tuning the output threshold."
+        "Now plot the [ROC](https://developers.google.com/machine-learning/glossary#ROC). This plot is useful because it shows, at a glance, the range of performance the model can reach by tuning the output threshold over its full range (0 to 1). So each point corresponds to a single value of the threshold."
       ]
     },
     {
@@ -969,9 +1047,9 @@
         "id": "Y5twGRLfNwmO"
       },
       "source": [
-        "### Plot the AUPRC\r\n",
+        "### Plot the PRC\n",
         "\n",
-        "Now plot the [AUPRC](https://developers.google.com/machine-learning/glossary?hl=en#PR_AUC). Area under the interpolated precision-recall curve, obtained by plotting (recall, precision) points for different values of the classification threshold. Depending on how it's calculated, PR AUC may be equivalent to the average precision of the model.\r\n"
+        "Now plot the [AUPRC](https://developers.google.com/machine-learning/glossary?hl=en#PR_AUC). Area under the interpolated precision-recall curve, obtained by plotting (recall, precision) points for different values of the classification threshold. Depending on how it's calculated, PR AUC may be equivalent to the average precision of the model.\n"
       ]
     },
     {
@@ -982,14 +1060,14 @@
       },
       "outputs": [],
       "source": [
-        "def plot_prc(name, labels, predictions, **kwargs):\r\n",
-        "    precision, recall, _ = sklearn.metrics.precision_recall_curve(labels, predictions)\r\n",
-        "\r\n",
-        "    plt.plot(precision, recall, label=name, linewidth=2, **kwargs)\r\n",
-        "    plt.xlabel('Precision')\r\n",
-        "    plt.ylabel('Recall')\r\n",
-        "    plt.grid(True)\r\n",
-        "    ax = plt.gca()\r\n",
+        "def plot_prc(name, labels, predictions, **kwargs):\n",
+        "    precision, recall, _ = sklearn.metrics.precision_recall_curve(labels, predictions)\n",
+        "\n",
+        "    plt.plot(precision, recall, label=name, linewidth=2, **kwargs)\n",
+        "    plt.xlabel('Precision')\n",
+        "    plt.ylabel('Recall')\n",
+        "    plt.grid(True)\n",
+        "    ax = plt.gca()\n",
         "    ax.set_aspect('equal')"
       ]
     },
@@ -1001,8 +1079,8 @@
       },
       "outputs": [],
       "source": [
-        "plot_prc(\"Train Baseline\", train_labels, train_predictions_baseline, color=colors[0])\r\n",
-        "plot_prc(\"Test Baseline\", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')\r\n",
+        "plot_prc(\"Train Baseline\", train_labels, train_predictions_baseline, color=colors[0])\n",
+        "plot_prc(\"Test Baseline\", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')\n",
         "plt.legend(loc='lower right');"
       ]
     },
@@ -1032,7 +1110,7 @@
       "source": [
         "### Calculate class weights\n",
         "\n",
-        "The goal is to identify fraudulent transactions, but you don't have very many of those positive samples to work with, so you would want to have the classifier heavily weight the few examples that are available. You can do this by passing Keras weights for each class through a parameter. These will cause the model to \"pay more attention\" to examples from an under-represented class."
+        "The goal is to identify fraudulent transactions, but you don't have very many of those positive samples to work with, so you would want to have the classifier heavily weight the few examples that are available. You can do this by passing Keras weights for each class through a parameter. These will cause the model to \"pay more attention\" to examples from an under-represented class. Note, however, that this does not increase in any way the amount of information of your dataset. In the end, using class weights is more or less equivalent to changing the output bias or to changing the threshold. Let's see how it works out."
       ]
     },
     {
@@ -1153,7 +1231,9 @@
         "id": "PTh1rtDn8r4-"
       },
       "source": [
-        "Here you can see that with class weights the accuracy and precision are lower because there are more false positives, but conversely the recall and AUC are higher because the model also found more true positives. Despite having lower accuracy, this model has higher recall (and identifies more fraudulent transactions). Of course, there is a cost to both types of error (you wouldn't want to bug users by flagging too many legitimate transactions as fraudulent, either). Carefully consider the trade-offs between these different types of errors for your application."
+        "Here you can see that with class weights the accuracy and precision are lower because there are more false positives, but conversely the recall and AUC are higher because the model also found more true positives. Despite having lower accuracy, this model has higher recall (and identifies more fraudulent transactions than the baseline model at threshold 50%). Of course, there is a cost to both types of error (you wouldn't want to bug users by flagging too many legitimate transactions as fraudulent, either). Carefully consider the trade-offs between these different types of errors for your application.\n",
+        "\n",
+        "Compared to the baseline model with changed threshold, the class weighted model is clearly inferior. The superiority of the baseline model is further confirmed by the lower test loss value (cross entropy and mean squared error) and additionally can be seen by plotting the ROC curves of both models together."
       ]
     },
     {
@@ -1189,7 +1269,7 @@
         "id": "_0krS8g1OTbD"
       },
       "source": [
-        "### Plot the AUPRC"
+        "### Plot the PRC"
       ]
     },
     {
@@ -1200,13 +1280,13 @@
       },
       "outputs": [],
       "source": [
-        "plot_prc(\"Train Baseline\", train_labels, train_predictions_baseline, color=colors[0])\r\n",
-        "plot_prc(\"Test Baseline\", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')\r\n",
-        "\r\n",
-        "plot_prc(\"Train Weighted\", train_labels, train_predictions_weighted, color=colors[1])\r\n",
-        "plot_prc(\"Test Weighted\", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')\r\n",
-        "\r\n",
-        "\r\n",
+        "plot_prc(\"Train Baseline\", train_labels, train_predictions_baseline, color=colors[0])\n",
+        "plot_prc(\"Test Baseline\", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')\n",
+        "\n",
+        "plot_prc(\"Train Weighted\", train_labels, train_predictions_weighted, color=colors[1])\n",
+        "plot_prc(\"Test Weighted\", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')\n",
+        "\n",
+        "\n",
         "plt.legend(loc='lower right');"
       ]
     },
@@ -1581,7 +1661,6 @@
         "for name, value in zip(resampled_model.metrics_names, resampled_results):\n",
         "  print(name, ': ', value)\n",
         "print()\n",
-        "\n",
         "plot_cm(test_labels, test_predictions_resampled)"
       ]
     },
@@ -1604,10 +1683,8 @@
       "source": [
         "plot_roc(\"Train Baseline\", train_labels, train_predictions_baseline, color=colors[0])\n",
         "plot_roc(\"Test Baseline\", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')\n",
-        "\n",
         "plot_roc(\"Train Weighted\", train_labels, train_predictions_weighted, color=colors[1])\n",
         "plot_roc(\"Test Weighted\", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')\n",
-        "\n",
         "plot_roc(\"Train Resampled\", train_labels, train_predictions_resampled, color=colors[2])\n",
         "plot_roc(\"Test Resampled\", test_labels, test_predictions_resampled, color=colors[2], linestyle='--')\n",
         "plt.legend(loc='lower right');"