updates up to report deliverable

Author: avaamini

lab2/solutions/Part2_Debiasing_Solution.ipynb

@@ -534,7 +534,7 @@
         "\n",
         "We will apply our SS-VAE to a *supervised classification* problem -- the facial detection task. Importantly, note how the encoder portion in the SS-VAE architecture also outputs a single supervised variable, $z_o$, corresponding to the class prediction -- face or not face. Usually, VAEs are not trained to output any supervised variables (such as a class prediction)! This is the key distinction between the SS-VAE and a traditional VAE. \n",
         "\n",
-        "Keep in mind that we only want to learn the latent representation of *faces*, as that is where we are interested in uncovering potential biases, even though we are training a model on a binary classification problem. So, we will need to ensure that, **for faces**, our SS-VAE model both learns a representation of the unsupervised latent variables, captured by the distribution $q_\\phi(z|x)$, **and** outputs a supervised class prediction $z_o$, but that, **for negative examples**, it only outputs a class prediction $z_o$."
+        "Keep in mind that we only want to learn the latent representation of *faces*, as that is where we are interested in uncovering potential biases, even though we are training a model on a binary classification problem. So, we will need to ensure that, **for faces**, our SS-VAE model both learns a representation of the unsupervised latent variables, captured by the distribution $q_\\phi(z|x)$, and outputs a supervised class prediction $z_o$, but that, **for negative examples**, it only outputs a class prediction $z_o$."
       ],
       "metadata": {
         "id": "A3IOB3d61WSN"
@@ -840,38 +840,27 @@
         "id": "Eo34xC7MbaiQ"
       },
       "source": [
-        "## 2.6 Evaluation of Supervised VAE on Test Dataset\n",
+        "## 2.6 Using the SS-VAE to uncover and diagnose biases\n",
         "\n",
-        "Finally let's test our model on the test dataset, looking specifically at its accuracy on each the \"Dark Male\", \"Dark Female\", \"Light Male\", and \"Light Female\" demographics. We will compare the performance of this debiased model against the (potentially biased) standard CNN from earlier in the lab."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "bgK77aB9oDtX"
-      },
-      "outputs": [],
-      "source": [
-        "dbvae_logits = [dbvae.predict(np.array(x, dtype=np.float32)) for x in test_faces]\n",
-        "dbvae_probs = tf.squeeze(tf.sigmoid(dbvae_logits))\n",
-        "\n",
-        "xx = np.arange(len(keys))\n",
-        "plt.bar(xx, standard_classifier_probs.numpy().mean(1), width=0.2, label=\"Standard CNN\")\n",
-        "plt.bar(xx+0.2, dbvae_probs.numpy().mean(1), width=0.2, label=\"Supervised VAE\")\n",
-        "plt.xticks(xx, keys); \n",
-        "plt.title(\"Network predictions on test dataset\")\n",
-        "plt.ylabel(\"Probability\"); plt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")"
+        "With the SS-VAE model trained, we are ready to use it to uncover and diagnose hidden biases that exist within the dataset.\n",
+        "\n",
+        "Recall that our goal with the SS-VAE was to learn the underlying ***latent distribution*** of features in the training dataset, in order to uncover potential feature representation disparities that exist within the data.\n",
+        "\n",
+        "Additionally, training the SS-VAE required both a VAE reconstruction loss as well as a supervised classification loss. The VAE reconstruction loss direclty reflects how well the model is able to handle particular input data -- the higher the reconstruction loss, the harder that particular example is for the model to learn.\n",
+        "\n",
+        "We consider both these aspects to understand sources of uncertainty and bias within the model."
       ]
     },
     {
       "cell_type": "markdown",
-      "metadata": {
-        "id": "hdGHO9FHtMyx"
-      },
       "source": [
-        "Now, let's investigate how well the VAE actually learned the latent features of the faces! To do this, we'll look at the examples in the test dataset with the highest loss. What can you tell about which features seemed harder to learn for the VAE? What might this tell us about how the model is biased?"
-      ]
+        "### Linking model performance to uncertainty and bias\n",
+        "\n",
+        "We begin by considering the examples in the test dataset with the highest loss. What can you tell about which features seemed harder to learn for the VAE? What might this tell us about where the model struggles, and what predictions it may be more biased or uncertain about?"
+      ],
+      "metadata": {
+        "id": "QfVngr5J6sj3"
+      }
     },
     {
       "cell_type": "code",
@@ -881,7 +870,7 @@
       },
       "outputs": [],
       "source": [
-        "# Load a random sample of 2000 faces from our dataset and compute the model performance on them\n",
+        "# Load a random sample of 5000 faces from our dataset and compute the model performance on them\n",
         "(x, y) = loader.get_batch(5000, only_faces=True)\n",
         "y_logit, z_mean, z_logsigma, x_recon = dbvae(x)\n",
         "loss, class_loss, vae_loss = debiasing_loss_function(x, x_recon, y, y_logit, z_mean, z_logsigma)\n",
@@ -922,9 +911,13 @@
         "id": "8SQSszTFjstZ"
       },
       "source": [
-        "## 2.7 Calculating Bias Scores\n",
+        "### Uncovering hidden biases through learned latent features \n",
         "\n",
-        "As we've seen above, loss is a powerful way to visualize which samples in our dataset have high *uncertainty*, or which ones the model has had trouble learning. However, this isn't necessarily the same as bias! How can determine the *probability* of a sample occurring in our dataset, and debias based off of that? In this section, we'll develop a way to score samples based on their bias and adapt this score during training."
+        "As we've seen above, loss is a powerful way to visualize which samples in our dataset the model has had trouble learning -- these examples are those that have high *model uncertainty*. However, this is not necessarily the same as bias!\n",
+        "\n",
+        "How can we determine the relative frequencies and distributions of different latent features learned by the model? How may these metrics reveal underlying biases?\n",
+        "\n",
+        "Let's investigate how well the SS-VAE actually learned the latent features of the faces. To do this, we will inspect individual latent features -- holding all others constant -- and look at the distribution of these features in the data and their corresponding examples. We can examine how the shape and probability density of the learned latent features."
       ]
     },
     {
@@ -960,6 +953,19 @@
       "execution_count": null,
       "outputs": []
     },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Carefully inspect the different latent variables and their corresponding frequency distributions. What can you tell about which features are under- or over-represented in the data? What might this tell us about how the model is biased?\n",
+        "\n",
+        "How do these feature distribution differences affect classification performance? In addition to these qualitative inspections, we can directly compare different values of individual latent variables to corresponding relative classification accuracies (marginalizing out the effects of the other latent variables).\n",
+        "\n",
+        "What trends do you observe with this evaluation? How does this affect your understanding of the bias of the facial detection classifier?"
+      ],
+      "metadata": {
+        "id": "y97C5Qsh8GvB"
+      }
+    },
     {
       "cell_type": "code",
       "source": [