Skip to content

Commit b1c0267

Browse files
avaaminiavaamini
committed
part2 up to bias identifying
1 parent 109b965 commit b1c0267

File tree

1 file changed

+71
-34
lines changed

1 file changed

+71
-34
lines changed

‎lab3/solutions/Lab3_Bias_And_Uncertainty.ipynb

Lines changed: 71 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,60 @@
11
{
22
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"source": [
6+
"<table align=\"center\">\n",
7+
" <td align=\"center\"><a target=\"_blank\" href=\"http://introtodeeplearning.com\">\n",
8+
" <img src=\"https://i.ibb.co/Jr88sn2/mit.png\" style=\"padding-bottom:5px;\" />\n",
9+
" Visit MIT Deep Learning</a></td>\n",
10+
" <td align=\"center\"><a target=\"_blank\" href=\"https://colab.research.google.com/github/aamini/introtodeeplearning/blob/2023/lab3/solutions/Lab3_Part_1_Introduction_to_CAPSA.ipynb\">\n",
11+
" <img src=\"https://i.ibb.co/2P3SLwK/colab.png\" style=\"padding-bottom:5px;\" />Run in Google Colab</a></td>\n",
12+
" <td align=\"center\"><a target=\"_blank\" href=\"https://github.com/aamini/introtodeeplearning/blob/2023/lab3/solutions/Lab3_Part_1_Introduction_to_CAPSA.ipynb\">\n",
13+
" <img src=\"https://i.ibb.co/xfJbPmL/github.png\" height=\"70px\" style=\"padding-bottom:5px;\" />View Source on GitHub</a></td>\n",
14+
"</table>\n",
15+
"\n",
16+
"# Copyright Information"
17+
],
18+
"metadata": {
19+
"id": "Kxl9-zNYhxlQ"
20+
}
21+
},
22+
{
23+
"cell_type": "code",
24+
"source": [
25+
"# Copyright 2023 MIT Introduction to Deep Learning. All Rights Reserved.\n",
26+
"# \n",
27+
"# Licensed under the MIT License. You may not use this file except in compliance\n",
28+
"# with the License. Use and/or modification of this code outside of MIT Introduction\n",
29+
"# to Deep Learning must reference:\n",
30+
"#\n",
31+
"# © MIT Introduction to Deep Learning\n",
32+
"# http://introtodeeplearning.com\n",
33+
"#"
34+
],
35+
"metadata": {
36+
"id": "aAcJJN3Xh3S1"
37+
},
38+
"execution_count": null,
39+
"outputs": []
40+
},
341
{
442
"cell_type": "markdown",
543
"metadata": {
644
"id": "IgYKebt871EK"
745
},
846
"source": [
9-
"# Laboratory 3: Detecting and mitigating bias and uncertainty in Facial Detection Systems\n",
10-
"In this lab, we'll continue to explore how to mitigate algorithmic bias in facial recognition systems. In addition, we'll explore the notion of *uncertainty* in datasets, and learn how to reduce both data-based and model-based uncertainty.\n",
47+
"# Laboratory 3: Debiasing, Uncertainty, and Robustness\n",
1148
"\n",
12-
"As we've seen in lecture 5, bias and uncertainty underlie many common issues with machine learning models today, and these are not just limited to classification tasks. Automatically detecting and mitigating uncertainty is crucial to deploying fair and safe models. \n",
49+
"# Part 2: Mitigating Bias and Uncertainty in Facial Detection Systems\n",
1350
"\n",
14-
"In this lab, we'll be using [CAPSA](https://github.com/themis-ai/capsa/), a software package developed by [Themis AI](https://themisai.io/), which automatically *wraps* models to make them risk-aware and plugs into training workflows. We'll explore how we can use CAPSA to diagnose uncertainties, and then develop methods for automatically mitigating them.\n",
51+
"In Lab 2, we defined a semi-supervised VAE (SS-VAE) to diagnose feature representation disparities and biases in facial detection systems. In Lab 3 Part 1, we gained experience with Capsa and its ability to build risk-aware models automatically through wrapping. Now in this lab, we will put these two together: using Capsa to build systems that can *automatically* uncover and mitigate bias and uncertainty in facial detection systems.\n",
1552
"\n",
53+
"As we have seen, automatically detecting and mitigating bias and uncertainty is crucial to deploying fair and safe models. Building off our foundation with [Capsa](https://github.com/themis-ai/capsa/), developed by [Themis AI](https://themisai.io/), we will now use Capsa for the facial detection problem, in order to diagnose risks in facial detection models. You will then design and create strategies to mitigate these risks, with goal of improving model performance across the entire facial detection dataset.\n",
1654
"\n",
17-
"Run the next code block for a short video from Google that explores how and why it's important to consider bias when thinking about machine learning:"
55+
"**Your goal in this lab -- and the associated competition -- is to design a strategic solution for bias and uncertainty mitigation, using Capsa.** The approaches and solutions with oustanding performance will be recognized with outstanding prizes! Details on the submission process are at the end of this lab.\n",
56+
"\n",
57+
"![Capsa overview](https://raw.githubusercontent.com/aamini/introtodeeplearning/2023/lab3/img/capsa_overview.png)"
1858
]
1959
},
2060
{
@@ -23,7 +63,7 @@
2363
"id": "6JTRoM7E71EU"
2464
},
2565
"source": [
26-
"Let's get started by installing the relevant dependencies:"
66+
"Let's get started by installing the necessary dependencies:"
2767
]
2868
},
2969
{
@@ -40,7 +80,7 @@
4080
"base_uri": "https://localhost:8080/"
4181
}
4282
},
43-
"execution_count": 1,
83+
"execution_count": null,
4484
"outputs": [
4585
{
4686
"output_type": "stream",
@@ -74,7 +114,7 @@
74114
},
75115
{
76116
"cell_type": "code",
77-
"execution_count": 17,
117+
"execution_count": null,
78118
"metadata": {
79119
"id": "2PdAhs1371EU",
80120
"outputId": "dd327495-e85d-4849-9487-4f71535b6cae",
@@ -118,22 +158,21 @@
118158
"id": "6VKVqLb371EV"
119159
},
120160
"source": [
121-
"## 3.1 Datasets\n",
161+
"# 3.1 Datasets\n",
122162
"\n",
123-
"We'll be using the same datasets from lab 2 in this lab. Note that in this dataset, we've intentionally perturbed some of the samples in some ways (it's up to you to figure out how!) that are not necessarily present in the actual dataset. \n",
163+
"Since we are again focusing on the facial detection problem, we will use the same datasets from Lab 2. To remind you, we have a dataset of positive examples (i.e., of faces) and a dataset of negative examples (i.e., of things that are not faces).\n",
124164
"\n",
125-
"1. **Positive training data**: [CelebA Dataset](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). A large-scale (over 200K images) of celebrity faces. \n",
126-
"2. **Negative training data**: [ImageNet](http://www.image-net.org/). Many images across many different categories. We'll take negative examples from a variety of non-human categories. \n",
127-
"[Fitzpatrick Scale](https://en.wikipedia.org/wiki/Fitzpatrick_scale) skin type classification system, with each image labeled as \"Lighter'' or \"Darker''.\n",
165+
"1. **Positive training data**: [CelebA Dataset](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). A large-scale dataset (over 200K images) of celebrity faces. \n",
166+
"2. **Negative training data**: [ImageNet](http://www.image-net.org/). A large-scale dataset with many images across many different categories. We will take negative examples from a variety of non-human categories.\n",
128167
"\n",
129-
"Like before, let's begin by importing these datasets. We've written a class that does a bit of data pre-processing to import the training data in a usable format.\n",
168+
"We will evaluate trained models on an independent test dataset of face images to diagnose and mitigate potential issues with *bias, fairness, and confidence*. This will be a larger test dataset for evaluation purposes.\n",
130169
"\n",
131-
"Also note that in this lab, we'll be using a much larger test dataset for evaluation purposes."
170+
"We begin by importing these datasets. We have defined a `DatasetLoader` class that does a bit of data pre-processing to import the training data in a usable format."
132171
]
133172
},
134173
{
135174
"cell_type": "code",
136-
"execution_count": 6,
175+
"execution_count": null,
137176
"metadata": {
138177
"id": "HIA6EA1D71EW",
139178
"outputId": "162f7d36-81aa-4fb9-b265-af9a8fa9395d",
@@ -173,11 +212,9 @@
173212
"source": [
174213
"### Recap: Thinking about bias and uncertainty\n",
175214
"\n",
176-
"Remember that we'll be training our facial detection classifiers on the large, well-curated CelebA dataset (and ImageNet), and then evaluating their accuracy by testing them on an independent test dataset. Our goal is to build a model that trains on CelebA *and* achieves high classification accuracy on the the test dataset across all demographics, and to thus show that this model does not suffer from any hidden bias. \n",
177-
"\n",
178-
"In addition to thinking about bias, we want to detect areas of high *aleatoric* uncertainty in the dataset, which is defined as data noise: in the context of facial detection, this means that we may have very similar inputs with different labels-- think about the scenario where one face is labeled correctly as a positive, and another face is labeled incorrectly as a negative. \n",
215+
"Remember that we'll be training our facial detection classifiers on the large, well-curated CelebA dataset (and ImageNet), and then evaluating their accuracy by testing them on an independent test dataset. Our goal is to build a model that trains on CelebA *and* achieves high classification accuracy on the the test dataset across all demographics. We want to mitigate the effects of unwanted bias and uncertainty on the model's predictions and performance. and to thus show that this model does not suffer from any hidden bias.\n",
179216
"\n",
180-
"Finally, we want to look at samples with high *epistemic*, or predictive, uncertainty. These may be samples that are anomalous or out of distribution, samples that contain adversarial noise, or samples that are \"harder\" to learn in some way. Importantly, epistemic uncertainty is not the same as bias! We may have well-represented samples that still have high epistemic uncertainty. "
217+
"You may consider the three metrics introduced with Capsa: (1) representation bias, (2) data or aleatoric uncertainty, and (3) model or epistemic uncertainty. Note that all three of these metrics are different! For example, we can have well-represented examples that still have high epistemic uncertainty. "
181218
]
182219
},
183220
{
@@ -202,7 +239,7 @@
202239
},
203240
{
204241
"cell_type": "code",
205-
"execution_count": 50,
242+
"execution_count": null,
206243
"metadata": {
207244
"id": "5hQb75Vm71EZ"
208245
},
@@ -261,7 +298,7 @@
261298
},
262299
{
263300
"cell_type": "code",
264-
"execution_count": 51,
301+
"execution_count": null,
265302
"metadata": {
266303
"id": "zTat3K8E71Eb"
267304
},
@@ -294,7 +331,7 @@
294331
},
295332
{
296333
"cell_type": "code",
297-
"execution_count": 52,
334+
"execution_count": null,
298335
"metadata": {
299336
"id": "i4JmvmMA71Ec"
300337
},
@@ -315,7 +352,7 @@
315352
},
316353
{
317354
"cell_type": "code",
318-
"execution_count": 53,
355+
"execution_count": null,
319356
"metadata": {
320357
"id": "NmshVdLM71Ed",
321358
"outputId": "873a0931-c1ae-43cc-f8f0-53b11f30ae4a",
@@ -387,7 +424,7 @@
387424
},
388425
{
389426
"cell_type": "code",
390-
"execution_count": 55,
427+
"execution_count": null,
391428
"metadata": {
392429
"id": "1dCqvPFH71Ed",
393430
"outputId": "a3a19ee7-5c8e-4563-d6df-709f39dd357d",
@@ -420,7 +457,7 @@
420457
},
421458
{
422459
"cell_type": "code",
423-
"execution_count": 14,
460+
"execution_count": null,
424461
"metadata": {
425462
"id": "OYMRqq5E71Ee"
426463
},
@@ -434,7 +471,7 @@
434471
},
435472
{
436473
"cell_type": "code",
437-
"execution_count": 56,
474+
"execution_count": null,
438475
"metadata": {
439476
"id": "UAYaFUj-71Ee",
440477
"outputId": "39f4fe7d-ca14-47f4-d229-97d868dcc182",
@@ -473,7 +510,7 @@
473510
},
474511
{
475512
"cell_type": "code",
476-
"execution_count": 57,
513+
"execution_count": null,
477514
"metadata": {
478515
"id": "CnbR3qAF71Ef",
479516
"outputId": "6a84a235-88b6-41e1-b616-17c3f9a0096b",
@@ -521,7 +558,7 @@
521558
},
522559
{
523560
"cell_type": "code",
524-
"execution_count": 58,
561+
"execution_count": null,
525562
"metadata": {
526563
"id": "DfzlOhWi71Ef",
527564
"outputId": "65e20e7a-5e3e-4452-ae49-559a33a063ba",
@@ -580,7 +617,7 @@
580617
"height": 153
581618
}
582619
},
583-
"execution_count": 33,
620+
"execution_count": null,
584621
"outputs": [
585622
{
586623
"output_type": "execute_result",
@@ -632,7 +669,7 @@
632669
},
633670
{
634671
"cell_type": "code",
635-
"execution_count": 59,
672+
"execution_count": null,
636673
"metadata": {
637674
"id": "AwGPvdZm71Eg"
638675
},
@@ -646,7 +683,7 @@
646683
},
647684
{
648685
"cell_type": "code",
649-
"execution_count": 60,
686+
"execution_count": null,
650687
"metadata": {
651688
"id": "kB8Iqrfb71Eg",
652689
"outputId": "dc16b7d3-a6f1-4542-dcf2-135e0737e80e",
@@ -685,7 +722,7 @@
685722
},
686723
{
687724
"cell_type": "code",
688-
"execution_count": 62,
725+
"execution_count": null,
689726
"metadata": {
690727
"id": "miu5h2Pc71Eh",
691728
"outputId": "00442875-f1bb-4160-d11c-8cfdfb5765ac",
@@ -733,7 +770,7 @@
733770
},
734771
{
735772
"cell_type": "code",
736-
"execution_count": 63,
773+
"execution_count": null,
737774
"metadata": {
738775
"id": "rzQwvSvA71Eh",
739776
"outputId": "4299761d-22b3-4f44-b9b3-d9194d04a6f2",

0 commit comments

Comments
 (0)