ENH - Final tweaks to theoretical_validity_tests.ipynb

Author: Valentin-Laurent

notebooks/risk_control/risk_control_theoretical_tests_proto.ipynb renamed to notebooks/risk_control/theoretical_validity_tests.ipynb

@@ -3,9 +3,29 @@
   {
    "metadata": {},
    "cell_type": "markdown",
-   "source": "# Binary classification risk control - Theoretical tests prototype",
+   "source": "# Binary classification risk control - Theoretical tests to validate implementation",
    "id": "ed592eb3f8989aa8"
   },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "# Protocol description\n",
+    "Testing theoretical guarantees of risk control in binary classification using a random classifier and synthetic data.\n",
+    "\n",
+    "Each test case looks at a combination of parameters, for which we repeat the experiment `n_repeat` times. The model is the same for all experiments (basically a random classifier), but the data is different each time.\n",
+    "\n",
+    "Each experiment consists of the following:\n",
+    " - We calibrate a BinaryClassificationController. It gives us the list of lambda values that control the risk according to LTT.\n",
+    " - Because we know that the model is random, we know the theoretical risk associated with each lambda value. So we are able to check if the lambda values given by LTT actually control the risk. If not, we count 1 \"error\". Note that *each* lambda value should control the risk, not just one of them.\n",
+    "\n",
+    "After n_repeat experiments, we compute the proportion of errors, that should be less than delta (1 - confidence_level).\n",
+    "\n",
+    "# Results\n",
+    "The risk is controlled in all the test cases. Overall, LTT seems very conservative (to achieve a high percentage of errors, we need to lower the confidence level significantly (0.01) and use only one threshold to avoid the Bonferroni effect). But this is likely due to the model being random, and thus having a lot of variance. It would be interesting to see how this evolves with a better model."
+   ],
+   "id": "8c1746b673c148dd"
+  },
   {
    "metadata": {
     "ExecuteTime": {
@@ -51,7 +71,7 @@
    },
    "cell_type": "code",
    "source": [
-    "# Using sklearn.dummy.DummyClassifier would be clearer\n",
+    "# Using sklearn.dummy.DummyClassifier would be cleaner\n",
     "class RandomClassifier:\n",
     "    def __init__(self, seed=42, threshold=0.5):\n",
     "        self.seed = seed\n",
@@ -131,7 +151,9 @@
     "        valid_parameters = controller.valid_thresholds\n",
     "        total_nb_valid_params += len(valid_parameters)\n",
     "\n",
-    "        # The following works because the data is balanced\n",
+    "        # In the following, we check that all the valid thresholds found by LTT actually control the risk.\n",
+    "        # Instead of sampling a large test set, we use the fact that we know the theoretical risk of a random classifier.\n",
+    "        # The calculations here are valid only for a balanced data generator.\n",
     "        if risk[\"risk\"] == precision or risk[\"risk\"] == accuracy:\n",
     "            if target_level > 0.5 and len(valid_parameters) >= 1:\n",
     "                nb_errors += 1\n",