ENH - First iteration of the new binary risk control feature (#749)

See PR description for detailed content.

Author: Valentin-Laurent

.gitignore

@@ -13,7 +13,7 @@ doc/_build/
 doc/examples_classification/
 doc/examples_regression/
 doc/examples_calibration/
-doc/examples_multilabel_classification/
+doc/examples_risk_control/
 doc/examples_mondrian/
 doc/auto_examples/
 doc/modules/generated/

AUTHORS.rst

@@ -5,17 +5,17 @@ Credits
 Development Lead
 ----------------
 
-* Thibault Cordier <[email protected]>
 * Vincent Blot <[email protected]>
-* Louis Lacombe <[email protected]>
 * Valentin Laurent <[email protected]>
-* Hussein Jawad <[email protected]>
+* Adrien Le Coz <[email protected]>
 
 Emeritus Core Developers
 ------------------------
 
 * Grégoire Martinon <[email protected]>
 * Vianney Taquet <[email protected]>
+* Thibault Cordier <[email protected]>
+* Louis Lacombe <[email protected]>
 
 Contributors
 ------------
@@ -43,6 +43,7 @@ Contributors
 * Ambros Marzetta <ambrosm>
 * Carl McBride Ellis <Carl-McBride-Ellis>
 * Baptiste Calot <[email protected]>
+* Hussein Jawad <[email protected]>
 * Leonardo Garma <[email protected]>
 * Mohammed Jawhar <[email protected]>
 * Syed Affan <[email protected]>

HISTORY.rst

@@ -16,6 +16,7 @@ History
 * MAPIE now supports Python versions up to the latest release (currently 3.13)
 * Change `prefit` default value to `True` in split methods' docstrings to remain consistent with the implementation
 * Fix issue 699 to replace `TimeSeriesRegressor.partial_fit` with `TimeSeriesRegressor.update`
+* Revert incorrect renaming of calibration to conformalization in risk_control.py
 
 1.0.1 (2025-05-22)
 ------------------

README.rst

@@ -60,7 +60,7 @@ MAPIE relies notably on the fields of Conformal Prediction and Distribution-Free
 
 MAPIE runs on:
 
-- Python >=3.9, <3.12
+- Python >=3.9
 - NumPy >=1.23
 - scikit-learn >=1.4
 

doc/Makefile

@@ -50,7 +50,7 @@ clean:
 	-rm -rf $(BUILDDIR)/*
 	-rm -rf examples_regression/
 	-rm -rf examples_classification/
-	-rm -rf examples_multilabel_classification/
+	-rm -rf examples_risk_control/
 	-rm -rf examples_calibration/
 	-rm -rf examples_mondrian/
 	-rm -rf generated/*

doc/api.rst

@@ -102,6 +102,8 @@ Risk Control
    :template: class.rst
 
    mapie.risk_control.PrecisionRecallController
+   mapie.risk_control.BinaryClassificationController
+   mapie.risk_control.BinaryClassificationRisk
 
 Calibration
 ===========

doc/conf.py

@@ -321,14 +321,14 @@
     "examples_dirs": [
         "../examples/regression",
         "../examples/classification",
-        "../examples/multilabel_classification",
+        "../examples/risk_control",
         "../examples/calibration",
         "../examples/mondrian",
     ],
     "gallery_dirs": [
         "examples_regression",
         "examples_classification",
-        "examples_multilabel_classification",
+        "examples_risk_control",
         "examples_calibration",
         "examples_mondrian",
     ],

doc/index.rst

@@ -24,7 +24,8 @@
    :caption: Control prediction errors
 
    theoretical_description_risk_control
-   examples_multilabel_classification/1-quickstart/plot_tutorial_risk_control
+   examples_risk_control/1-quickstart/plot_risk_control_binary_classification
+   examples_risk_control/index
    external_risk_control_package
 
 .. toctree::

doc/quick_start.rst

@@ -40,4 +40,10 @@ Here, we generate one-dimensional noisy data that we fit with a MLPRegressor: `U
 3. Classification
 =======================
 
-Similarly, it's possible to do the same for a basic classification problem: `Use MAPIE to plot prediction sets <https://mapie.readthedocs.io/en/stable/examples_classification/1-quickstart/plot_quickstart_classification.html>`_
+Similarly, it's possible to do the same for a basic classification problem: `Use MAPIE to plot prediction sets <https://mapie.readthedocs.io/en/stable/examples_classification/1-quickstart/plot_quickstart_classification.html>`_
+
+
+4. Risk Control
+=======================
+
+MAPIE implements risk control methods for multilabel classification (in particular, image segmentation) and binary classification: `Use MAPIE to control risk for a binary classifier <https://mapie.readthedocs.io/en/stable/examples_risk_control/1-quickstart/plot_risk_control_binary_classification.html>`_

doc/theoretical_description_risk_control.rst

@@ -13,26 +13,43 @@ Getting started with risk control in MAPIE
 Overview
 ========
 
+This section provides an overview of risk control in MAPIE. For those unfamiliar with the concept of risk control, the next section provides an introduction to the topic.
+
 Three methods of risk control have been implemented in MAPIE so far :
 **Risk-Controlling Prediction Sets** (RCPS) [1], **Conformal Risk Control** (CRC) [2] and **Learn Then Test** (LTT) [3].
-The difference between these methods is the way the conformity scores are computed.
 
-As of now, MAPIE supports risk control for two machine learning tasks: **binary classification**, as well as **multi-label classification** (including applications like image segmentation).
+As of now, MAPIE supports risk control for two machine learning tasks: **binary classification**, as well as **multi-label classification** (in particular applications like image segmentation).
 The table below details the available methods for each task:
 
+.. |br| raw:: html
+
+   <br />
+   
 .. list-table:: Available risk control methods in MAPIE for each ML task
    :header-rows: 1
 
-   * - Risk control method
-     - Binary classification
-     - Multi-label classification (image segmentation)
+   * - Risk control |br| method
+     - Type of |br| control
+     - Assumption |br| on the data
+     - Non-monotonic |br| risks
+     - Binary |br| classification
+     - Multi-label |br| classification
    * - RCPS
+     - Probability
+     - i.i.d.
+     - ❌
      - ❌
      - ✅
    * - CRC
+     - Expectation
+     - Exchangeable
+     - ❌
      - ❌
      - ✅
    * - LTT
+     - Probability
+     - i.i.d
+     - ✅
      - ✅
      - ✅
 
@@ -41,7 +58,7 @@ In MAPIE for multi-label classification, CRC and RCPS are used for recall contro
 1. What is risk control?
 ========================
 
-Before diving into risk control, let's take the simple example of a binary classification model, which separates the incoming data into the two classes thanks to its threshold: predictions above it are classified as 1, and those below as 0. Suppose we want to find a threshold that guarantees that our model achieves a certain level of precision. A naive, yet straightforward approach to do this is to evaluate how precision varies with different threshold values on a validation dataset. By plotting this relationship (see plot below), we can identify the range of thresholds that meet our desired precision requirement (green zone on the graph).
+Before diving into risk control, let's take the simple example of a binary classification model, which separates the incoming data into two classes. Predicted probabilities above a given threshold (e.g., 0.5) correspond to predicting the "positive" class and probabilities below correspond to the "negative" class. Suppose we want to find a threshold that guarantees that our model achieves a certain level of precision. A naive, yet straightforward approach to do this is to evaluate how precision varies with different threshold values on a validation dataset. By plotting this relationship (see plot below), we can identify the range of thresholds that meet our desired precision requirement (green zone on the graph).
 
 .. image:: images/example_without_risk_control.png
    :width: 600
@@ -54,7 +71,7 @@ So far, so good. But here is the catch: while the chosen threshold effectively k
 Risk control is the science of adjusting a model's parameter, typically denoted :math:`\lambda`, so that a given risk stays below a desired level with high probability on unseen data.
 Note that here, the term *risk* is used to describe an undesirable outcome of the model (e.g., type I error): therefore, it is a value we want to minimize, and in our case, keep under a certain level. Also note that risk control can easily be applied to metrics we want to maximize (e.g., precision), simply by controlling the complement (e.g., 1-precision).
 
-The strength of risk control lies in the statistical guarantees it provides on unseen data. Unlike the naive method presented earlier, it determines a value of :math:`\lambda` that ensures the risk is controlled *beyond* the training data.
+The strength of risk control lies in the statistical guarantees it provides on unseen data. Unlike the naive method presented earlier, it determines a value of :math:`\lambda` that ensures the risk is controlled *beyond* the validation data.
 
 Applying risk control to the previous example would allow us to get a new — albeit narrower — range of thresholds (blue zone on the graph) that are **statistically guaranteed**.
 
@@ -66,7 +83,7 @@ This guarantee is critical in a wide range of use cases (especially in high-stak
 
 —
 
-To express risk control in mathematical terms, we denote by R the risk we want to control, and introduce the following two parameters:
+To express risk control in mathematical terms, we denote by :math:`R` the risk we want to control, and introduce the following two parameters:
 
 - :math:`\alpha`: the target level below which we want the risk to remain, as shown in the figure below;
 
@@ -76,13 +93,13 @@ To express risk control in mathematical terms, we denote by R the risk we want t
 
 - :math:`\delta`: the confidence level associated with the risk control.
 
-In other words, the risk is said to be controlled if :math:`R \leq \alpha` with probability at least :math:`1 - \delta`.
+In other words, the risk is said to be controlled if :math:`R \leq \alpha` with probability at least :math:`1 - \delta`, where the probability is over the randomness in the sampling of the dataset.
 
 The three risk control methods implemented in MAPIE — RCPS, CRC and LTT — rely on different assumptions, and offer slightly different guarantees:
 
 - **CRC** requires the data to be **exchangeable**, and gives a guarantee on the **expectation of the risk**: :math:`\mathbb{E}(R) \leq \alpha`;
 
-- **RCPS** and **LTT** both impose stricter assumptions, requiring the data to be **independent and identically distributed** (i.i.d.), which implies exchangeability. The guarantee they provide is on the **probability that the risk does not exceed :math:`\alpha`**: :math:`\mathbb{P}(R \leq \alpha) \geq 1 - \delta`.
+- **RCPS** and **LTT** both impose stricter assumptions, requiring the data to be **independent and identically distributed** (i.i.d.), which implies exchangeability. The guarantee they provide is on the **probability that the risk does not exceed** :math:`\boldsymbol{\alpha}`: :math:`\mathbb{P}(R \leq \alpha) \geq 1 - \delta`.
 
 .. image:: images/risk_distribution.png
    :width: 600
@@ -94,12 +111,13 @@ The plot above gives a visual representation of the difference between the two t
 
 - The risk is controlled in probability (RCPS/LTT) if at least :math:`1 - \delta` percent of its distribution over unseen data is below :math:`\alpha`.
 
-Note that at the opposite of the other two methods, LTT allows to control any non-monotonic risk.
+Note that contrary to the other two methods, LTT allows to control any non-monotonic risk.
 
 The following section provides a detailed overview of each method.
 
 2. Theoretical description
 ==========================
+
 2.1 Risk-Controlling Prediction Sets
 ------------------------------------
 2.1.1 General settings
@@ -234,7 +252,7 @@ We are going to present the Learn Then Test framework that allows the user to co
 This method has been introduced in article [3].
 The settings here are the same as RCPS and CRC, we just need to introduce some new parameters:
 
-- Let :math:`\Lambda` be a discretized for our :math:`\lambda`, meaning that :math:`\Lambda = \{\lambda_1, ..., \lambda_n\}`.
+- Let :math:`\Lambda` be a discretized set for our :math:`\lambda`, meaning that :math:`\Lambda = \{\lambda_1, ..., \lambda_n\}`.
 
 - Let :math:`p_\lambda` be a valid p-value for the null hypothesis :math:`\mathbb{H}_j: R(\lambda_j)>\alpha`.
 
@@ -250,7 +268,7 @@ In order to find all the parameters :math:`\lambda` that satisfy the above condi
   :math:`\{(x_1, y_1), \dots, (x_n, y_n)\}`.
 
 - For each :math:`\lambda_j` in a discrete set :math:`\Lambda = \{\lambda_1, \lambda_2,\dots, \lambda_n\}`, we associate the null hypothesis
-  :math:`\mathcal{H}_j: R(\lambda_j) > \alpha`, as rejecting the hypothesis corresponds to selecting :math:`\lambda_j` as a point where risk the risk
+  :math:`\mathcal{H}_j: R(\lambda_j) > \alpha`, as rejecting the hypothesis corresponds to selecting :math:`\lambda_j` as a point where the risk
   is controlled.
 
 - For each null hypothesis, we compute a valid p-value using a concentration inequality :math:`p_{\lambda_j}`. Here we choose to compute the Hoeffding-Bentkus p-value
@@ -259,6 +277,7 @@ In order to find all the parameters :math:`\lambda` that satisfy the above condi
 - Return :math:`\hat{\Lambda} =  \mathcal{A}(\{p_j\}_{j\in\{1,\dots,\lvert \Lambda \rvert})`, where :math:`\mathcal{A}`, is an algorithm
   that controls the family-wise error rate (FWER), for example, Bonferonni correction.
 
+Note that a notebook testing theoretical guarantees of risk control in binary classification using a random classifier and synthetic data is available here: `theoretical_validity_tests.ipynb <https://github.com/scikit-learn-contrib/MAPIE/tree/master/notebooks/risk_control/theoretical_validity_tests.ipynb>`__.
 
 References
 ==========