Merge branch 'master' into 477-mapieregressor-with-prefit-optimized-model-that-used-training-and-calibration-data

Author: LacombeLouis

.github/PULL_REQUEST_TEMPLATE.md

@@ -22,8 +22,8 @@ Please describe the tests that you ran to verify your changes. Provide instructi
 
 # Checklist
 
-- [ ] I have read the [contributing guidelines](https://github.com/simai-ml/MAPIE/blob/master/CONTRIBUTING.rst)
-- [ ] I have updated the [HISTORY.rst](https://github.com/simai-ml/MAPIE/blob/master/HISTORY.rst) and [AUTHORS.rst](https://github.com/simai-ml/MAPIE/blob/master/AUTHORS.rst) files
+- [ ] I have read the [contributing guidelines](https://github.com/scikit-learn-contrib/MAPIE/blob/master/CONTRIBUTING.rst)
+- [ ] I have updated the [HISTORY.rst](https://github.com/scikit-learn-contrib/MAPIE/blob/master/HISTORY.rst) and [AUTHORS.rst](https://github.com/scikit-learn-contrib/MAPIE/blob/master/AUTHORS.rst) files
 - [ ] Linting passes successfully : `make lint`
 - [ ] Typing passes successfully : `make type-check`
 - [ ] Unit tests pass successfully : `make tests`

CONTRIBUTING.rst

@@ -6,15 +6,15 @@ What to work on?
 ----------------
 
 You are welcome to propose and contribute new ideas.
-We encourage you to `open an issue <https://github.com/simai-ml/MAPIE/issues>`_ so that we can align on the work to be done.
+We encourage you to `open an issue <https://github.com/scikit-learn-contrib/MAPIE/issues>`so that we can align on the work to be done.
 It is generally a good idea to have a quick discussion before opening a pull request that is potentially out-of-scope.
 
 Fork/clone/pull
 ---------------
 
 The typical workflow for contributing to `mapie` is:
 
-1. Fork the `master` branch from the `GitHub repository <https://github.com/simai-ml/MAPIE>`_.
+1. Fork the `master` branch from the `GitHub repository <https://github.com/scikit-learn-contrib/MAPIE>`_.
 2. Clone your fork locally.
 3. Commit changes.
 4. Push the changes to your fork.
@@ -64,8 +64,8 @@ Updating changelog
 
 You can make your contribution visible by :
 
-1. adding your name to the Contributors sections of `AUTHORS.rst <https://github.com/simai-ml/MAPIE/blob/master/AUTHORS.rst>`_
-2. adding a line describing your change into `HISTORY.rst <https://github.com/simai-ml/MAPIE/blob/master/HISTORY.rst>`_
+1. adding your name to the Contributors sections of `AUTHORS.rst <https://github.com/scikit-learn-contrib/MAPIE/blob/master/AUTHORS.rst>`_
+2. adding a line describing your change into `HISTORY.rst <https://github.com/scikit-learn-contrib/MAPIE/blob/master/HISTORY.rst>`_
 
 Testing
 -------

HISTORY.rst

@@ -5,6 +5,19 @@ History
 0.8.x (2024-xx-xx)
 ------------------
 
+* Add Mondrian Conformal Prediction for regression and classification
+* Add `** predict_params` in fit and predict method for Mapie Regression
+* Update the ts-changepoint notebook with the tutorial
+* Change import related to conformity scores into ts-changepoint notebook
+* Replace `assert np.array_equal` by `np.testing.assert_array_equal` in Mapie unit tests
+* Replace `github.com/simai-ml/MAPIE` by `github.com/scikit-learn-contrib/MAPIE`in all Mapie files
+* Extend `ConformityScore` to support regression (with `BaseRegressionScore`) and to support classification (with `BaseClassificationScore`)
+* Extend `EnsembleEstimator` to support regression (with `EnsembleRegressor`) and to support classification (with `EnsembleClassifier`)
+* Refactor `MapieClassifier` by separating the handling of the `MapieClassifier` estimator into a new class called `EnsembleClassifier`
+* Refactor `MapieClassifier` by separating the handling of the `MapieClassifier` conformity score into a new class called `BaseClassificationScore`
+* Add severals non-conformity scores for classification (`LAC`, `APS`, `RAPS`, `TopK`) based on `BaseClassificationScore`
+* Transfer the logic of classification methods into the non-conformity score classes (`LAC`, `APS`, `RAPS`, `TopK`)
+* Extend the classification strategy definition by supporting `method` and `conformity_score` attributes
 * Building unit tests for different `Subsample` and `BlockBooststrap` instances
 * Change the sign of C_k in the `Kolmogorov-Smirnov` test documentation
 * Building a training set with a fraction between 0 and 1 with `n_samples` attribute when using `split` method from `Subsample` class.

doc/Makefile

@@ -52,6 +52,7 @@ clean:
 	-rm -rf examples_classification/
 	-rm -rf examples_multilabel_classification/
 	-rm -rf examples_calibration/
+	-rm -rf examples_mondrian/
 	-rm -rf generated/*
 	-rm -rf modules/generated/*
 

doc/api.rst

@@ -73,17 +73,32 @@ Metrics
    metrics.spiegelhalter_statistic
    metrics.top_label_ece
 
-Conformity scores
-=================
+Conformity scores (regression)
+==============================
 
 .. autosummary::
    :toctree: generated/
    :template: class.rst
 
+   conformity_scores.BaseRegressionScore
    conformity_scores.AbsoluteConformityScore
    conformity_scores.GammaConformityScore
    conformity_scores.ResidualNormalisedScore
 
+Conformity scores (classification)
+==================================
+
+.. autosummary::
+   :toctree: generated/
+   :template: class.rst
+
+   conformity_scores.BaseClassificationScore
+   conformity_scores.NaiveConformityScore
+   conformity_scores.LACConformityScore
+   conformity_scores.APSConformityScore
+   conformity_scores.RAPSConformityScore
+   conformity_scores.TopKConformityScore
+
 Resampling
 ==========
 
@@ -93,3 +108,13 @@ Resampling
 
    subsample.BlockBootstrap
    subsample.Subsample
+
+
+Mondrian
+==========
+
+.. autosummary::
+   :toctree: generated/
+   :template: class.rst
+
+   mondrian.MondrianCP

doc/conf.py

@@ -316,13 +316,15 @@
         "../examples/regression",
         "../examples/classification",
         "../examples/multilabel_classification",
-        "../examples/calibration"
+        "../examples/calibration",
+        "../examples/mondrian",
     ],
     "gallery_dirs": [
         "examples_regression",
         "examples_classification",
         "examples_multilabel_classification",
-        "examples_calibration"
+        "examples_calibration",
+        "examples_mondrian",
     ],
     "doc_module": "mapie",
     "backreferences_dir": os.path.join("generated"),

doc/images/mondrian.png

@@ -49,6 +49,14 @@
    examples_multilabel_classification/1-quickstart/plot_tutorial_multilabel_classification
    notebooks_multilabel_classification
 
+.. toctree::
+   :maxdepth: 2
+   :hidden:
+   :caption: MONDRIAN
+
+   theoretical_description_mondrian
+   examples_mondrian/1-quickstart/plot_main-tutorial-mondrian-regression
+
 .. toctree::
    :maxdepth: 2
    :hidden:

doc/index.rst

@@ -0,0 +1,46 @@
+.. title:: Theoretical Description Mondrian : contents
+
+.. _theoretical_description_mondrian:
+
+#######################
+Theoretical Description
+#######################
+
+Mondrian conformal prediction (MCP) [1] is a method that allows to build prediction sets with a group-conditional
+coverage guarantee.  The coverage guarantee is given by:
+
+.. math::
+    P \{Y_{n+1} \in \hat{C}_{n, \alpha}(X_{n+1}) | G_{n+1} = g\} \geq 1 - \alpha
+
+where :math:`G_{n+1}` is the group of the new test point :math:`X_{n+1}` and :math:`g`
+is a group in the set of groups :math:`\mathcal{G}`.
+
+MCP can be used with any split conformal predictor and can be particularly useful when one have a prior 
+knowledge about existing groups wheter the information is directly included in the features
+of the data or not.
+In a classifcation setting, the groups can be defined as the predicted classes of the data. Doing so,
+one can ensure that, for each predicted class, the coverage guarantee is satisfied.
+
+In order to achieve the group-conditional coverage guarantee, MCP simply classifies the data
+according to the groups and then applies the split conformal predictor to each group separately.
+
+The quantile of each group is defined as:
+
+.. math::
+   \widehat{q}^g =Quantile\left(s_1, ..., s_{n^g} ,\frac{\lceil (n^{(g)} + 1)(1-\alpha)\rceil}{n^{(g)}} \right)
+
+Where :math:`s_1, ..., s_{n^g}` are the conformity scores of the training points in group :math:`g` and :math:`n^{(g)}`
+is the number of training points in group :math:`g`.
+
+The following figure (from [1]) explains the process of Mondrian conformal prediction:
+
+.. image:: images/mondrian.png
+   :width: 600
+   :align: center
+
+References
+----------
+
+[1] Vladimir Vovk, David Lindsay, Ilia Nouretdinov, and Alex Gammerman.
+Mondrian confidence machine.
+Technical report, Royal Holloway University of London, 2003