Possible regression between 0.9.1 and 0.9.2 (time series)

[Valentin-Laurent]

The last image of the tutorial for time series is different between v0.9.1 and v0.9.2 and above. This may be due to a regression, a bugfix, or a different random state. Links:

• 0.9.1
• 0.9.2
• stable (1.0.1)

[hmaissoro]

When TimeSeriesRegression.method="aci", the update method does not appear to update the internal state in the same way as it does for "enbpi". Instead, it delegates to adapt_conformal_inference, which seems to not perform update.

• With method="enbpi", update properly refreshes conformity scores.
• With method="aci", update calls adapt_conformal_inference, which does not seem to provide an equivalent update.

As a result, some examples explicitly call both update and adapt_conformal_inference, or even temporarily switch method to "enbpi" to force a proper update before returning to "aci".

# mapie/regression/time_series_regression.py
def update() -> TimeSeriesRegressor:
        .....
        if self.method == "enbpi":
            return self._update_conformity_scores_with_ensemble(X, y, ensemble=ensemble)
        elif self.method == "aci":
            return self.adapt_conformal_inference(...)

# examples/regression/1-quickstart/plot_ts-tutorial.py
for step in range(gap, len(X_test), gap):
    mapie_aci.update(
        X_test.iloc[(step - gap) : step, :].to_numpy(),
        y_test.iloc[(step - gap) : step].to_numpy(),
    )
    mapie_aci.adapt_conformal_inference(
        X_test.iloc[(step - gap) : step, :].to_numpy(),
        y_test.iloc[(step - gap) : step].to_numpy(),
        gamma=0.05,
    )

# notebooks/regression/ts-changepoint.ipynb
# We note an implicite call of
#    the method : _update_conformity_scores_with_ensemble
#    through mapie_aci.method = "enbpi"
for step in range(gap, len(X_test), gap):
    mapie_aci.method = "enbpi"
    mapie_aci.update(
        X_test.iloc[(step - gap):step, :],
        y_test.iloc[(step - gap):step],
    )
    mapie_aci.method = "aci"
    mapie_aci.adapt_conformal_inference(
        X_test.iloc[(step - gap):step, :].to_numpy(),
        y_test.iloc[(step - gap):step].to_numpy(),
        gamma = 0.05
    )

The issue appears in multiple examples and notebooks when TimeSeriesRegression.method="aci" is used:

• Tutorial for time series
• Time series: example of the EnbPI technique
• Adaptive conformal predictions for time series, Zaffran et al. (2022)
• notebook : Estimating prediction intervals of time series forecast with EnbPI

This issue is also related to the need for clearer API definitions, as described in #379

It is also related to the issue #588 : Parameter optimize_beta is not working