mapie.regression.MapieRegressor

class mapie.regression.MapieRegressor(estimator: Optional[sklearn.base.RegressorMixin] = None, method: str = 'plus', cv: Optional[Union[int, str, sklearn.model_selection._split.BaseCrossValidator]] = None, n_jobs: Optional[int] = None, agg_function: Optional[str] = 'mean', verbose: int = 0, conformity_score: Optional[mapie.conformity_scores.ConformityScore] = None)[source]

Prediction interval with out-of-fold conformity scores.

This class implements the jackknife+ strategy and its variations for estimating prediction intervals on single-output data. The idea is to evaluate out-of-fold conformity scores (signed residuals, absolute residuals, residuals normalized by the predicted mean…) on hold-out validation sets and to deduce valid confidence intervals with strong theoretical guarantees.

Parameters
estimatorOptional[RegressorMixin]

Any regressor with scikit-learn API (i.e. with fit and predict methods), by default None. If None, estimator defaults to a LinearRegression instance.

method: str, optional

Method to choose for prediction interval estimates. Choose among:

  • “naive”, based on training set conformity scores,

  • “base”, based on validation sets conformity scores,

  • “plus”, based on validation conformity scores and testing predictions,

  • “minmax”, based on validation conformity scores and testing predictions (min/max among cross-validation clones).

By default “plus”.

cv: Optional[Union[int, str, BaseCrossValidator]]

The cross-validation strategy for computing conformity scores. It directly drives the distinction between jackknife and cv variants. Choose among:

  • None, to use the default 5-fold cross-validation

  • integer, to specify the number of folds. If equal to -1, equivalent to sklearn.model_selection.LeaveOneOut().

  • CV splitter: any sklearn.model_selection.BaseCrossValidator Main variants are: - sklearn.model_selection.LeaveOneOut (jackknife), - sklearn.model_selection.KFold (cross-validation), - subsample.Subsample object (bootstrap).

  • "prefit", assumes that estimator has been fitted already, and the method parameter is ignored. All data provided in the fit method is then used for computing conformity scores only. At prediction time, quantiles of these conformity scores are used to provide a prediction interval with fixed width. The user has to take care manually that data for model fitting and conformity scores estimate are disjoint.

By default None.

n_jobs: Optional[int]

Number of jobs for parallel processing using joblib via the “locky” backend. If -1 all CPUs are used. If 1 is given, no parallel computing code is used at all, which is useful for debugging. For n_jobs below -1, (n_cpus + 1 - n_jobs) are used. None is a marker for unset that will be interpreted as n_jobs=1 (sequential execution).

By default None.

agg_functionstr

Determines how to aggregate predictions from perturbed models, both at training and prediction time.

If None, it is ignored except if cv class is Subsample, in which case an error is raised. If “mean” or “median”, returns the mean or median of the predictions computed from the out-of-folds models. Note: if you plan to set the ensemble argument to True in the predict method, you have to specify an aggregation function. Otherwise an error would be raised.

The Jackknife+ interval can be interpreted as an interval around the median prediction, and is guaranteed to lie inside the interval, unlike the single estimator predictions.

When the cross-validation strategy is Subsample (i.e. for the Jackknife+-after-Bootstrap method), this function is also used to aggregate the training set in-sample predictions.

If cv is "prefit", agg_function is ignored.

By default “mean”.

verboseint, optional

The verbosity level, used with joblib for multiprocessing. The frequency of the messages increases with the verbosity level. If it more than 10, all iterations are reported. Above 50, the output is sent to stdout.

By default 0.

conformity_scoreOptional[ConformityScore]

ConformityScore instance. It defines the link between the observed values, the predicted ones and the conformity scores. For instance, the default None value correspondonds to a conformity score which assumes y_obs = y_pred + conformity_score.

  • None, to use the default AbsoluteConformityScore conformity score

  • ConformityScore: any ConformityScore class

By default None.

References

Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. “Predictive inference with the jackknife+.” Ann. Statist., 49(1):486–507, February 2021.

Byol Kim, Chen Xu, and Rina Foygel Barber. “Predictive Inference Is Free with the Jackknife+-after-Bootstrap.” 34th Conference on Neural Information Processing Systems (NeurIPS 2020).

Examples

>>> import numpy as np
>>> from mapie.regression import MapieRegressor
>>> from sklearn.linear_model import LinearRegression
>>> X_toy = np.array([[0], [1], [2], [3], [4], [5]])
>>> y_toy = np.array([5, 7.5, 9.5, 10.5, 12.5, 15])
>>> mapie_reg = MapieRegressor(LinearRegression()).fit(X_toy, y_toy)
>>> y_pred, y_pis = mapie_reg.predict(X_toy, alpha=0.5)
>>> print(y_pis[:, :, 0])
[[ 4.7972973   5.8       ]
 [ 6.69767442  7.65540541]
 [ 8.59883721  9.58108108]
 [10.5        11.40116279]
 [12.4        13.30232558]
 [14.25       15.20348837]]
>>> print(y_pred)
[ 5.28571429  7.17142857  9.05714286 10.94285714 12.82857143 14.71428571]
Attributes
valid_methods: List[str]

List of all valid methods.

single_estimator_sklearn.RegressorMixin

Estimator fitted on the whole training set.

estimators_list

List of out-of-folds estimators.

conformity_scores_ArrayLike of shape (n_samples_train,)

Conformity scores between y_train and y_pred.

k_ArrayLike
  • Array of nans, of shape (len(y), 1) if cv is "prefit" (defined but not used)

  • Dummy array of folds containing each training sample, otherwise. Of shape (n_samples_train, cv.get_n_splits(X_train, y_train)).

n_features_in_: int

Number of features passed to the fit method.

__init__(estimator: Optional[sklearn.base.RegressorMixin] = None, method: str = 'plus', cv: Optional[Union[int, str, sklearn.model_selection._split.BaseCrossValidator]] = None, n_jobs: Optional[int] = None, agg_function: Optional[str] = 'mean', verbose: int = 0, conformity_score: Optional[mapie.conformity_scores.ConformityScore] = None) None[source]
fit(X: Union[numpy._typing._array_like._SupportsArray[numpy.dtype], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]], y: Union[numpy._typing._array_like._SupportsArray[numpy.dtype], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]], sample_weight: Optional[Union[numpy._typing._array_like._SupportsArray[numpy.dtype], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]]] = None) mapie.regression.MapieRegressor[source]

Fit estimator and compute conformity scores used for prediction intervals. Fit the base estimator under the single_estimator_ attribute. Fit all cross-validated estimator clones and rearrange them into a list, the estimators_ attribute. Out-of-fold conformity scores are stored under the conformity_scores_ attribute.

Parameters
XArrayLike of shape (n_samples, n_features)

Training data.

yArrayLike of shape (n_samples,)

Training labels.

sample_weightOptional[ArrayLike] of shape (n_samples,)

Sample weights for fitting the out-of-fold models. If None, then samples are equally weighted. If some weights are null, their corresponding observations are removed before the fitting process and hence have no conformity scores. If weights are non-uniform, conformity scores are still uniformly weighted.

By default None.

Returns
MapieRegressor

The model itself.

predict(X: Union[numpy._typing._array_like._SupportsArray[numpy.dtype], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]], ensemble: bool = False, alpha: Optional[Union[float, Iterable[float]]] = None) Union[numpy.ndarray[Any, numpy.dtype[numpy._typing._generic_alias.ScalarType]], Tuple[numpy.ndarray[Any, numpy.dtype[numpy._typing._generic_alias.ScalarType]], numpy.ndarray[Any, numpy.dtype[numpy._typing._generic_alias.ScalarType]]]][source]

Predict target on new samples with confidence intervals. Conformity scores from the training set and predictions from the model clones are central to the computation. Prediction Intervals for a given alpha are deduced from either

  • quantiles of conformity scores (naive and base methods),

  • quantiles of (predictions +/- conformity scores) (plus method),

  • quantiles of (max/min(predictions) +/- conformity scores)

(minmax method).

Parameters
XArrayLike of shape (n_samples, n_features)

Test data.

ensemble: bool

Boolean determining whether the predictions are ensembled or not. If False, predictions are those of the model trained on the whole training set. If True, predictions from perturbed models are aggregated by the aggregation function specified in the agg_function attribute.

If cv is "prefit", ensemble is ignored.

By default False.

alpha: Optional[Union[float, Iterable[float]]]

Can be a float, a list of floats, or a ArrayLike of floats. Between 0 and 1, represents the uncertainty of the confidence interval. Lower alpha produce larger (more conservative) prediction intervals. alpha is the complement of the target coverage level.

By default None.

Returns
Union[NDArray, Tuple[NDArray, NDArray]]
  • NDArray of shape (n_samples,) if alpha is None.
  • Tuple[NDArray, NDArray] of shapes
(n_samples,) and (n_samples, 2, n_alpha) if alpha is not None.
  • [:, 0, :]: Lower bound of the prediction interval.

  • [:, 1, :]: Upper bound of the prediction interval.

Examples using mapie.regression.MapieRegressor