mapie.classification.MapieClassifier

class mapie.classification.MapieClassifier(estimator: Optional[sklearn.base.ClassifierMixin] = None, method: str = 'lac', cv: Optional[Union[int, str, sklearn.model_selection._split.BaseCrossValidator]] = None, test_size: Optional[Union[int, float]] = None, n_jobs: Optional[int] = None, random_state: Optional[Union[int, numpy.random.mtrand.RandomState]] = None, verbose: int = 0)[source]

Prediction sets for classification.

This class implements several conformal prediction strategies for estimating prediction sets for classification. Instead of giving a single predicted label, the idea is to give a set of predicted labels (or prediction sets) which come with mathematically guaranteed coverages.

Parameters
estimator: Optional[ClassifierMixin]

Any classifier with scikit-learn API (i.e. with fit, predict, and predict_proba methods), by default None. If None, estimator defaults to a LogisticRegression instance.

method: Optional[str]

Method to choose for prediction interval estimates. Choose among:

  • "naive", sum of the probabilities until the 1-alpha thresold.

  • "lac" (formerly called "score"), Least Ambiguous set-valued Classifier. It is based on the the scores (i.e. 1 minus the softmax score of the true label) on the calibration set. See [1] for more details.

  • "aps" (formerly called “cumulated_score”), Adaptive Prediction Sets method. It is based on the sum of the softmax outputs of the labels until the true label is reached, on the calibration set. See [2] for more details.

  • "raps", Regularized Adaptive Prediction Sets method. It uses the same technique as "aps" method but with a penalty term to reduce the size of prediction sets. See [3] for more details. For now, this method only works with "prefit" and "split" strategies.

  • "top_k", based on the sorted index of the probability of the true label in the softmax outputs, on the calibration set. In case two probabilities are equal, both are taken, thus, the size of some prediction sets may be different from the others. See [3] for more details.

By default "lac".

cv: Optional[str]

The cross-validation strategy for computing scores. It directly drives the distinction between jackknife and cv variants. Choose among:

  • None, to use the default 5-fold cross-validation

  • integer, to specify the number of folds. If equal to -1, equivalent to sklearn.model_selection.LeaveOneOut().

  • CV splitter: any sklearn.model_selection.BaseCrossValidator Main variants are: - sklearn.model_selection.LeaveOneOut (jackknife), - sklearn.model_selection.KFold (cross-validation)

  • "split", does not involve cross-validation but a division of the data into training and calibration subsets. The splitter used is the following: sklearn.model_selection.ShuffleSplit.

  • "prefit", assumes that estimator has been fitted already. All data provided in the fit method is then used to calibrate the predictions through the score computation. At prediction time, quantiles of these scores are used to estimate prediction sets.

By default None.

test_size: Optional[Union[int, float]]

If float, should be between 0.0 and 1.0 and represent the proportion of the dataset to include in the test split. If int, represents the absolute number of test samples. If None, it will be set to 0.1.

If cv is not "split", test_size is ignored.

By default None.

n_jobs: Optional[int]

Number of jobs for parallel processing using joblib via the “locky” backend. At this moment, parallel processing is disabled. If -1 all CPUs are used. If 1 is given, no parallel computing code is used at all, which is useful for debugging. For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. None is a marker for unset that will be interpreted as n_jobs=1 (sequential execution).

By default None.

random_state: Optional[Union[int, RandomState]]

Pseudo random number generator state used for random uniform sampling for evaluation quantiles and prediction sets. Pass an int for reproducible output across multiple function calls.

By default None.

verbose: int, optional

The verbosity level, used with joblib for multiprocessing. At this moment, parallel processing is disabled. The frequency of the messages increases with the verbosity level. If it more than 10, all iterations are reported. Above 50, the output is sent to stdout.

By default 0.

References

[1] Mauricio Sadinle, Jing Lei, and Larry Wasserman. “Least Ambiguous Set-Valued Classifiers with Bounded Error Levels.”, Journal of the American Statistical Association, 114, 2019.

[2] Yaniv Romano, Matteo Sesia and Emmanuel J. Candès. “Classification with Valid and Adaptive Coverage.” NeurIPS 202 (spotlight) 2020.

[3] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan and Jitendra Malik. “Uncertainty Sets for Image Classifiers using Conformal Prediction.” International Conference on Learning Representations 2021.

Examples

>>> import numpy as np
>>> from sklearn.naive_bayes import GaussianNB
>>> from mapie.classification import MapieClassifier
>>> X_toy = np.arange(9).reshape(-1, 1)
>>> y_toy = np.stack([0, 0, 1, 0, 1, 2, 1, 2, 2])
>>> clf = GaussianNB().fit(X_toy, y_toy)
>>> mapie = MapieClassifier(estimator=clf, cv="prefit").fit(X_toy, y_toy)
>>> _, y_pi_mapie = mapie.predict(X_toy, alpha=0.2)
>>> print(y_pi_mapie[:, :, 0])
[[ True False False]
 [ True False False]
 [ True  True False]
 [ True  True False]
 [False  True False]
 [False  True  True]
 [False  True  True]
 [False False  True]
 [False False  True]]
Attributes
valid_methods: List[str]

List of all valid methods.

single_estimator_: sklearn.ClassifierMixin

Estimator fitted on the whole training set.

n_features_in_: int

Number of features passed to the fit method.

conformity_scores_: ArrayLike of shape (n_samples_train)

The conformity scores used to calibrate the prediction sets.

quantiles_: ArrayLike of shape (n_alpha)

The quantiles estimated from conformity_scores_ and alpha values.

__init__(estimator: Optional[sklearn.base.ClassifierMixin] = None, method: str = 'lac', cv: Optional[Union[int, str, sklearn.model_selection._split.BaseCrossValidator]] = None, test_size: Optional[Union[int, float]] = None, n_jobs: Optional[int] = None, random_state: Optional[Union[int, numpy.random.mtrand.RandomState]] = None, verbose: int = 0) → None[source]
fit(X: Union[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]], y: Union[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]], sample_weight: Optional[Union[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]]] = None, size_raps: Optional[float] = 0.2, groups: Optional[Union[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]]] = None, **fit_params) → mapie.classification.MapieClassifier[source]

Fit the base estimator or use the fitted base estimator.

Parameters
X: ArrayLike of shape (n_samples, n_features)

Training data.

y: NDArray of shape (n_samples,)

Training labels.

sample_weight: Optional[ArrayLike] of shape (n_samples,)

Sample weights for fitting the out-of-fold models. If None, then samples are equally weighted. If some weights are null, their corresponding observations are removed before the fitting process and hence have no prediction sets.

By default None.

size_raps: Optional[float]

Percentage of the data to be used for choosing lambda_star and k_star for the RAPS method.

By default .2.

groups: Optional[ArrayLike] of shape (n_samples,)

Group labels for the samples used while splitting the dataset into train/test set.

By default None.

**fit_paramsdict

Additional fit parameters.

Returns
MapieClassifier

The model itself.

predict(X: Union[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]], numpy._typing._nested_sequence._NestedSequence[numpy._typing._array_like._SupportsArray[numpy.dtype[Any]]], bool, int, float, complex, str, bytes, numpy._typing._nested_sequence._NestedSequence[Union[bool, int, float, complex, str, bytes]]], alpha: Optional[Union[float, Iterable[float]]] = None, include_last_label: Optional[Union[bool, str]] = True, agg_scores: Optional[str] = 'mean') → Union[numpy.ndarray[Any, numpy.dtype[numpy._typing._array_like._ScalarType_co]], Tuple[numpy.ndarray[Any, numpy.dtype[numpy._typing._array_like._ScalarType_co]], numpy.ndarray[Any, numpy.dtype[numpy._typing._array_like._ScalarType_co]]]][source]

Prediction prediction sets on new samples based on target confidence interval. Prediction sets for a given alpha are deduced from:

  • quantiles of softmax scores ("lac" method)

  • quantiles of cumulated scores ("aps" method)

Parameters
X: ArrayLike of shape (n_samples, n_features)

Test data.

alpha: Optional[Union[float, Iterable[float]]]

Can be a float, a list of floats, or a ArrayLike of floats. Between 0 and 1, represent the uncertainty of the confidence interval. Lower alpha produce larger (more conservative) prediction sets. alpha is the complement of the target coverage level.

By default None.

include_last_label: Optional[Union[bool, str]]

Whether or not to include last label in prediction sets for the “aps” method. Choose among:

  • False, does not include label whose cumulated score is just over the quantile.

  • True, includes label whose cumulated score is just over the quantile, unless there is only one label in the prediction set.

  • “randomized”, randomly includes label whose cumulated score is just over the quantile based on the comparison of a uniform number and the difference between the cumulated score of the last label and the quantile.

When set to True or False, it may result in a coverage higher than 1 - alpha (because contrary to the “randomized” setting, none of this methods create empty prediction sets). See [2] and [3] for more details.

By default True.

agg_scores: Optional[str]

How to aggregate the scores output by the estimators on test data if a cross-validation strategy is used. Choose among:

  • “mean”, take the mean of scores.

  • “crossval”, compare the scores between all training data and each test point for each label to estimate if the label must be included in the prediction set. Follows algorithm 2 of Romano+2020.

By default “mean”.

Returns
Union[NDArray, Tuple[NDArray, NDArray]]
  • NDArray of shape (n_samples,) if alpha is None.
  • Tuple[NDArray, NDArray] of shapes
(n_samples,) and (n_samples, n_classes, n_alpha) if alpha is not None.
set_fit_request(*, groups: Union[bool, None, str] = '$UNCHANGED$', sample_weight: Union[bool, None, str] = '$UNCHANGED$', size_raps: Union[bool, None, str] = '$UNCHANGED$') → mapie.classification.MapieClassifier

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

New in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a pipeline.Pipeline. Otherwise it has no effect.

Parameters
groupsstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for groups parameter in fit.

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

size_rapsstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for size_raps parameter in fit.

Returns
selfobject

The updated object.

set_predict_request(*, agg_scores: Union[bool, None, str] = '$UNCHANGED$', alpha: Union[bool, None, str] = '$UNCHANGED$', include_last_label: Union[bool, None, str] = '$UNCHANGED$') → mapie.classification.MapieClassifier

Request metadata passed to the predict method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to predict.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

New in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a pipeline.Pipeline. Otherwise it has no effect.

Parameters
agg_scoresstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for agg_scores parameter in predict.

alphastr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for alpha parameter in predict.

include_last_labelstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for include_last_label parameter in predict.

Returns
selfobject

The updated object.

set_score_request(*, sample_weight: Union[bool, None, str] = '$UNCHANGED$') → mapie.classification.MapieClassifier

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

New in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a pipeline.Pipeline. Otherwise it has no effect.

Parameters
sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns
selfobject

The updated object.