mapie.conformity_scores.TopKConformityScore

class mapie.conformity_scores.TopKConformityScore

Top-K method-based non-conformity score.

It is based on the sorted index of the probability of the true label in the softmax outputs, on the calibration set. In case two probabilities are equal, both are taken, thus, the size of some prediction sets may be different from the others.

References

[1] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan and Jitendra Malik. “Uncertainty Sets for Image Classifiers using Conformal Prediction.” International Conference on Learning Representations 2021.

Attributes

classes: Optional[ArrayLike]

Names of the classes.

random_state: Optional[Union[int, RandomState]]

Pseudo random number generator state.

quantiles_: ArrayLike of shape (n_alpha)

The quantiles estimated from get_sets method.

__init__() → None

get_conformity_score_quantiles(conformity_scores: ndarray[Any, dtype[_ScalarType_co]], alpha_np: ndarray[Any, dtype[_ScalarType_co]], estimator: EnsembleClassifier, **kwargs) → ndarray[Any, dtype[_ScalarType_co]]

Get the quantiles of the conformity scores for each uncertainty level.

get_conformity_scores(y: ndarray[Any, dtype[_ScalarType_co]], y_pred: ndarray[Any, dtype[_ScalarType_co]], y_enc: Optional[ndarray[Any, dtype[_ScalarType_co]]] = None, **kwargs) → ndarray[Any, dtype[_ScalarType_co]]

Get the conformity score.

Parameters

y: NDArray of shape (n_samples,)

Observed target values.

y_pred: NDArray of shape (n_samples,)

Predicted target values.

y_enc: NDArray of shape (n_samples,)

Target values as normalized encodings.

Returns

NDArray of shape (n_samples,)

Conformity scores.

get_prediction_sets(y_pred_proba: ndarray[Any, dtype[_ScalarType_co]], conformity_scores: ndarray[Any, dtype[_ScalarType_co]], alpha_np: ndarray[Any, dtype[_ScalarType_co]], estimator: EnsembleClassifier, **kwargs) → ndarray[Any, dtype[_ScalarType_co]]

Generate prediction sets based on the probability predictions, the conformity scores and the uncertainty level.

get_predictions(X: ndarray[Any, dtype[_ScalarType_co]], alpha_np: ndarray[Any, dtype[_ScalarType_co]], estimator: EnsembleClassifier, **kwargs) → ndarray[Any, dtype[_ScalarType_co]]

Get predictions from an EnsembleClassifier.

This method should be implemented by any subclass of the current class.