mapie.conformity_scores
.APSConformityScore¶
- class mapie.conformity_scores.APSConformityScore[source]¶
Adaptive Prediction Sets (APS) method-based non-conformity score. It is based on the sum of the softmax outputs of the labels until the true label is reached, on the calibration set. See [1] for more details.
References
[1] Yaniv Romano, Matteo Sesia and Emmanuel J. Candès. “Classification with Valid and Adaptive Coverage.” NeurIPS 202 (spotlight) 2020.
- Attributes
- classes: Optional[ArrayLike]
Names of the classes.
- random_state: Optional[Union[int, RandomState]]
Pseudo random number generator state.
- quantiles_: ArrayLike of shape (n_alpha)
The quantiles estimated from
get_sets
method.
- get_conformity_score_quantiles(conformity_scores: ndarray[Any, dtype[_ScalarType_co]], alpha_np: ndarray[Any, dtype[_ScalarType_co]], estimator: EnsembleClassifier, agg_scores: Optional[str] = 'mean', **kwargs) ndarray[Any, dtype[_ScalarType_co]] [source]¶
Get the quantiles of the conformity scores for each uncertainty level.
- get_conformity_scores(y: ndarray[Any, dtype[_ScalarType_co]], y_pred: ndarray[Any, dtype[_ScalarType_co]], y_enc: Optional[ndarray[Any, dtype[_ScalarType_co]]] = None, **kwargs) ndarray[Any, dtype[_ScalarType_co]] [source]¶
Get the conformity score.
- Parameters
- y: NDArray of shape (n_samples,)
Observed target values.
- y_pred: NDArray of shape (n_samples,)
Predicted target values.
- y_enc: NDArray of shape (n_samples,)
Target values as normalized encodings.
- Returns
- NDArray of shape (n_samples,)
Conformity scores.
- get_prediction_sets(y_pred_proba: ndarray[Any, dtype[_ScalarType_co]], conformity_scores: ndarray[Any, dtype[_ScalarType_co]], alpha_np: ndarray[Any, dtype[_ScalarType_co]], estimator: EnsembleClassifier, agg_scores: Optional[str] = 'mean', include_last_label: Optional[Union[bool, str]] = True, **kwargs) ndarray[Any, dtype[_ScalarType_co]] [source]¶
Generate prediction sets based on the probability predictions, the conformity scores and the uncertainty level.
- get_predictions(X: ndarray[Any, dtype[_ScalarType_co]], alpha_np: ndarray[Any, dtype[_ScalarType_co]], estimator: EnsembleClassifier, agg_scores: Optional[str] = 'mean', **kwargs) ndarray[Any, dtype[_ScalarType_co]] [source]¶
Get predictions from an EnsembleClassifier.
- static get_true_label_cumsum_proba(y: Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]], y_pred_proba: ndarray[Any, dtype[_ScalarType_co]], classes: Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]) Tuple[ndarray[Any, dtype[_ScalarType_co]], ndarray[Any, dtype[_ScalarType_co]]] [source]¶
Compute the cumsumed probability of the true label.
- Parameters
- y: NDArray of shape (n_samples, )
Array with the labels.
- y_pred_proba: NDArray of shape (n_samples, n_classes)
Predictions of the model.
- classes: NDArray of shape (n_classes, )
Array with the classes.
- Returns
- Tuple[NDArray, NDArray] of shapes (n_samples, 1) and (n_samples, ).
The first element is the cumsum probability of the true label. The second is the sorted position of the true label.