mapie.metrics.regression_ssc_score

mapie.metrics.regression_ssc_score(y_true: ndarray[Any, dtype[_ScalarType_co]], y_intervals: ndarray[Any, dtype[_ScalarType_co]], num_bins: int = 3) → ndarray[Any, dtype[_ScalarType_co]]

Aggregate by the minimum for each alpha the Size-Stratified Coverage [3]: returns the maximum violation of the conditional coverage (with the groups defined).

Warning: This metric should be used only with non constant intervals (intervals of different sizes), with constant intervals the result may be misinterpreted.

[3] Angelopoulos, A. N., & Bates, S. (2021). A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511.

Parameters

y_true: NDArray of shape (n_samples,)

True labels.

y_intervals: NDArray of shape (n_samples, 2, n_alpha) or (n_samples, 2)

Prediction intervals given by booleans of labels.

num_bins: int n

Number of groups. Should be less than the number of different interval widths.

Returns

NDArray of shape (n_alpha,)

Examples

>>> from mapie.metrics import regression_ssc
>>> import numpy as np
>>> y_true = np.array([5, 7.5, 9.5])
>>> y_intervals = np.array([
... [[4, 4], [6, 7.5]],
... [[6.0, 8], [9.0, 10]],
... [[9, 9], [10.0, 10.0]]
... ])
>>> print(regression_ssc_score(y_true, y_intervals, num_bins=2))
[1.  0.5]

Examples using mapie.metrics.regression_ssc_score

Estimating conditional coverage

Tutorial for residual normalised score