mapie.metrics.regression.regression_mwi_score

mapie.metrics.regression.regression_mwi_score(y_true: ndarray[Any, dtype[_ScalarType_co]], y_pis: ndarray[Any, dtype[_ScalarType_co]], confidence_level: float) → float

The Winkler score, proposed by Winkler (1972), is a measure used to evaluate prediction intervals, combining the length of the interval with a penalty that increases proportionally to the distance of an observation outside the interval.

Parameters

y_true: ArrayLike of shape (n_samples,)

Ground truth values

y_pis: ArrayLike of shape (n_samples, 2, 1)

Lower and upper bounds of prediction intervals output from a MAPIE regressor

confidence_level: float

The value of confidence_level

Returns

float

The mean Winkler interval score

References

[1] Robert L. Winkler “A Decision-Theoretic Approach to Interval Estimation”, Journal of the American Statistical Association, volume 67, pages 187-191 (1972) (https://doi.org/10.1080/01621459.1972.10481224) [2] Tilmann Gneiting and Adrian E Raftery “Strictly Proper Scoring Rules, Prediction, and Estimation”, Journal of the American Statistical Association, volume 102, pages 359-378 (2007) (https://doi.org/10.1198/016214506000001437) (Section 6.2)