mapie.metrics.kuiper_p_value

mapie.metrics.kuiper_p_value(y_true: ndarray[Any, dtype[_ScalarType_co]], y_score: ndarray[Any, dtype[_ScalarType_co]]) → float

Compute Kuiper statistic p-value. Deduced from the corresponding statistic and CDF. It represents the probability of the observed statistic under the null hypothesis of perfect calibration.

Parameters

y_trueNDArray of shape (n_samples,)

An array of ground truth.

y_scoreNDArray of shape (n_samples,)

An array of scores.

Returns

float

The Kuiper p-value.

References

Tygert M. Calibration of P-values for calibration and for deviation of a subpopulation from the full population. arXiv preprint arXiv:2202.00100. 2022 Jan 31.

William Feller. The Asymptotic Distribution of the Range of Sums of Independent Random Variables. Ann. Math. Statist. 22 (3) 427 - 432 September, 1951.

Examples

>>> import pandas as pd
>>> from mapie.metrics import kuiper_p_value
>>> y_true = np.array([1, 0, 1, 0, 1, 0])
>>> y_score = np.array([0.8, 0.3, 0.5, 0.5, 0.7, 0.1])
>>> ku_p_value = kuiper_p_value(y_true, y_score)
>>> print(np.round(ku_p_value, 4))
0.9684

Examples using mapie.metrics.kuiper_p_value

Evaluating the asymptotic convergence of p-values