mapie.metrics.kolmogorov_smirnov_statistic

mapie.metrics.kolmogorov_smirnov_statistic(y_true: numpy.ndarray[Any, numpy.dtype[numpy._typing._array_like._ScalarType_co]], y_score: numpy.ndarray[Any, numpy.dtype[numpy._typing._array_like._ScalarType_co]]) → float[source]

Compute Kolmogorov-smirnov’s statistic for calibration test. Also called ECCE-MAD (Estimated Cumulative Calibration Errors - Maximum Absolute Deviation). The closer to zero, the better the scores are calibrated. Indeed, if the scores are perfectly calibrated, the cumulative differences between y_true and y_score should share the same properties of a standard Brownian motion asymptotically.

Parameters
y_trueNDArray of shape (n_samples,)

An array of ground truth.

y_scoreNDArray of shape (n_samples,)

An array of scores..

Returns
float

Kolmogorov-smirnov’s statistic.

References

Arrieta-Ibarra I, Gujral P, Tannen J, Tygert M, Xu C. Metrics of calibration for probabilistic predictions. The Journal of Machine Learning Research. 2022 Jan 1;23(1):15886-940.