mapie.metrics.kolmogorov_smirnov_statistic(y_true: ndarray[Any, dtype[_ScalarType_co]], y_score: ndarray[Any, dtype[_ScalarType_co]]) float[source]

Compute Kolmogorov-smirnov’s statistic for calibration test. Also called ECCE-MAD (Estimated Cumulative Calibration Errors - Maximum Absolute Deviation). The closer to zero, the better the scores are calibrated. Indeed, if the scores are perfectly calibrated, the cumulative differences between y_true and y_score should share the same properties of a standard Brownian motion asymptotically.

y_trueNDArray of shape (n_samples,)

An array of ground truth.

y_scoreNDArray of shape (n_samples,)

An array of scores..


Kolmogorov-smirnov’s statistic.


Arrieta-Ibarra I, Gujral P, Tannen J, Tygert M, Xu C. Metrics of calibration for probabilistic predictions. The Journal of Machine Learning Research. 2022 Jan 1;23(1):15886-940.