.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples_calibration/2-advanced-analysis/plot_asymptotic_convergence_of_p_values.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_calibration_2-advanced-analysis_plot_asymptotic_convergence_of_p_values.py: Evaluating the asymptotic convergence of p-values ================================================= This example uses `kolmogorov_smirnov_pvalue`, `kuiper_pvalue` and `spieglehalter_pvalue`. We investigate the asymptotic convergence of these functions toward real p-values. Indeed, these quantities are only asymptotic p-values, i.e. when the number of observations is infinite. However, they can be safely used as real p-values even with moderate dataset sizes. This is what we are going to illustrate in this exampple. To this end, we generate many datasets that are calibrated by nature, and plot the distribution of the p-values. A p-value must follow a uniform distribution when computed on statistical samples following the null hypothesis. The argument can be retrieved from the following references: [1] Arrieta-Ibarra I, Gujral P, Tannen J, Tygert M, Xu C. Metrics of calibration for probabilistic predictions. The Journal of Machine Learning Research. 2022 Jan 1;23(1):15886-940. [2] Tygert M. Calibration of P-values for calibration and for deviation of a subpopulation from the full population. arXiv preprint arXiv:2202.00100. 2022 Jan 31. [3] D. A. Darling. A. J. F. Siegert. The First Passage Problem for a Continuous Markov Process. Ann. Math. Statist. 24 (4) 624 - 639, December, 1953. .. GENERATED FROM PYTHON SOURCE LINES 37-46 .. code-block:: Python import numpy as np from matplotlib import pyplot as plt from sklearn.utils import check_random_state from numpy.typing import NDArray from mapie.metrics.calibration import spiegelhalter_p_value from mapie.metrics.calibration import kolmogorov_smirnov_p_value, kuiper_p_value .. GENERATED FROM PYTHON SOURCE LINES 47-50 First we need to generate scores that are perfecty calibrated. To do so, we simply start from a given array of probabilities between 0 and 1, and draw random labels 0 or 1 according to these probabilities. .. GENERATED FROM PYTHON SOURCE LINES 50-59 .. code-block:: Python def generate_y_true_calibrated(y_prob: NDArray, random_state: int = 1) -> NDArray: generator = check_random_state(random_state) uniform = generator.uniform(size=len(y_prob)) y_true = (uniform <= y_prob).astype(float) return y_true .. GENERATED FROM PYTHON SOURCE LINES 60-63 Then, we draw many different calibrated datasets, each with a fixed dataset size. For each of these datasets, we compute the available p-values implemented in MAPIE. .. GENERATED FROM PYTHON SOURCE LINES 63-83 .. code-block:: Python n_sets = 10000 n_points = 500 ks_p_values = [] ku_p_values = [] sp_p_values = [] for i in range(n_sets): y_score = np.linspace(0, 1, n_points) y_true = generate_y_true_calibrated(y_score, random_state=i) ks_p_value = kolmogorov_smirnov_p_value(y_true, y_score) ku_p_value = kuiper_p_value(y_true, y_score) sp_p_value = spiegelhalter_p_value(y_true, y_score) ks_p_values.append(ks_p_value) ku_p_values.append(ku_p_value) sp_p_values.append(sp_p_value) ks_p_values = np.sort(ks_p_values) ku_p_values = np.sort(ku_p_values) sp_p_values = np.sort(sp_p_values) .. GENERATED FROM PYTHON SOURCE LINES 84-90 Finally, we plot the empirical cumulative distribution function of the p-values computed on these many datasets. We see that even for moderately sized datasets, the p-values computed closely follow the expected uniform distribution under the null hypothesis. It appears that Kuiper p-value is the slowest to converge compared to Spiegelhalter and Kolmogorov-Smirnov. .. GENERATED FROM PYTHON SOURCE LINES 90-117 .. code-block:: Python plt.hist( ks_p_values, 100, cumulative=True, density=True, histtype="step", label="Kolmogorov-Smirnov", ) plt.hist( ku_p_values, 100, cumulative=True, density=True, histtype="step", label="Kuiper" ) plt.hist( sp_p_values, 100, cumulative=True, density=True, histtype="step", label="Spiegelhalter", ) plt.plot([0, 1], [0, 1], "--", color="black") plt.title("Distribution of p-values for calibrated datasets") plt.xlabel("p-values") plt.ylabel("Cumulative count (%)") plt.grid() plt.legend() plt.show() .. image-sg:: /examples_calibration/2-advanced-analysis/images/sphx_glr_plot_asymptotic_convergence_of_p_values_001.png :alt: Distribution of p-values for calibrated datasets :srcset: /examples_calibration/2-advanced-analysis/images/sphx_glr_plot_asymptotic_convergence_of_p_values_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 8.001 seconds) .. _sphx_glr_download_examples_calibration_2-advanced-analysis_plot_asymptotic_convergence_of_p_values.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_asymptotic_convergence_of_p_values.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_asymptotic_convergence_of_p_values.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_asymptotic_convergence_of_p_values.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_