.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples_exchangeability_testing/2-advanced-analysis/plot_permutation_test.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_permutation_test.py: Permutation test for exchangeability ==================================== This example illustrates how to run `PValuePermutationTest` and `SequentialMonteCarloTest` on a toy regression problem. We compare a dataset with exchangeable residuals to a dataset where the second half of the residuals is shifted, which breaks the exchangeability assumption used by conformal methods. .. GENERATED FROM PYTHON SOURCE LINES 11-20 .. code-block:: Python import matplotlib.pyplot as plt import numpy as np from mapie.exchangeability_testing.permutations import ( PValuePermutationTest, SequentialMonteCarloTest, ) .. GENERATED FROM PYTHON SOURCE LINES 21-39 Build two fixed datasets ------------------------ We build a toy regression signal and compare two datasets: - one with residuals centered around zero, which should be close to exchangeable - one where the second half of the targets is shifted upward, breaking exchangeability No estimator is provided explicitly in this example. The permutation tests internally build and fit a default `SplitConformalRegressor` on a split of the data passed to `run`. This is necessary to be able to compute non-conformity scores which measure some sort of model prediction error. Permutation tests then test the exchangeability of the non-conformity scores. This approach allows more flexibility as complex dataset are reduced to a list of values. As non-conformity scores are exchangeability preserving transformations, the exchangeability test result is valid for the original dataset. .. GENERATED FROM PYTHON SOURCE LINES 39-50 .. code-block:: Python # If an estimator fitted on some training data is provided, the permutation tests # should be run on the test data. Results would be better as no split would be needed # to fit a model internally. rng = np.random.RandomState(0) X = np.linspace(0.1, 0.9, 500).reshape(-1, 1) y_exchangeable = 3 * X.ravel() + rng.normal(scale=0.1, size=X.shape[0]) y_shifted = y_exchangeable.copy() y_shifted[len(y_shifted) // 2 :] += 0.8 .. GENERATED FROM PYTHON SOURCE LINES 51-55 Run the exchangeable case ------------------------- We start with a dataset that should satisfy exchangeability. .. GENERATED FROM PYTHON SOURCE LINES 55-151 .. code-block:: Python test_level = 0.1 num_permutations = 200 exchangeable_pvalue_test = PValuePermutationTest( test_level=test_level, random_state=7, num_permutations=num_permutations, ) exchangeable_aggressive_test = SequentialMonteCarloTest( strategy="aggressive", test_level=test_level, random_state=7, num_permutations=num_permutations, ) exchangeable_binomial_test = SequentialMonteCarloTest( strategy="binomial", test_level=test_level, random_state=7, num_permutations=num_permutations, ) exchangeable_binomial_mixture_test = SequentialMonteCarloTest( strategy="binomial_mixture", test_level=test_level, random_state=7, num_permutations=num_permutations, ) exchangeable_pvalue_test.run(X, y_exchangeable) exchangeable_pvalue_detected = exchangeable_pvalue_test.is_exchangeable exchangeable_aggressive_test.run(X, y_exchangeable) exchangeable_aggressive_detected = exchangeable_aggressive_test.is_exchangeable exchangeable_binomial_test.run(X, y_exchangeable) exchangeable_binomial_detected = exchangeable_binomial_test.is_exchangeable exchangeable_binomial_mixture_test.run(X, y_exchangeable) exchangeable_binomial_mixture_detected = ( exchangeable_binomial_mixture_test.is_exchangeable ) print("\nExchangeable dataset") print("--------------------") print( f"PValuePermutationTest: data exchangeability={exchangeable_pvalue_detected}. Detection after {len(exchangeable_pvalue_test.p_values) - 1} permutations." ) print( f"SequentialMonteCarloTest (aggressive): data exchangeability={exchangeable_aggressive_detected}. Detection after {len(exchangeable_aggressive_test.p_values) - 1} permutations." ) print( f"SequentialMonteCarloTest (binomial): data exchangeability={exchangeable_binomial_detected}. Detection after {len(exchangeable_binomial_test.p_values) - 1} permutations." ) print( "SequentialMonteCarloTest (binomial_mixture): " f"data exchangeability={exchangeable_binomial_mixture_detected}. Detection after {len(exchangeable_binomial_mixture_test.p_values) - 1} permutations." ) test_level = exchangeable_pvalue_test.test_level plt.figure(figsize=(8, 4)) plt.plot(exchangeable_pvalue_test.p_values, label="PValuePermutationTest", zorder=4) plt.plot( exchangeable_aggressive_test.p_values, label="SMC aggressive", linestyle="-", linewidth=3.0, zorder=1, ) plt.plot( exchangeable_binomial_test.p_values, label="SMC binomial", linestyle="--", linewidth=2.0, zorder=2, ) plt.plot( exchangeable_binomial_mixture_test.p_values, label="SMC binomial_mixture", linestyle=":", linewidth=2.0, zorder=3, ) plt.axhline( test_level, color="black", linestyle="--", label=f"test_level = {test_level:.2f}", ) plt.xlabel("Number of permutations") plt.ylabel("Running p-value") plt.title("Exchangeable residuals") plt.grid(alpha=0.3) plt.legend() plt.tight_layout() plt.show() .. image-sg:: /examples_exchangeability_testing/2-advanced-analysis/images/sphx_glr_plot_permutation_test_001.png :alt: Exchangeable residuals :srcset: /examples_exchangeability_testing/2-advanced-analysis/images/sphx_glr_plot_permutation_test_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Exchangeable dataset -------------------- PValuePermutationTest: data exchangeability=True. Detection after 200 permutations. SequentialMonteCarloTest (aggressive): data exchangeability=True. Detection after 101 permutations. SequentialMonteCarloTest (binomial): data exchangeability=True. Detection after 101 permutations. SequentialMonteCarloTest (binomial_mixture): data exchangeability=True. Detection after 101 permutations. .. GENERATED FROM PYTHON SOURCE LINES 152-156 Run the non-exchangeable case ----------------------------- We now repeat the same comparison on the shifted dataset. .. GENERATED FROM PYTHON SOURCE LINES 156-242 .. code-block:: Python shifted_pvalue_test = PValuePermutationTest( test_level=test_level, random_state=7, num_permutations=num_permutations, ) shifted_aggressive_test = SequentialMonteCarloTest( strategy="aggressive", test_level=test_level, random_state=7, num_permutations=num_permutations, ) shifted_binomial_test = SequentialMonteCarloTest( strategy="binomial", test_level=test_level, random_state=7, num_permutations=num_permutations, ) shifted_binomial_mixture_test = SequentialMonteCarloTest( strategy="binomial_mixture", test_level=test_level, random_state=7, num_permutations=num_permutations, ) shifted_pvalue_test.run(X, y_shifted) shifted_pvalue_detected = shifted_pvalue_test.is_exchangeable shifted_aggressive_test.run(X, y_shifted) shifted_aggressive_detected = shifted_aggressive_test.is_exchangeable shifted_binomial_test.run(X, y_shifted) shifted_binomial_detected = shifted_binomial_test.is_exchangeable shifted_binomial_mixture_test.run(X, y_shifted) shifted_binomial_mixture_detected = shifted_binomial_mixture_test.is_exchangeable print("\nNon-exchangeable dataset") print("------------------------") print( f"PValuePermutationTest: data exchangeability={shifted_pvalue_detected}. Detection after {len(shifted_pvalue_test.p_values) - 1} permutations." ) print( f"SequentialMonteCarloTest (aggressive): data exchangeability={shifted_aggressive_detected}. Detection after {len(shifted_aggressive_test.p_values) - 1} permutations." ) print( f"SequentialMonteCarloTest (binomial): data exchangeability={shifted_binomial_detected}. Detection after {len(shifted_binomial_test.p_values) - 1} permutations." ) print( "SequentialMonteCarloTest (binomial_mixture): " f"data exchangeability={shifted_binomial_mixture_detected}. Detection after {len(shifted_binomial_mixture_test.p_values) - 1} permutations." ) plt.figure(figsize=(8, 4)) plt.plot(shifted_pvalue_test.p_values, label="PValuePermutationTest", zorder=4) plt.plot( shifted_aggressive_test.p_values, label="SMC aggressive", linestyle="-", linewidth=3.0, zorder=1, ) plt.plot( shifted_binomial_test.p_values, label="SMC binomial", linestyle="--", linewidth=2.0, zorder=2, ) plt.plot( shifted_binomial_mixture_test.p_values, label="SMC binomial_mixture", linestyle=":", linewidth=2.0, zorder=3, ) plt.axhline( test_level, color="black", linestyle="--", label=f"test_level = {test_level:.2f}", ) plt.xlabel("Number of permutations") plt.ylabel("Running p-value") plt.title("Non-exchangeable residuals") plt.grid(alpha=0.3) plt.legend() plt.tight_layout() plt.show() .. image-sg:: /examples_exchangeability_testing/2-advanced-analysis/images/sphx_glr_plot_permutation_test_002.png :alt: Non-exchangeable residuals :srcset: /examples_exchangeability_testing/2-advanced-analysis/images/sphx_glr_plot_permutation_test_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Non-exchangeable dataset ------------------------ PValuePermutationTest: data exchangeability=False. Detection after 200 permutations. SequentialMonteCarloTest (aggressive): data exchangeability=False. Detection after 101 permutations. SequentialMonteCarloTest (binomial): data exchangeability=False. Detection after 200 permutations. SequentialMonteCarloTest (binomial_mixture): data exchangeability=False. Detection after 101 permutations. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 3.211 seconds) .. _sphx_glr_download_examples_exchangeability_testing_2-advanced-analysis_plot_permutation_test.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_permutation_test.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_permutation_test.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_permutation_test.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_