.. _sphx_glr_examples_exchangeability_testing_2-advanced-analysis:

.. _exchangeability_testing_examples_2:

2. Advanced analysis
--------------------

The following examples use MAPIE for discussing more complex exchangeability
testing problems.



.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example illustrates how to run PValuePermutationTest on a multiclass classification problem using a user-provided fitted SplitConformalClassifier. We use the &quot;top_k&quot; conformity score instead of the default one, compare an exchangeable and a non-exchangeable held-out dataset, and then continue with the usual conformalization and prediction pipeline.">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_permutation_test_with_fitted_classifier_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_permutation_test_with_fitted_classifier.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Permutation test with a fitted classifier</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="When a predictive model is deployed in production, a common question arises: are newly arriving labeled observations still exchangeable with the data used for model development?">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_online_martingale_tests_for_deployed_regression_model_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_online_martingale_tests_for_deployed_regression_model.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Online martingale exchangeability tests for a deployed regressor</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example illustrates how to run PValuePermutationTest  and SequentialMonteCarloTest on a toy regression problem. We compare a dataset with exchangeable residuals to a dataset where the second half of the residuals is shifted, which breaks the exchangeability assumption used by conformal methods.">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_permutation_test_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_permutation_test.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Permutation test for exchangeability</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="When a predictive model is deployed in production, a common question arises: are the newly arriving labeled observations still exchangeable with the data used during model development?">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_online_martingale_tests_for_deployed_classification_model_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_online_martingale_tests_for_deployed_classification_model.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Online martingale exchangeability tests for a deployed classifier</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples_exchangeability_testing/2-advanced-analysis/plot_permutation_test_with_fitted_classifier
   /examples_exchangeability_testing/2-advanced-analysis/plot_online_martingale_tests_for_deployed_regression_model
   /examples_exchangeability_testing/2-advanced-analysis/plot_permutation_test
   /examples_exchangeability_testing/2-advanced-analysis/plot_online_martingale_tests_for_deployed_classification_model