:orphan:

.. _exchangeability_testing_examples:

All exchangeability testing examples
====================================

Following is a collection of examples demonstrating how to use MAPIE for
exchangeability testing.



.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. thumbnail-parent-div-close

.. raw:: html

    </div>


1. Quickstart
-------------

The following examples introduce MAPIE exchangeability testing on a fixed
dataset and on an online stream.



.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This quickstart demonstrates how to test exchangeability on a labeled online stream, i.e. data arriving sequentially in batches.">

.. only:: html

  .. image:: /examples_exchangeability_testing/1-quickstart/images/thumb/sphx_glr_plot_exchangeability_online_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_1-quickstart_plot_exchangeability_online.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Exchangeability testing on an online stream</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This quickstart demonstrates how to test exchangeability on a fixed dataset.">

.. only:: html

  .. image:: /examples_exchangeability_testing/1-quickstart/images/thumb/sphx_glr_plot_exchangeability_fixed_dataset_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_1-quickstart_plot_exchangeability_fixed_dataset.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Exchangeability testing on a fixed dataset</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This quickstart demonstrates how to use RiskMonitoring to track the misclassification risk of a deployed binary classifier on an online stream.">

.. only:: html

  .. image:: /examples_exchangeability_testing/1-quickstart/images/thumb/sphx_glr_plot_risk_monitoring_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_1-quickstart_plot_risk_monitoring.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Detect harmful shifts with RiskMonitoring</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


2. Advanced analysis
--------------------

The following examples use MAPIE for discussing more complex exchangeability
testing problems.



.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example illustrates how to run PValuePermutationTest on a multiclass classification problem using a user-provided fitted SplitConformalClassifier. We use the &quot;top_k&quot; conformity score instead of the default one, compare an exchangeable and a non-exchangeable held-out dataset, and then continue with the usual conformalization and prediction pipeline.">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_permutation_test_with_fitted_classifier_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_permutation_test_with_fitted_classifier.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Permutation test with a fitted classifier</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="When a predictive model is deployed in production, a common question arises: are newly arriving labeled observations still exchangeable with the data used for model development?">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_online_martingale_tests_for_deployed_regression_model_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_online_martingale_tests_for_deployed_regression_model.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Online martingale exchangeability tests for a deployed regressor</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example illustrates how to run PValuePermutationTest  and SequentialMonteCarloTest on a toy regression problem. We compare a dataset with exchangeable residuals to a dataset where the second half of the residuals is shifted, which breaks the exchangeability assumption used by conformal methods.">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_permutation_test_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_permutation_test.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Permutation test for exchangeability</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="When a predictive model is deployed in production, a common question arises: are the newly arriving labeled observations still exchangeable with the data used during model development?">

.. only:: html

  .. image:: /examples_exchangeability_testing/2-advanced-analysis/images/thumb/sphx_glr_plot_online_martingale_tests_for_deployed_classification_model_thumb.png
    :alt:

  :ref:`sphx_glr_examples_exchangeability_testing_2-advanced-analysis_plot_online_martingale_tests_for_deployed_classification_model.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Online martingale exchangeability tests for a deployed classifier</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:
   :includehidden:


   /examples_exchangeability_testing/1-quickstart/index.rst
   /examples_exchangeability_testing/2-advanced-analysis/index.rst


.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-gallery

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download all examples in Python source code: examples_exchangeability_testing_python.zip </examples_exchangeability_testing/examples_exchangeability_testing_python.zip>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download all examples in Jupyter notebooks: examples_exchangeability_testing_jupyter.zip </examples_exchangeability_testing/examples_exchangeability_testing_jupyter.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_