.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples_classification/1-quickstart/plot_quickstart_classification.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_classification_1-quickstart_plot_quickstart_classification.py: ====================================================== Use MAPIE to plot prediction sets ====================================================== In this example, we explain how to use MAPIE on a basic classification setting. .. GENERATED FROM PYTHON SOURCE LINES 10-12 We will use MAPIE to estimate prediction sets on a two-dimensional dataset with three labels. .. GENERATED FROM PYTHON SOURCE LINES 12-24 .. code-block:: default import numpy as np from sklearn.neighbors import KNeighborsClassifier from sklearn.datasets import make_blobs from matplotlib import pyplot as plt from matplotlib.colors import ListedColormap from mapie.utils import train_conformalize_test_split from mapie.classification import SplitConformalClassifier from mapie.metrics.classification import classification_coverage_score np.random.seed(42) .. GENERATED FROM PYTHON SOURCE LINES 25-26 Firstly, let us create our dataset: .. GENERATED FROM PYTHON SOURCE LINES 26-34 .. code-block:: default X, y = make_blobs(n_samples=500, n_features=2, centers=3, cluster_std=3.4) (X_train, X_conformalize, X_test, y_train, y_conformalize, y_test) = train_conformalize_test_split( X, y, train_size=0.4, conformalize_size=0.4, test_size=0.2 ) .. GENERATED FROM PYTHON SOURCE LINES 35-41 We fit our training data with a KNN estimator. Then, we initialize a :class:`~mapie.classification.SplitConformalClassifier` using our estimator, indicating that it has already been fitted with `prefit=True`. Lastly, we compute the prediction sets with the desired confidence level using the ``conformalize`` and ``predict_set`` methods. .. GENERATED FROM PYTHON SOURCE LINES 41-52 .. code-block:: default classifier = KNeighborsClassifier(n_neighbors=10) classifier.fit(X_train, y_train) confidence_level = 0.95 mapie_classifier = SplitConformalClassifier( estimator=classifier, confidence_level=confidence_level, prefit=True ) mapie_classifier.conformalize(X_conformalize, y_conformalize) y_pred, y_pred_set = mapie_classifier.predict_set(X_test) .. GENERATED FROM PYTHON SOURCE LINES 53-58 ``y_pred`` represents the point predictions as a ``np.ndarray`` of shape ``(n_samples)``. ``y_pred_set`` corresponds to the prediction sets as a ``np.ndarray`` of shape ``(n_samples, 3, 1)``. This array contains only boolean values: ``True`` if the label is included in the prediction set, and ``False`` if not. .. GENERATED FROM PYTHON SOURCE LINES 60-62 Finally, we can easily compute the coverage score (i.e., the proportion of times the true labels fall within the predicted sets). .. GENERATED FROM PYTHON SOURCE LINES 62-68 .. code-block:: default coverage_score = classification_coverage_score(y_test, y_pred_set) print(f"For a confidence level of {confidence_level:.2f}, " f"the target coverage is {confidence_level:.3f}, " f"and the effective coverage is {coverage_score[0]:.3f}.") .. rst-class:: sphx-glr-script-out Out: .. code-block:: none For a confidence level of 0.95, the target coverage is 0.950, and the effective coverage is 0.980. .. GENERATED FROM PYTHON SOURCE LINES 69-73 In this example, the effective coverage is slightly above the target coverage (i.e., 0.95), indicating that the confidence level we set has been reached. Therefore, we can confirm that the prediction sets effectively contain the true label more than 95% of the time. .. GENERATED FROM PYTHON SOURCE LINES 75-78 Now, let us plot the confidence regions across the plane. This plot will give us insights about what the prediction set looks like for each point. .. GENERATED FROM PYTHON SOURCE LINES 78-113 .. code-block:: default x_min, x_max = np.min(X[:, 0]), np.max(X[:, 0]) y_min, y_max = np.min(X[:, 1]), np.max(X[:, 1]) step = 0.1 xx, yy = np.meshgrid(np.arange(x_min, x_max, step), np.arange(y_min, y_max, step)) X_test_mesh = np.stack([xx.ravel(), yy.ravel()], axis=1) y_pred_set = mapie_classifier.predict_set(X_test_mesh)[1][:, :, 0] cmap_back = ListedColormap( [(0.7803921568627451, 0.9137254901960784, 0.7529411764705882), (0.9921568627450981, 0.8156862745098039, 0.6352941176470588), (0.6196078431372549, 0.6039215686274509, 0.7843137254901961), (0.7764705882352941, 0.8588235294117647, 0.9372549019607843), (0.6196078431372549, 0.6039215686274509, 0.7843137254901961), (0.6196078431372549, 0.6039215686274509, 0.7843137254901961)] ) cmap_dots = ListedColormap( [(0.19215686274509805, 0.5098039215686274, 0.7411764705882353), (0.9019607843137255, 0.3333333333333333, 0.050980392156862744), (0.19215686274509805, 0.6392156862745098, 0.32941176470588235)] ) plt.scatter( X_test_mesh[:, 0], X_test_mesh[:, 1], c=np.ravel_multi_index(y_pred_set.T, (2, 2, 2)), cmap=cmap_back, marker='.', s=10 ) plt.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_dots) plt.xlabel("x1") plt.ylabel("x2") plt.title("Confidence regions with KNN") plt.show() .. image-sg:: /examples_classification/1-quickstart/images/sphx_glr_plot_quickstart_classification_001.png :alt: Confidence regions with KNN :srcset: /examples_classification/1-quickstart/images/sphx_glr_plot_quickstart_classification_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 114-120 On the plot above, the dots represent the samples from our dataset, with their color indicating their respective label. The blue, orange and green zones correspond to prediction sets containing only the blue label, orange label and green label respectively. The purple zone represents areas where the prediction sets contain more than one label, indicating that the model is uncertain. .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 0.869 seconds) .. _sphx_glr_download_examples_classification_1-quickstart_plot_quickstart_classification.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_quickstart_classification.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_quickstart_classification.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_