MAPIE Logo

Getting Started

  • Quick Start with MAPIE
    • 1. Download and install the module
    • 2. Regression
    • 3. Classification
    • 4. Risk Control
  • The conformalization (“calibration”) set
    • 1. Split conformal predictions
      • Split conformal predictions with a pre-trained model
      • Split conformal predictions with an untrained model
    • 2. Cross conformal predictions
  • Choosing the right algorithm
  • MAPIE v1 release notes
    • Introduction
    • API changes overview
    • Python, scikit-learn and NumPy versions support
    • API changes in detail
      • Regression and classification API changes (excluding time series)
        • Classes
        • Workflow and methods
        • Parameters
      • Other API changes
        • Time series
        • Risk control
        • Calibration
        • Mondrian
        • Metrics
        • Conformity scores

Measure predictions uncertainty

  • Prediction intervals (regression)
    • Plot prediction intervals
    • Use a pre-trained model
    • All regression examples
      • 1. Quickstart
        • Plot prediction intervals
        • Data with gamma distribution
        • Use a pre-trained model
        • Data with uneven uncertainty
        • Data with constant uncertainty
        • Tutorial for time series
      • 2. Advanced analysis
        • Conformal Predictive Distribution
        • The symmetric correction parameter in conformalized quantile regression
        • Hyperparameters tuning with cross-conformal regression
        • Estimating aleatoric and epistemic uncertainties
        • EnbPI technique for time series
        • Focus on intervals width
        • Focus on residual normalised score
        • Focus on local (or “conditional”) coverage
        • Conformalized quantile regression on gamma distributed data
        • Coverage validity for regression tasks
        • 1. Estimating the aleatoric uncertainty of homoscedastic noisy data
        • 2. Estimating the aleatoric uncertainty of heteroscedastic noisy data
        • 3. Estimating the epistemic uncertainty of out-of-distribution data
        • 4. More Jupyter notebooks for regression
      • 3. Simulations from scientific articles
        • Predictive inference with the jackknife+, Foygel-Barber et al. (2020)
        • Adaptive conformal predictions for time series, Zaffran et al. (2022)
        • Predictive inference is free with the Jackknife+-after-Bootstrap, Kim et al. (2020)
      • 4. Other notebooks
    • Theoretical Description
    • Theoretical Description for Conformity Scores
  • Prediction sets (classification)
    • Plot prediction sets
    • All classification examples
      • 1. Quickstart examples
        • Plot prediction sets
      • 2. Advanced analysis
        • LAC and APS methods explained
        • Set prediction example in the binary classification setting
        • Cross conformal classification explained
      • 3. Simulations from scientific articles
        • Least Ambiguous Set-Valued Classifiers with Bounded Error Levels, Sadinle et al. (2019)
      • 4. Other notebooks
    • Theoretical Description
    • The binary classification case
      • Set prediction example in the binary classification setting
      • Theoretical Description

Control prediction errors

  • Getting started with risk control
  • Precision control for a binary classifier
  • All risk control examples
    • 1. Quickstart examples
      • Precision control for a binary classifier
    • 2. Advanced analysis
      • Control the risk of a multi-label classifier
      • Control risk of a binary classifier with multiple prediction parameters
      • Recall control for semantic segmentation
      • Precision control for semantic segmentation
      • Comparing FWER methods for risk control in binary classification
      • Split Fixed Sequence Testing for Precision Control under Multiple Testing
      • Control multiple risks of a binary classifier
      • Risk Control for LLM as a Judge with Abstention

Calibrate classifiers

  • Theoretical Description
  • Calibration examples
    • 1. Quickstart examples
      • Calibrating binary classifier with Venn-ABERS
      • Testing for calibration in binary classification settings
      • Calibrating multi-class classifier with Venn-ABERS
    • 2. Advanced analysis
      • Evaluating the asymptotic convergence of p-values
  • Calibration notebooks

Exchangeability Testing

  • Exchangeability Tests
  • All exchangeability testing examples
    • 1. Quickstart
      • Exchangeability testing on an online stream
      • Exchangeability testing on a fixed dataset
      • Detect harmful shifts with RiskMonitoring
    • 2. Advanced analysis
      • Permutation test with a fitted classifier
      • Online martingale exchangeability tests for a deployed regressor
      • Permutation test for exchangeability
      • Online martingale exchangeability tests for a deployed classifier

Question & Answers

  • Metrics: how to measure conformal prediction performance?
    • 1. General Metrics
      • Regression Coverage Score
      • Regression Mean Width Score
      • Classification Coverage Score
      • Classification Mean Width Score
      • Size-Stratified Coverage
      • Hilbert-Schmidt Independence Criterion
      • Coverage Width-Based Criterion
      • Mean Winkler Interval Score
    • 2. Calibration Metrics
      • Expected Calibration Error
      • Top-Label Expected Calibration Error (Top-Label ECE)
      • Cumulative Differences
      • Kolmogorov-Smirnov Statistic for Calibration
      • Kuiper’s Test
      • Spiegelhalter’s Test
    • References
  • Mondrian: how to use prior knowledge on groups when measuring uncertainty?
    • 1. Create the noisy dataset
    • 2. Split the dataset into a training set, a conformalization set, and a test set
    • 3. Fit a random forest regressor on the training set
    • 4. Build the classical conformal prediction intervals
      • Conformalize a SplitConformalRegressor on the conformalization set
      • Predict the prediction intervals on the test set
      • Evaluate the coverage score by group
    • 5. Build the Mondrian conformal prediction intervals
      • Conformalize a SplitConformalRegressor on the conformalization set for each group
      • Predict the prediction intervals on the test set
    • 6. Compare the coverage by partition, plot both methods side by side
    • Theoretical Description
      • References
  • How to control LLM risks?

API

  • MAPIE API
    • Regression
      • Conformalizers
        • mapie.regression.SplitConformalRegressor
        • mapie.regression.CrossConformalRegressor
        • mapie.regression.JackknifeAfterBootstrapRegressor
        • mapie.regression.ConformalizedQuantileRegressor
        • mapie.regression.TimeSeriesRegressor
      • Metrics
        • mapie.metrics.regression.regression_coverage_score
        • mapie.metrics.regression.regression_mean_width_score
        • mapie.metrics.regression.regression_ssc
        • mapie.metrics.regression.regression_ssc_score
        • mapie.metrics.regression.hsic
        • mapie.metrics.regression.coverage_width_based
        • mapie.metrics.regression.regression_mwi_score
      • Conformity Scores
        • mapie.conformity_scores.BaseRegressionScore
        • mapie.conformity_scores.AbsoluteConformityScore
        • mapie.conformity_scores.GammaConformityScore
        • mapie.conformity_scores.ResidualNormalisedScore
      • Resampling
        • mapie.subsample.BlockBootstrap
        • mapie.subsample.Subsample
    • Classification
      • Conformalizers
        • mapie.classification.SplitConformalClassifier
        • mapie.classification.CrossConformalClassifier
      • Metrics
        • mapie.metrics.classification.classification_coverage_score
        • mapie.metrics.classification.classification_mean_width_score
        • mapie.metrics.classification.classification_ssc
        • mapie.metrics.classification.classification_ssc_score
      • Conformity Scores
        • mapie.conformity_scores.BaseClassificationScore
        • mapie.conformity_scores.NaiveConformityScore
        • mapie.conformity_scores.LACConformityScore
        • mapie.conformity_scores.APSConformityScore
        • mapie.conformity_scores.RAPSConformityScore
        • mapie.conformity_scores.TopKConformityScore
    • Risk Control
      • mapie.risk_control.MultiLabelClassificationController
        • MultiLabelClassificationController
        • Examples using mapie.risk_control.MultiLabelClassificationController
      • mapie.risk_control.BinaryClassificationController
        • BinaryClassificationController
        • Examples using mapie.risk_control.BinaryClassificationController
      • mapie.risk_control.SemanticSegmentationController
        • SemanticSegmentationController
        • Examples using mapie.risk_control.SemanticSegmentationController
      • mapie.risk_control.BinaryRisk
        • BinaryRisk
        • Examples using mapie.risk_control.BinaryRisk
      • mapie.risk_control.BinaryClassificationRisk
        • BinaryClassificationRisk
    • Calibration
      • Conformalizer
        • mapie.calibration.TopLabelCalibrator
      • Metrics
        • mapie.metrics.calibration.expected_calibration_error
        • mapie.metrics.calibration.top_label_ece
        • mapie.metrics.calibration.cumulative_differences
        • mapie.metrics.calibration.kolmogorov_smirnov_cdf
        • mapie.metrics.calibration.kolmogorov_smirnov_p_value
        • mapie.metrics.calibration.kolmogorov_smirnov_statistic
        • mapie.metrics.calibration.kuiper_cdf
        • mapie.metrics.calibration.kuiper_p_value
        • mapie.metrics.calibration.kuiper_statistic
        • mapie.metrics.calibration.length_scale
        • mapie.metrics.calibration.spiegelhalter_p_value
        • mapie.metrics.calibration.spiegelhalter_statistic
    • Utils
      • mapie.utils.train_conformalize_test_split
        • train_conformalize_test_split()
        • Examples using mapie.utils.train_conformalize_test_split
MAPIE
📢 Notice:
The MAPIE documentation for versions up to 1.4.0 is hosted here.

Starting from version 1.5.0, the documentation has moved to:
scikit-learn-contrib.github.io/MAPIE

This Read the Docs site will not receive updates for versions 1.5.0 and above.
  • Prediction sets (classification)
  • The binary classification case
  • View page source

The binary classification case

  • Set prediction example in the binary classification setting
    • 1. Conformal Prediction method using the softmax score of the true label
  • Theoretical Description
    • 1. Set Prediction
    • 2. Probabilistic Prediction
    • 3. Calibration
    • References
Previous Next

© Copyright 2026, Capgemini Invent.

Built with Sphinx using a theme provided by Read the Docs.