MAPIE Logo

Getting Started

  • Quick Start with MAPIE
    • 1. Download and install the module
    • 2. Regression
    • 3. Classification
    • 4. Risk Control
  • The conformalization (or “calibration”) set
    • 1. Split conformal predictions
      • Split conformal predictions with a pre-trained model
      • Split conformal predictions with an untrained model
    • 2. Cross conformal predictions
  • Choosing the right algorithm
  • MAPIE v1 release notes
    • Introduction
    • API changes overview
    • Python, scikit-learn and NumPy versions support
    • API changes in detail
      • Regression and classification API changes (excluding time series)
        • Classes
        • Workflow and methods
        • Parameters
      • Other API changes
        • Time series
        • Risk control
        • Calibration
        • Mondrian
        • Metrics
        • Conformity scores

Measure predictions uncertainty

  • Prediction intervals (regression)
    • Choose the right algorithm
    • Use MAPIE to plot prediction intervals
    • Use MAPIE with a pre-trained model
      • 1. Use a neural network
      • 1.1 Pre-train a neural network
      • 1.2 Use MAPIE to conformalize the models
      • 1.3 Plot results
      • 2. Use LGBM models
      • 2.1 Pre-train LGBM models
      • 2.2 Use MAPIE to conformalize the models
      • 2.3 Plot results
    • All regression examples
      • 1. Quickstart
      • 2. Advanced analysis
      • 3. Simulations from scientific articles
      • 4. Other notebooks
        • 1. Quickstart
        • 2. Advanced analysis
        • 3. Simulations from scientific articles
        • 4. Other notebooks
    • Theoretical Description
      • 1. The “Naive” method
      • 2. The split method
      • 3. The jackknife method
      • 4. The jackknife+ method
      • 5. The jackknife-minmax method
      • 6. The CV+ method
      • 7. The CV and CV-minmax methods
      • 8. The jackknife+-after-bootstrap method
      • 9. The Conformalized Quantile Regression (CQR) Method
        • Notations and Definitions
        • Mathematical Formulation
      • 10. The ensemble batch prediction intervals (EnbPI) method
      • Key takeaways
      • References
    • Theoretical Description for Conformity Scores
      • 1. The absolute residual score
      • 2. The gamma score
      • 3. The residual normalized score
      • Key takeaways
      • References
  • Prediction sets (classification)
    • Choosing the right algorithm
    • Use MAPIE to plot prediction sets
    • All classification examples
      • 1. Quickstart examples
      • 2. Advanced analysis
      • 3. Simulations from scientific articles
      • 4. Other notebooks
        • 1. Quickstart examples
        • 2. Advanced analysis
        • 3. Simulations from scientific articles
        • 4. Other notebooks
    • Theoretical Description
      • 1. LAC
      • 2. Top-K
      • 3. Adaptive Prediction Sets (APS)
      • 4. Regularized Adaptive Prediction Sets (RAPS)
      • 5. Split- and cross-conformal methods
      • References
    • The binary classification case
      • Set prediction example in the binary classification setting
        • 1. Conformal Prediction method using the softmax score of the true label
      • Theoretical Description
        • 1. Set Prediction
        • 2. Probabilistic Prediction
        • 3. Calibration
        • References

Control prediction errors

  • Getting started with risk control in MAPIE
    • Overview
    • 1. What is risk control?
    • 2. Theoretical description
      • 2.1 Risk-Controlling Prediction Sets
        • 2.1.1 General settings
        • 2.1.2 Bounds calculation
      • 2.2 Conformal Risk Control
      • 2.3 Learn Then Test
    • References
  • Use MAPIE to control the precision of a binary classifier
  • All risk control examples
    • 1. Quickstart examples
    • 2. Advanced analysis
      • 1. Quickstart examples
        • Use MAPIE to control the precision of a binary classifier
      • 2. Advanced analysis
        • Use MAPIE to control risk of a binary classifier with multiple prediction parameters
        • Tutorial for recall and precision control for multi-label classification
        • Use MAPIE to control multiple risks of a binary classifier
  • Additional resources: external risk control package

Calibrate multi-class predictions

  • Theoretical Description
    • Top-Label
    • References
  • Calibration examples
    • 1. Quickstart examples
    • 2. Advanced analysis
      • 1. Quickstart examples
        • Testing for calibration in binary classification settings
      • 2. Advanced analysis
        • Evaluating the asymptotic convergence of p-values
  • Calibration notebooks
    • 1. Top-label calibration for outputs of ML models : notebook

Question & Answers

  • Metrics: how to measure conformal prediction performance?
    • 1. General Metrics
      • Regression Coverage Score
      • Regression Mean Width Score
      • Classification Coverage Score
      • Classification Mean Width Score
      • Size-Stratified Coverage
      • Hilbert-Schmidt Independence Criterion
      • Coverage Width-Based Criterion
      • Mean Winkler Interval Score
    • 2. Calibration Metrics
      • Expected Calibration Error
      • Top-Label Expected Calibration Error (Top-Label ECE)
      • Cumulative Differences
      • Kolmogorov-Smirnov Statistic for Calibration
      • Kuiper’s Test
      • Spiegelhalter’s Test
    • References
  • Mondrian: how to use prior knowledge on groups when measuring uncertainty?
    • Tutorial: how to ensure fairness across groups with Mondrian
      • 1. Create the noisy dataset
      • 2. Split the dataset into a training set, a conformalization set, and a test set
      • 3. Fit a random forest regressor on the training set
      • 4. Build the classical conformal prediction intervals
        • Conformalize a SplitConformalRegressor on the conformalization set
        • Predict the prediction intervals on the test set
        • Evaluate the coverage score by group
      • 5. Build the Mondrian conformal prediction intervals
        • Conformalize a SplitConformalRegressor on the conformalization set for each group
        • Predict the prediction intervals on the test set
      • 6. Compare the coverage by partition, plot both methods side by side
    • Theoretical Description
      • References
  • How to control LLM risks?

API

  • MAPIE API
    • Regression
      • Conformalizers
        • mapie.regression.SplitConformalRegressor
        • mapie.regression.CrossConformalRegressor
        • mapie.regression.JackknifeAfterBootstrapRegressor
        • mapie.regression.ConformalizedQuantileRegressor
        • mapie.regression.TimeSeriesRegressor
      • Metrics
        • mapie.metrics.regression.regression_coverage_score
        • mapie.metrics.regression.regression_mean_width_score
        • mapie.metrics.regression.regression_ssc
        • mapie.metrics.regression.regression_ssc_score
        • mapie.metrics.regression.hsic
        • mapie.metrics.regression.coverage_width_based
        • mapie.metrics.regression.regression_mwi_score
      • Conformity Scores
        • mapie.conformity_scores.BaseRegressionScore
        • mapie.conformity_scores.AbsoluteConformityScore
        • mapie.conformity_scores.GammaConformityScore
        • mapie.conformity_scores.ResidualNormalisedScore
      • Resampling
        • mapie.subsample.BlockBootstrap
        • mapie.subsample.Subsample
    • Classification
      • Conformalizers
        • mapie.classification.SplitConformalClassifier
        • mapie.classification.CrossConformalClassifier
      • Metrics
        • mapie.metrics.classification.classification_coverage_score
        • mapie.metrics.classification.classification_mean_width_score
        • mapie.metrics.classification.classification_ssc
        • mapie.metrics.classification.classification_ssc_score
      • Conformity Scores
        • mapie.conformity_scores.BaseClassificationScore
        • mapie.conformity_scores.NaiveConformityScore
        • mapie.conformity_scores.LACConformityScore
        • mapie.conformity_scores.APSConformityScore
        • mapie.conformity_scores.RAPSConformityScore
        • mapie.conformity_scores.TopKConformityScore
    • Risk Control
      • mapie.risk_control.PrecisionRecallController
        • PrecisionRecallController
        • Examples using mapie.risk_control.PrecisionRecallController
      • mapie.risk_control.BinaryClassificationController
        • BinaryClassificationController
        • Examples using mapie.risk_control.BinaryClassificationController
      • mapie.risk_control.BinaryClassificationRisk
        • BinaryClassificationRisk
        • Examples using mapie.risk_control.BinaryClassificationRisk
    • Calibration
      • Conformalizer
        • mapie.calibration.TopLabelCalibrator
      • Metrics
        • mapie.metrics.calibration.expected_calibration_error
        • mapie.metrics.calibration.top_label_ece
        • mapie.metrics.calibration.cumulative_differences
        • mapie.metrics.calibration.kolmogorov_smirnov_cdf
        • mapie.metrics.calibration.kolmogorov_smirnov_p_value
        • mapie.metrics.calibration.kolmogorov_smirnov_statistic
        • mapie.metrics.calibration.kuiper_cdf
        • mapie.metrics.calibration.kuiper_p_value
        • mapie.metrics.calibration.kuiper_statistic
        • mapie.metrics.calibration.length_scale
        • mapie.metrics.calibration.spiegelhalter_p_value
        • mapie.metrics.calibration.spiegelhalter_statistic
    • Utils
      • mapie.utils.train_conformalize_test_split
        • train_conformalize_test_split()
        • Examples using mapie.utils.train_conformalize_test_split
MAPIE
  • Search

© Copyright 2022, Quantmetry.

Built with Sphinx using a theme provided by Read the Docs.