Skip to content

MAPIE

Overview

scikit-learn-contrib/MAPIE

API Reference¶

Complete API documentation for MAPIE.

Regression ¶

Conformalizers¶

Item	Description
`TimeSeriesRegressor`	Prediction intervals with out-of-fold residuals for time series. This class only has two valid `method` : `"enbpi"` or `"aci"`
`SplitConformalRegressor`	Computes prediction intervals using the split conformal regression technique:
`CrossConformalRegressor`	Computes prediction intervals using the cross conformal regression technique:
`JackknifeAfterBootstrapRegressor`	Computes prediction intervals using the jackknife-after-bootstrap technique:
`ConformalizedQuantileRegressor`	Computes prediction intervals using the conformalized quantile regression technique:
`CrossConformalizedQuantileRegressor`	Computes prediction intervals using the cross-conformalized quantile regression technique.

Classification ¶

Conformalizers¶

Item	Description
`SplitConformalClassifier`	Computes prediction sets using the split conformal classification technique:
`CrossConformalClassifier`	Computes prediction sets using the cross conformal classification technique:

Conditional Conformal Prediction ¶

Conformalizers¶

Item	Description
`ConditionalSplitConformalRegressor`	Shared machinery for the conditional conformal procedure of Gibbs et al.
`ConditionalSplitConformalClassifier`	Shared machinery for the conditional conformal procedure of Gibbs et al.

Conformity Scores ¶

Regression¶

Item	Description
`BaseRegressionScore`	Base conformity score class for regression task.
`AbsoluteConformityScore`	Absolute conformity score.
`GammaConformityScore`	Gamma conformity score.
`QuantileRegressionScore`	Quantile conformity score for quantile regression.
`AbsoluteQuantileRegressionScore`	Absolute conformity score for quantile regression.
`ResidualNormalisedScore`	Residual Normalised score.
`StdConformityScore`	Standardized non-conformity score.

Classification¶

Item	Description
`BaseClassificationScore`	Base conformity score class for classification task.
`NaiveConformityScore`	Naive classification non-conformity score method that is based on the cumulative sum of probabilities until the 1-alpha threshold.
`LACConformityScore`	Least Ambiguous set-valued Classifier (LAC) method-based non conformity score (also formerly called `"score"`).
`APSConformityScore`	Adaptive Prediction Sets (APS) method-based non-conformity score. It is based on the sum of the softmax outputs of the labels until the true label is reached, on the conformalization set. See [1] for more details.
`RAPSConformityScore`	Regularized Adaptive Prediction Sets (RAPS) method-based non-conformity score. It uses the same technique as `APSConformityScore` class but with a penalty term to reduce the size of prediction sets. See [1] for more details. For now, this method only works with `"prefit"` and `"split"` strategies.
`TopKConformityScore`	Top-K method-based non-conformity score.

Metrics ¶

Regression¶

Item	Description
`regression_mean_width_score`	Effective mean width score obtained by the prediction intervals.
`regression_coverage_score`	Effective coverage obtained by the prediction intervals.
`regression_ssc`	Compute Size-Stratified Coverage metrics proposed in [3] that is the conditional coverage conditioned by the size of the intervals. The intervals are ranked by their size (ascending) and then divided into num_bins groups: one value of coverage by groups is computed.
`regression_ssc_score`	Aggregate by the minimum for each confidence level the Size-Stratified Coverage [3]: returns the maximum violation of the conditional coverage (with the groups defined).
`hsic`	Compute the square root of the hsic coefficient. HSIC is Hilbert-Schmidt independence criterion that is a correlation measure. Here we use it as proposed in [4], to compute the correlation between the indicator of coverage and the interval size.
`coverage_width_based`	Coverage Width-based Criterion (CWC) obtained by the prediction intervals.
`regression_mwi_score`	The Winkler score, proposed by Winkler (1972), is a measure used to evaluate prediction intervals, combining the length of the interval with a penalty that increases proportionally to the distance of an observation outside the interval.

Classification¶

Item	Description
`classification_mean_width_score`	Mean width of prediction set output by `mapie.classification._MapieClassifier`.
`classification_coverage_score`	Effective coverage score obtained by the prediction sets.
`classification_ssc`	Compute Size-Stratified Coverage metrics proposed in [3] that is the conditional coverage conditioned by the size of the predictions sets. The sets are ranked by their size (ascending) and then divided into num_bins groups: one value of coverage by groups is computed.
`classification_ssc_score`	Aggregate by the minimum for each confidence level the Size-Stratified Coverage [3]: returns the maximum violation of the conditional coverage (with the groups defined).

Conditional Coverage¶

Item	Description
`coverage_gap`	Compute the coverage gap across groups.
`worst_slab_coverage`	Compute the worst-case slab coverage.
`excess_risk_target_coverage`	Compute the excess risk of the target coverage.

Uncertainty¶

Item	Description
`auroc`	Area Under the ROC Curve measuring how well confidence ranks correct predictions.
`auarc`	Area Under the Accuracy-Rejection Curve (AUARC).

Calibration¶

Item	Description
`expected_calibration_error`	The Expected Calibration Error (ECE), which measures the difference between confidence scores and accuracy per bin.
`top_label_ece`	The Top-Label ECE which is a method adapted to fit the ECE to a Top-Label setting [2].
`add_jitter`	Add a tiny normal distributed perturbation to an array x.
`sort_xy_by_y`	Sort two arrays x and y according to y values.
`cumulative_differences`	Compute the cumulative difference between y_true and y_score, both ordered according to y_scores array.
`length_scale`	Compute the mean square root of the sum of s * (1 - s). This is basically the standard deviation of the cumulative differences.
`kolmogorov_smirnov_statistic`	Compute Kolmogorov-smirnov's statistic for calibration test. Also called ECCE-MAD (Estimated Cumulative Calibration Errors - Maximum Absolute Deviation). The closer to zero, the better the scores are calibrated. Indeed, if the scores are perfectly calibrated, the cumulative differences between `y_true` and `y_score` should share the same properties of a standard Brownian motion asymptotically.
`kolmogorov_smirnov_cdf`	Compute the Kolmogorov-smirnov cumulative distribution function (CDF) for the float x. This is interpreted as the CDF of the maximum absolute value of the standard Brownian motion over the unit interval [0, 1]. The function is approximated by its power series, truncated so as to hit machine precision error.
`kolmogorov_smirnov_p_value`	Compute Kolmogorov Smirnov p-value. Deduced from the corresponding statistic and CDF. It represents the probability of the observed statistic under the null hypothesis of perfect calibration.
`kuiper_statistic`	Compute Kuiper's statistic for calibration test. Also called ECCE-R (Estimated Cumulative Calibration Errors - Range). The closer to zero, the better the scores are calibrated. Indeed, if the scores are perfectly calibrated, the cumulative differences between `y_true` and `y_score` should share the same properties of a standard Brownian motion asymptotically.
`kuiper_cdf`	Compute the Kuiper cumulative distribution function (CDF) for the float x. This is interpreted as the CDF of the range of the standard Brownian motion over the unit interval [0, 1]. The function is approximated by its power series, truncated so as to hit machine precision error.
`kuiper_p_value`	Compute Kuiper statistic p-value. Deduced from the corresponding statistic and CDF. It represents the probability of the observed statistic under the null hypothesis of perfect calibration.
`spiegelhalter_statistic`	Compute Spiegelhalter's statistic for calibration test. The closer to zero, the better the scores are calibrated. Indeed, if the scores are perfectly calibrated, the Brier score simplifies to an expression whose expectancy and variance are easy to compute. The statistic is no more that a z-score on this normalized expression.
`spiegelhalter_p_value`	Compute Spiegelhalter statistic p-value. Deduced from the corresponding statistic and CDF, which is no more than the normal distribution. It represents the probability of the observed statistic under the null hypothesis of perfect calibration.

Risk Control ¶

Controllers¶

Item	Description
`ConditionalExpectedRiskController`	Conformal risk control with a learned, input-dependent prediction parameter.
`MultiLabelClassificationController`	Prediction sets for multilabel-classification.
`SemanticSegmentationController`	Risk controller for semantic segmentation tasks, inheriting from MultiLabelClassificationController.
`BinaryClassificationController`	Controls the risk or performance of a binary classifier.

Risks¶

Item	Description
`miscoverage_loss`	Define a PyTorch-compatible loss for automatically adaptive CRC (AA-CRC).
`recall_loss`	Define a PyTorch-compatible loss for automatically adaptive CRC (AA-CRC).
`BinaryRisk`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`BinaryClassificationRisk`	Deprecated alias for `BinaryRisk`.
`ContinuousRisk`	Define a continuous risk (or performance metric) to be used with a risk controller, for problems where predictions and targets are real-valued (e.g. regression).
`RiskLoss`	Define a PyTorch-compatible loss for automatically adaptive CRC (AA-CRC).
`accuracy`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`false_positive_rate`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`mae`	Define a continuous risk (or performance metric) to be used with a risk controller, for problems where predictions and targets are real-valued (e.g. regression).
`mean_absolute_error`	Define a continuous risk (or performance metric) to be used with a risk controller, for problems where predictions and targets are real-valued (e.g. regression).
`mean_squared_error`	Define a continuous risk (or performance metric) to be used with a risk controller, for problems where predictions and targets are real-valued (e.g. regression).
`mse`	Define a continuous risk (or performance metric) to be used with a risk controller, for problems where predictions and targets are real-valued (e.g. regression).
`precision`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`recall`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`predicted_positive_fraction`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`positive_predictive_value`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`negative_predictive_value`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.
`abstention_rate`	Define a risk (or a performance metric) to be used with the BinaryClassificationController. Predefined instances are implemented, see `mapie.risk_control.precision`, `mapie.risk_control.recall`, `mapie.risk_control.accuracy`, `mapie.risk_control.false_positive_rate`, and `mapie.risk_control.predicted_positive_fraction`.

FWER Procedures¶

Item	Description
`control_fwer`	Apply a Family-Wise Error Rate (FWER) control procedure.
`FWERProcedure`	Base class for procedures controlling the Family-Wise Error Rate (FWER).
`FWERBonferroniHolm`	Holm step-down procedure for controlling the FWER [1].
`FWERFixedSequenceTesting`	Fixed Sequential Testing (ascending) procedure with multi-start for controlling the Family-Wise Error Rate (FWER) [1].
`FWERBonferroniCorrection`	Bonferroni procedure for controlling the FWER [1].

Calibration ¶

Calibrators¶

Item	Description
`TopLabelCalibrator`	Top-label calibration for multi-class problems. Performs a calibration on the class with the highest score given both score and class, see section 2 of [1].
`VennAbersCalibrator`	Venn-ABERS calibration for binary and multi-class problems.

Exchangeability Testing ¶

High-level Interfaces¶

Item	Description
`FixedDatasetExchangeabilityTest`	Run one or several exchangeability tests on a labeled dataset.
`OnlineExchangeabilityTest`	Monitor exchangeability online with one or several martingale tests.
`RiskMonitoring`	Monitor a risk on an online stream relative to a reference set.

Individual Tests¶

Item	Description
`OnlineMartingaleTest`	Online test of exchangeability based on conformal p-values and test martingales.
`PValuePermutationTest`	Permutation test based on p-values computed from conformity scores.
`PermutationTest`	Base class for exchangeability tests based on permutations.
`SequentialMonteCarloTest`	Sequential Monte Carlo exchangeability test.

Utilities ¶

Data Splitting¶

Item	Description
`train_conformalize_test_split`	Split arrays or matrices into train, conformalization and test subsets.

Resampling¶

Item	Description
`Subsample`	Generate a sampling method, that resamples the training set with possible bootstraps. It can be used as cv argument in `JackknifeAfterBootstrapRegressor`.
`BlockBootstrap`	Generate a sampling method, that block bootstraps the training set. It can replace KFold, LeaveOneOut or SubSample as cv argument in the TimeSeriesRegressor class.