Conditional Conformal Prediction

Conformal prediction methods with conditional validity guarantees.

These estimators require the optional conditional dependency:

pip install "mapie[conditional]"

See Theory and the runnable regression and classification examples.

Conformalizers

mapie.conditional_conformal_prediction.ConditionalSplitConformalRegressor

ConditionalSplitConformalRegressor(
    feature_map: Callable,
    estimator: RegressorMixin = LinearRegression(),
    confidence_level: Union[float, Iterable[float]] = 0.9,
    conformity_score: Union[
        str, BaseRegressionScore
    ] = "absolute",
    prefit: bool = True,
    n_jobs: Optional[int] = None,
    verbose: int = 0,
    randomize: bool = False,
    exact: bool = True,
    infinite_params: Optional[dict] = None,
    seed: int = 0,
)

Bases: _ConditionalConformalMixin, SplitConformalRegressor

Split conformal regressor with conditional validity guarantees.

In addition to the parameters of :class:~mapie.regression.SplitConformalRegressor, this class accepts settings for the conditional conformal procedure.

PARAMETER	DESCRIPTION
feature_map	Function mapping covariates to a finite basis used for exact conditional guarantees. TYPE: Callable
estimator	Base regressor used to predict points. TYPE: RegressorMixin DEFAULT: LinearRegression()
confidence_level	Desired coverage probability of the prediction intervals. TYPE: float or iterable of float DEFAULT: 0.9
conformity_score	Method used to compute conformity scores. See :class:~mapie.regression.SplitConformalRegressor. TYPE: str or BaseRegressionScore DEFAULT: "absolute"
prefit	Whether the base regressor is already fitted. TYPE: bool DEFAULT: True
n_jobs	Number of parallel jobs when applicable. TYPE: int DEFAULT: None
verbose	Verbosity level. TYPE: int DEFAULT: 0
randomize	Whether to use randomization to make coverage exact rather than conservative. If False, predictions are deterministic and coverage may be slightly above the target level. If True, predictions use auxiliary randomness to match the target coverage level more exactly. TYPE: bool DEFAULT: False
exact	Compute the conditional score cutoff exactly rather than by binary search. TYPE: bool DEFAULT: True
infinite_params	Parameters for the RKHS component of the fit. Valid keys are kernel, gamma, and lambda. Currently the infinite-dimensional (RKHS) component is not implemented: requesting a kernel raises NotImplementedError at construction time. The supporting code is retained for future work. TYPE: dict DEFAULT: None

Source code in mapie/conditional_conformal_prediction.py

def __init__(
    self,
    feature_map: Callable,
    estimator: RegressorMixin = LinearRegression(),
    confidence_level: Union[float, Iterable[float]] = 0.9,
    conformity_score: Union[str, BaseRegressionScore] = "absolute",
    prefit: bool = True,
    n_jobs: Optional[int] = None,
    verbose: int = 0,
    randomize: bool = False,
    exact: bool = True,
    infinite_params: Optional[dict] = None,
    seed: int = 0,
) -> None:
    """
    Split conformal regressor with conditional validity guarantees.

    In addition to the parameters of
    :class:`~mapie.regression.SplitConformalRegressor`, this class accepts
    settings for the conditional conformal procedure.

    Parameters
    ----------
    feature_map : Callable
        Function mapping covariates to a finite basis used for exact
        conditional guarantees.

    estimator : RegressorMixin, default=LinearRegression()
        Base regressor used to predict points.

    confidence_level : float or iterable of float, default=0.9
        Desired coverage probability of the prediction intervals.

    conformity_score : str or BaseRegressionScore, default="absolute"
        Method used to compute conformity scores. See
        :class:`~mapie.regression.SplitConformalRegressor`.

    prefit : bool, default=True
        Whether the base regressor is already fitted.

    n_jobs : int, optional
        Number of parallel jobs when applicable.

    verbose : int, default=0
        Verbosity level.

    randomize : bool, default=False
        Whether to use randomization to make coverage exact rather than
        conservative.

        If False, predictions are deterministic and coverage may be slightly above
        the target level. If True, predictions use auxiliary randomness to match the
        target coverage level more exactly.

    exact : bool, default=True
        Compute the conditional score cutoff exactly rather than by binary
        search.

    infinite_params : dict, optional
        Parameters for the RKHS component of the fit. Valid keys are
        ``kernel``, ``gamma``, and ``lambda``. Currently the
        infinite-dimensional (RKHS) component is not implemented:
        requesting a ``kernel`` raises ``NotImplementedError`` at
        construction time. The supporting code is retained for future work.
    """
    super().__init__(
        estimator=estimator,
        confidence_level=confidence_level,
        conformity_score=conformity_score,
        prefit=prefit,
        n_jobs=n_jobs,
        verbose=verbose,
    )
    self._init_conditional(feature_map, randomize, exact, infinite_params, seed)

conformalize

conformalize(
    X_conformalize: ArrayLike,
    y_conformalize: ArrayLike,
    predict_params: Optional[dict] = None,
) -> "ConditionalSplitConformalRegressor"

Conformalize the regressor and set up the final fitting problem for the given conformalization set.

Performs the standard split-conformal conformalization step from :meth:SplitConformalRegressor.conformalize, then builds the cvxpy problem used for the conditional procedure.

PARAMETER	DESCRIPTION
X_conformalize	Features of the conformalization set. TYPE: ArrayLike
y_conformalize	Targets of the conformalization set. TYPE: ArrayLike
predict_params	Parameters to pass to the predict method of the base regressor. TYPE: Optional[dict] DEFAULT: None

RETURNS	DESCRIPTION
Self	The conformalized ConditionalSplitConformalRegressor instance.

Source code in mapie/conditional_conformal_prediction.py

def conformalize(
    self,
    X_conformalize: ArrayLike,
    y_conformalize: ArrayLike,
    predict_params: Optional[dict] = None,
) -> "ConditionalSplitConformalRegressor":
    """
    Conformalize the regressor and set up the final fitting problem
    for the given conformalization set.

    Performs the standard split-conformal conformalization step from
    :meth:`SplitConformalRegressor.conformalize`, then builds the
    cvxpy problem used for the conditional procedure.

    Parameters
    ----------
    X_conformalize : ArrayLike
        Features of the conformalization set.

    y_conformalize : ArrayLike
        Targets of the conformalization set.

    predict_params : Optional[dict], default=None
        Parameters to pass to the ``predict`` method of the base
        regressor.

    Returns
    -------
    Self
        The conformalized ConditionalSplitConformalRegressor instance.
    """
    super().conformalize(
        X_conformalize, y_conformalize, predict_params=predict_params
    )

    self.y_calib = np.asarray(y_conformalize)
    self._conformalize_conditional(
        np.asarray(X_conformalize),
        self.conformity_scores,  # computed in super().conformalize
    )

    return self

predict_interval

predict_interval(
    X: ArrayLike,
    minimize_interval_width: bool = False,
    allow_infinite_bounds: bool = False,
) -> Tuple[NDArray, NDArray]

Predicts points (using the base regressor) and conditionally valid intervals.

If several confidence levels were provided during initialisation, several intervals will be predicted for each sample. See the return signature.

PARAMETER	DESCRIPTION
X	Features. TYPE: ArrayLike
minimize_interval_width	Not supported by the conditional procedure; provided for API compatibility with :class:~mapie.regression.SplitConformalRegressor. TYPE: bool DEFAULT: False
allow_infinite_bounds	Accepted for API compatibility with :class:~mapie.regression.SplitConformalRegressor. Note that the conditional procedure may return infinite bounds regardless of this flag (e.g. when no finite cutoff can be found). TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
Tuple[NDArray, NDArray]	Two arrays: Prediction points, of shape (n_samples,) Prediction intervals, of shape (n_samples, 2, n_confidence_levels)

Source code in mapie/conditional_conformal_prediction.py

def predict_interval(
    self,
    X: ArrayLike,
    minimize_interval_width: bool = False,
    allow_infinite_bounds: bool = False,
) -> Tuple[NDArray, NDArray]:
    """
    Predicts points (using the base regressor) and conditionally valid
    intervals.

    If several confidence levels were provided during initialisation,
    several intervals will be predicted for each sample. See the return
    signature.

    Parameters
    ----------
    X : ArrayLike
        Features.

    minimize_interval_width : bool, default=False
        Not supported by the conditional procedure; provided for API
        compatibility with
        :class:`~mapie.regression.SplitConformalRegressor`.

    allow_infinite_bounds : bool, default=False
        Accepted for API compatibility with
        :class:`~mapie.regression.SplitConformalRegressor`. Note that the
        conditional procedure may return infinite bounds regardless of this
        flag (e.g. when no finite cutoff can be found).

    Returns
    -------
    Tuple[NDArray, NDArray]
        Two arrays:

        - Prediction points, of shape `(n_samples,)`
        - Prediction intervals, of shape
          `(n_samples, 2, n_confidence_levels)`
    """
    _raise_error_if_previous_method_not_called(
        "predict_interval",
        "conformalize",
        self._is_conformalized,
    )
    if minimize_interval_width:
        raise NotImplementedError(
            "minimize_interval_width is not supported by "
            "ConditionalSplitConformalRegressor."
        )

    X = np.asarray(X)
    y_pred = self.predict(X)

    score = self._conformity_score
    alphas = list(self._alphas)
    n_samples = len(X)
    intervals = np.empty((n_samples, 2, len(alphas)))

    for j, alpha in enumerate(alphas):
        for i in range(n_samples):
            x_row = X[i].reshape(1, -1)
            if score.sym:
                # Symmetric scores are absolute, so a single cutoff at the
                # 1 - alpha quantile inverts to a two-sided interval.
                cutoff = self._predict_conditional_cutoff(1 - alpha, x_row)
                low = score.get_estimation_distribution(y_pred[i], -cutoff, X=x_row)
                up = score.get_estimation_distribution(y_pred[i], cutoff, X=x_row)
            else:
                # Signed scores need one cutoff per side.
                cutoff_low = self._predict_conditional_cutoff(alpha / 2, x_row)
                cutoff_up = self._predict_conditional_cutoff(1 - alpha / 2, x_row)
                low = score.get_estimation_distribution(
                    y_pred[i], cutoff_low, X=x_row
                )
                up = score.get_estimation_distribution(
                    y_pred[i], cutoff_up, X=x_row
                )
            intervals[i, 0, j] = float(low)
            intervals[i, 1, j] = float(up)

    return y_pred, intervals

---

mapie.conditional_conformal_prediction.ConditionalSplitConformalClassifier

ConditionalSplitConformalClassifier(
    feature_map: Callable,
    estimator: ClassifierMixin = LogisticRegression(),
    confidence_level: Union[float, Iterable[float]] = 0.9,
    conformity_score: Union[
        str, BaseClassificationScore
    ] = "lac",
    prefit: bool = True,
    n_jobs: Optional[int] = None,
    verbose: int = 0,
    randomize: bool = False,
    exact: bool = True,
    infinite_params: Optional[dict] = None,
    seed: int = 0,
)

Bases: _ConditionalConformalMixin, SplitConformalClassifier

Split conformal classifier with conditional validity guarantees.

In addition to the parameters of :class:~mapie.classification.SplitConformalClassifier, this class accepts settings for the conditional conformal procedure.

PARAMETER	DESCRIPTION
feature_map	Function mapping covariates to a finite basis used for exact conditional guarantees. TYPE: Callable
estimator	Base classifier used to predict labels. TYPE: ClassifierMixin DEFAULT: LogisticRegression()
confidence_level	Desired coverage probability of the prediction sets. TYPE: float or iterable of float DEFAULT: 0.9
conformity_score	Method used to compute conformity scores. The conditional procedure inverts a real-valued score cutoff into a prediction set, so only scores whose prediction sets are obtained by thresholding real-valued scores are supported ("lac", "aps"); "top_k" and "raps" are not. TYPE: str or BaseClassificationScore DEFAULT: "lac"
prefit	Whether the base classifier is already fitted. TYPE: bool DEFAULT: True
n_jobs	Number of parallel jobs when applicable. TYPE: int DEFAULT: None
verbose	Verbosity level. TYPE: int DEFAULT: 0
randomize	Whether to use randomization to make coverage exact rather than conservative. If False, predictions are deterministic and coverage may be slightly above the target level. If True, predictions use auxiliary randomness to match the target coverage level more exactly. TYPE: bool DEFAULT: False
exact	Compute the conditional score cutoff exactly rather than by binary search. TYPE: bool DEFAULT: True
infinite_params	Parameters for the RKHS component of the fit. Valid keys are kernel, gamma, and lambda. Currently the infinite-dimensional (RKHS) component is not implemented: requesting a kernel raises NotImplementedError at construction time. The supporting code is retained for future work. TYPE: dict DEFAULT: None

Source code in mapie/conditional_conformal_prediction.py

def __init__(
    self,
    feature_map: Callable,
    estimator: ClassifierMixin = LogisticRegression(),
    confidence_level: Union[float, Iterable[float]] = 0.9,
    conformity_score: Union[str, BaseClassificationScore] = "lac",
    prefit: bool = True,
    n_jobs: Optional[int] = None,
    verbose: int = 0,
    randomize: bool = False,
    exact: bool = True,
    infinite_params: Optional[dict] = None,
    seed: int = 0,
) -> None:
    """
    Split conformal classifier with conditional validity guarantees.

    In addition to the parameters of
    :class:`~mapie.classification.SplitConformalClassifier`, this class
    accepts settings for the conditional conformal procedure.

    Parameters
    ----------
    feature_map : Callable
        Function mapping covariates to a finite basis used for exact
        conditional guarantees.

    estimator : ClassifierMixin, default=LogisticRegression()
        Base classifier used to predict labels.

    confidence_level : float or iterable of float, default=0.9
        Desired coverage probability of the prediction sets.

    conformity_score : str or BaseClassificationScore, default="lac"
        Method used to compute conformity scores. The conditional
        procedure inverts a real-valued score cutoff into a prediction
        set, so only scores whose prediction sets are obtained by
        thresholding real-valued scores are supported ("lac", "aps");
        "top_k" and "raps" are not.

    prefit : bool, default=True
        Whether the base classifier is already fitted.

    n_jobs : int, optional
        Number of parallel jobs when applicable.

    verbose : int, default=0
        Verbosity level.

    randomize : bool, default=False
        Whether to use randomization to make coverage exact rather than
        conservative.

        If False, predictions are deterministic and coverage may be slightly above
        the target level. If True, predictions use auxiliary randomness to match the
        target coverage level more exactly.

    exact : bool, default=True
        Compute the conditional score cutoff exactly rather than by binary
        search.

    infinite_params : dict, optional
        Parameters for the RKHS component of the fit. Valid keys are
        ``kernel``, ``gamma``, and ``lambda``. Currently the
        infinite-dimensional (RKHS) component is not implemented:
        requesting a ``kernel`` raises ``NotImplementedError`` at
        construction time. The supporting code is retained for future work.
    """
    super().__init__(
        estimator=estimator,
        confidence_level=confidence_level,
        conformity_score=conformity_score,
        prefit=prefit,
        n_jobs=n_jobs,
        verbose=verbose,
    )
    if isinstance(
        self._conformity_score, (RAPSConformityScore, TopKConformityScore)
    ):
        raise ValueError(
            "ConditionalSplitConformalClassifier requires a conformity "
            "score whose prediction sets are obtained by thresholding "
            'real-valued scores, e.g. "lac" or "aps".'
        )
    self._init_conditional(feature_map, randomize, exact, infinite_params, seed)

conformalize

conformalize(
    X_conformalize: ArrayLike,
    y_conformalize: ArrayLike,
    predict_params: Optional[dict] = None,
) -> "ConditionalSplitConformalClassifier"

Conformalize the classifier and set up the final fitting problem for the given conformalization set.

Performs the standard split-conformal conformalization step from :meth:SplitConformalClassifier.conformalize, then builds the cvxpy problem used for the conditional procedure.

PARAMETER	DESCRIPTION
X_conformalize	Features of the conformalization set. TYPE: ArrayLike
y_conformalize	Targets of the conformalization set. TYPE: ArrayLike
predict_params	Parameters to pass to the predict and predict_proba methods of the base classifier. TYPE: Optional[dict] DEFAULT: None

RETURNS	DESCRIPTION
Self	The conformalized ConditionalSplitConformalClassifier instance.

Source code in mapie/conditional_conformal_prediction.py

def conformalize(
    self,
    X_conformalize: ArrayLike,
    y_conformalize: ArrayLike,
    predict_params: Optional[dict] = None,
) -> "ConditionalSplitConformalClassifier":
    """
    Conformalize the classifier and set up the final fitting problem
    for the given conformalization set.

    Performs the standard split-conformal conformalization step from
    :meth:`SplitConformalClassifier.conformalize`, then builds the
    cvxpy problem used for the conditional procedure.

    Parameters
    ----------
    X_conformalize : ArrayLike
        Features of the conformalization set.

    y_conformalize : ArrayLike
        Targets of the conformalization set.

    predict_params : Optional[dict], default=None
        Parameters to pass to the ``predict`` and ``predict_proba``
        methods of the base classifier.

    Returns
    -------
    Self
        The conformalized ConditionalSplitConformalClassifier instance.
    """
    super().conformalize(
        X_conformalize, y_conformalize, predict_params=predict_params
    )

    self._conformalize_conditional(
        np.asarray(X_conformalize),
        self.conformity_scores,  # computed in super().conformalize
    )

    return self

predict_set

predict_set(
    X: ArrayLike,
    conformity_score_params: Optional[dict] = None,
) -> Tuple[NDArray, NDArray]

For each sample in X, predicts a label (using the base classifier) and a conditionally valid set of labels.

If several confidence levels were provided during initialisation, several sets will be predicted for each sample. See the return signature.

PARAMETER	DESCRIPTION
X	Features. TYPE: ArrayLike
conformity_score_params	Parameters specific to conformity scores, used at prediction time (e.g. include_last_label for the "aps" conformity score). TYPE: Optional[dict] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[NDArray, NDArray]	Two arrays: Prediction labels, of shape (n_samples,) Prediction sets, of shape (n_samples, n_class, n_confidence_levels)

Source code in mapie/conditional_conformal_prediction.py

def predict_set(
    self,
    X: ArrayLike,
    conformity_score_params: Optional[dict] = None,
) -> Tuple[NDArray, NDArray]:
    """
    For each sample in X, predicts a label (using the base classifier)
    and a conditionally valid set of labels.

    If several confidence levels were provided during initialisation,
    several sets will be predicted for each sample. See the return
    signature.

    Parameters
    ----------
    X : ArrayLike
        Features.

    conformity_score_params : Optional[dict], default=None
        Parameters specific to conformity scores, used at prediction time
        (e.g. ``include_last_label`` for the "aps" conformity score).

    Returns
    -------
    Tuple[NDArray, NDArray]
        Two arrays:

        - Prediction labels, of shape `(n_samples,)`
        - Prediction sets, of shape
          `(n_samples, n_class, n_confidence_levels)`
    """
    _raise_error_if_previous_method_not_called(
        "predict_set",
        "conformalize",
        self._is_conformalized,
    )
    conformity_score_params_ = _prepare_params(conformity_score_params)
    include_last_label = conformity_score_params_.get("include_last_label", True)

    X = np.asarray(X)
    mapie_classifier = self._mapie_classifier
    y_pred_proba = mapie_classifier.estimator_.single_estimator_.predict_proba(
        X, **self._predict_params
    )
    y_pred_proba = check_proba_normalized(y_pred_proba, axis=1)
    y_pred = mapie_classifier.label_encoder_.inverse_transform(
        np.argmax(y_pred_proba, axis=1)
    )

    score = mapie_classifier.conformity_score_function_
    alphas = np.asarray(self._alphas)
    n_samples, n_classes = y_pred_proba.shape
    prediction_sets = np.empty((n_samples, n_classes, len(alphas)), dtype=bool)

    y_pred_proba = score.get_predictions(
        X,
        alphas,
        y_pred_proba,
        cv="prefit",
        include_last_label=include_last_label,
    )

    for i in range(n_samples):
        x_row = X[i].reshape(1, -1)
        # Classification scores are one-sided ("higher = less conforming"),
        # so a single cutoff at the 1 - alpha quantile inverts to the
        # prediction set {y : S(x, y) <= cutoff}. The inversion itself is
        # delegated to the conformity score, with the conditional cutoffs
        # in place of the marginal quantiles.
        score.quantiles_ = np.asarray(
            [self._predict_conditional_cutoff(1 - alpha, x_row) for alpha in alphas]
        )
        prediction_sets[i] = score.get_prediction_sets(
            y_pred_proba[[i]],
            self.scores_calib,
            alphas,
            cv="prefit",
            include_last_label=include_last_label,
        )[0]

    return y_pred, prediction_sets