pycsamt.ai.inversion.calibration#

Calibrated uncertainty quantification for EM neural-network inverters.

Two complementary post-hoc calibration strategies are provided, both of which convert the empirical uncertainty from EnsembleInverter or GCNInverter3D into statistically rigorous intervals:

ConformalPredictor — split conformal prediction (Vovk et al. 2005; Angelopoulos & Bates 2023). Guarantees exact marginal coverage \(P(y \in \hat{C}(x)) \ge 1-lpha\) without distributional assumptions.
PosteriorCalibrator — Gaussianization normalising flow (Kuleshov et al. 2018). Fits a monotone recalibration map \(g: [0,1] \to [0,1]\) via the Pool Adjacent Violators (PAVA) algorithm, then inverts it through the standard-normal quantile function to draw calibrated posterior samples. This constitutes a one-dimensional normalising flow that transforms the empirical residual distribution to a standard Gaussian — the asymptotic posterior.

Typical usage#

>>> from pycsamt.ai.inversion import EnsembleInverter
>>> from pycsamt.ai.inversion.calibration import ConformalPredictor, PosteriorCalibrator
>>> ens = EnsembleInverter(base_estimator=inv, n_estimators=5)
>>> ens.fit(ds_train)
>>> cp = ConformalPredictor(ens).calibrate(X_cal, y_cal)
>>> center, lo, hi = cp.predict_intervals(X_test, alpha=0.10)
>>> pc = PosteriorCalibrator().fit(y_cal, y_pred_cal, sigma_cal)
>>> samples = pc.predict_posterior(y_pred_test, sigma_test)

References

Vovk V., Gammerman A. & Shafer G. (2005) Algorithmic Learning in a Random World. Springer.

Angelopoulos A. N. & Bates S. (2023) Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16, 494–591.

Kuleshov V., Fenner N. & Ermon S. (2018) Accurate uncertainties for deep learning using calibrated regression. Proceedings of ICML, PMLR 80, 2796–2804.

Classes

`ConformalPredictor`(base_predictor[, alpha, eps])	Split conformal prediction wrapper for calibrated regression intervals.
`PosteriorCalibrator`([n_bins])	Gaussianization normalising flow for calibrated posterior sampling.

class pycsamt.ai.inversion.calibration.ConformalPredictor(base_predictor, alpha=0.1, eps=1e-10)[source]#

Bases: object

Split conformal prediction wrapper for calibrated regression intervals.

Given a fitted base predictor \(f(\mathbf{x})\) that also exposes per-output uncertainty estimates \(\boldsymbol{\sigma}(\mathbf{x})\) (e.g. EnsembleInverter), this class fits normalised nonconformity scores on a held-out calibration set and uses them to produce prediction bands with provable marginal coverage.

Normalised nonconformity score (per sample, over all output dims):

\[s_i = \max_j \frac{|y_{ij} - f(\mathbf{x}_i)_j|} {\sigma_{ij} + \varepsilon}\]

Calibration quantile (finite-sample correction, Vovk et al. 2005):

\[\hat{q} = \text{Quantile}_{1-\alpha + \frac{1}{n_{\rm cal}+1}} (s_1, \ldots, s_{n_{\rm cal}})\]

Prediction interval for a new point \(\mathbf{x}\):

\[\hat{C}_j(\mathbf{x}) = \bigl[f(\mathbf{x})_j - \hat{q}\,\sigma_j(\mathbf{x}),\; f(\mathbf{x})_j + \hat{q}\,\sigma_j(\mathbf{x})\bigr]\]

Marginal coverage guarantee (Vovk et al. 2005):

\[P\!\bigl(y_j \in \hat{C}_j(\mathbf{x})\bigr) \;\ge\; 1 - \alpha \quad \forall j\]

Parameters:

base_predictor (any) – A fitted object exposing predict_with_uncertainty(X) -> (mean, std) where both outputs have shape (n_samples, n_params).
alpha (float) – Default coverage level 1-alpha. Can be overridden per-call.
eps (float) – Floor added to \(\boldsymbol{\sigma}\) to prevent division by zero.

Examples

>>> cp = ConformalPredictor(ens_inverter)
>>> cp.calibrate(X_cal, y_cal)
>>> center, lo, hi = cp.predict_intervals(X_new)

calibrate(X_cal, y_cal, alpha=None)[source]#

Fit nonconformity scores on a held-out calibration set.

Parameters:

X_cal (ndarray, shape (n_cal, n_features))
y_cal (ndarray, shape (n_cal, n_params))
alpha (float or None) – Coverage level for this calibration run; defaults to self.alpha.

Return type:

self

predict_intervals(X, alpha=None)[source]#

Return calibrated prediction intervals.

Parameters:

X (ndarray, shape (n_samples, n_features))
alpha (float or None) – Coverage level; defaults to the value used in calibrate().

Returns:

center (ndarray, shape (n_samples, n_params))
lower (ndarray, shape (n_samples, n_params))
upper (ndarray, shape (n_samples, n_params)) – The interval \([\text{center} - \hat{q}\,\sigma,\; \text{center} + \hat{q}\,\sigma]\) satisfies the marginal coverage guarantee.

Return type:

tuple[ndarray, ndarray, ndarray]

coverage(X_test, y_test, alpha=None)[source]#

Empirical marginal coverage on a test set.

A well-calibrated predictor returns a value close to 1 - alpha.

Parameters:

X_test (ndarray)
y_test (ndarray, shape (n_test, n_params))
alpha (float or None)

Returns:

coverage

Return type:

float ∈ [0, 1]

coverage_diagnostics(X_test, y_test, alphas=None)[source]#

Coverage table across a range of significance levels.

Useful for a reliability diagram: plot actual_coverage vs nominal_coverage = 1 - alpha.

Parameters:

X_test (ndarray)
y_test (ndarray)
alphas (sequence of float or None) – Significance levels to evaluate. Defaults to 25 values uniformly spaced in [0.02, 0.50].

Returns:

diag

Return type:

dict {alpha: actual_coverage}

class pycsamt.ai.inversion.calibration.PosteriorCalibrator(n_bins=100)[source]#

Bases: object

Gaussianization normalising flow for calibrated posterior sampling.

Implements the regression calibration scheme of Kuleshov et al. (2018): a monotone recalibration function \(g: [0,1] \to [0,1]\) is learned from calibration-set residuals via isotonic regression (PAVA), then composed with the standard-normal quantile function to produce draws from a calibrated posterior.

Step 1 — fit the recalibration map on \((y_{\rm cal}, \hat{y}_{\rm cal}, \sigma_{\rm cal})\):

\[p_i = \Phi\!\left(\frac{y_i - \hat{y}_i}{\sigma_i}\right), \qquad g = \text{PAVA}\!\left(p_{(i)},\, \frac{i}{n_{\rm cal}}\right)\]

where \(p_{(i)}\) are the sorted predicted CDF values and \(i/n_{\rm cal}\) are the empirical CDF targets.

Step 2 — calibrated posterior samples:

\[u \sim \mathcal{U}(0,1),\quad p^* = g^{-1}(u),\quad \theta^* = \hat{y} + \sigma \cdot \Phi^{-1}(p^*)\]

The map \(g^{-1}\) (inverse PAVA) is computed via linear interpolation. Because the composition \(\Phi^{-1} \circ g^{-1} \circ \Phi\) is a monotone bijection \(\mathbb{R} \to \mathbb{R}\), this constitutes a one-dimensional normalising flow that transforms the empirical residual distribution into a standard Gaussian.

Parameters:: n_bins (int) – Number of equally spaced CDF bins used as knots for the interpolated inverse map. Higher values give a smoother flow at the cost of slight over-smoothing in data-sparse tails.

Examples

>>> pc = PosteriorCalibrator()
>>> pc.fit(y_cal, y_pred_cal, sigma_cal)
>>> samples = pc.predict_posterior(y_pred_test, sigma_test)
>>> cal_std = pc.calibrated_std(sigma_raw_test)

References

Kuleshov V., Fenner N. & Ermon S. (2018) Accurate uncertainties for deep learning using calibrated regression. ICML, PMLR 80, 2796–2804.

fit(y_cal, y_pred_cal, sigma_cal)[source]#

Fit the Gaussianization recalibration map.

Parameters:

y_cal (ndarray, shape (n_cal, n_params)) – True parameter vectors on the calibration set.
y_pred_cal (ndarray, shape (n_cal, n_params)) – Point predictions (e.g. ensemble mean).
sigma_cal (ndarray, shape (n_cal, n_params)) – Predicted standard deviations (e.g. ensemble std).

Return type:

self

predict_posterior(y_pred, sigma_pred, n_samples=500, rng=None)[source]#

Draw samples from the calibrated posterior via inverse-CDF sampling.

Parameters:

y_pred (ndarray, shape (n_samples_pred, n_params)) – Point predictions (ensemble mean).
sigma_pred (ndarray, shape (n_samples_pred, n_params)) – Raw (uncalibrated) uncertainty estimates.
n_samples (int) – Number of posterior draws per input point.
rng (numpy.random.Generator or None) – Random number generator for reproducibility.

Returns:

draws – Calibrated posterior samples. draws.mean(axis=0) ≈ y_pred and draws.std(axis=0) ≈ calibrated_sigma.

Return type:

ndarray, shape (n_samples, n_samples_pred, n_params)

calibrated_std(sigma_raw)[source]#

Apply the global variance recalibration to raw sigma estimates.

Parameters:: sigma_raw (ndarray)
Returns:: sigma_cal – Recalibrated standard deviation such that coverage is approximately nominal on the calibration distribution.
Return type:: ndarray

calibration_error(y_test, y_pred_test, sigma_test)[source]#

Mean absolute calibration error (MACE, Kuleshov et al. 2018).

For a perfectly calibrated predictor, the fraction of test samples falling below the \(p\)-quantile of the predicted distribution should equal \(p\). MACE measures the average deviation:

\[\text{MACE} = \frac{1}{B} \sum_{b=1}^{B} |\hat{F}(q_b) - q_b|\]

where \(B\) is the number of CDF evaluation points.

Parameters:

y_test (ndarray, shape (n_test, n_params))
y_pred_test (ndarray, shape (n_test, n_params))
sigma_test (ndarray, shape (n_test, n_params))

Returns:

mace – 0 = perfectly calibrated, 1 = maximally mis-calibrated.

Return type:

float ∈ [0, 1]