pycsamt.ai.validation.calibration#

Uncertainty-calibration diagnostics for predictive intervals.

These diagnostics support the Uncertainty row of the validation matrix in the AI-inversion plan: “calibration, coverage, sharpness, OOD sensitivity”. Predictive distributions are assumed Gaussian, i.e. a cell’s predictive interval at nominal level p is mean +/- z(p) * std with z(p) the two-sided normal quantile; this matches how gaussian_nll_loss in pycsamt.ai.losses.uncertainty parameterizes aleatoric uncertainty.

All functions operate on plain NumPy arrays so the module stays importable without an optional deep-learning backend.

Functions

empirical_coverage(y_true, y_pred_mean, ...)

Compute empirical coverage of Gaussian predictive intervals.

predictive_sharpness(y_pred_std, *[, valid])

Return the masked mean predictive standard deviation.

reliability_curve(y_true, y_pred_mean, ...)

Build a full calibration report for Gaussian predictive intervals.

Classes

ReliabilityCurve(levels, coverage, ...)

Immutable calibration report for Gaussian predictive intervals.
class pycsamt.ai.validation.calibration.ReliabilityCurve(levels, coverage, calibration, sharpness, n_valid, shape)[source]

Bases: object

Immutable calibration report for Gaussian predictive intervals.

Parameters:

  • levels (ndarray) – Nominal confidence levels in (0, 1).

  • coverage (ndarray) – Empirical coverage at each level, same shape as levels.

  • calibration (UncertaintyLossResult) – Reduced deviation between coverage and levels, from calibration_loss().

  • sharpness (float) – Mean predictive standard deviation over included cells. Lower is sharper (more confident); meaningful only alongside good calibration.

  • n_valid (int) – Number of cells included after masking.

  • shape (tuple of int) – Shape of the compared y_true array.

Examples

>>> import numpy as np
>>> true = np.array([0.0, 0.0, 0.0, 0.0, 10.0])
>>> mean = np.zeros(5)
>>> std = np.ones(5)
>>> curve = reliability_curve(true, mean, std)
>>> curve.n_valid
5
levels: ndarray
coverage: ndarray
calibration: UncertaintyLossResult
sharpness: float
n_valid: int
shape: tuple[int, ...]
pycsamt.ai.validation.calibration.reliability_curve(y_true, y_pred_mean, y_pred_std, *, levels=None, valid=None, kind='l2', reduction='mean')[source]

Build a full calibration report for Gaussian predictive intervals.

Parameters:

  • y_true (array-like) – True values.

  • y_pred_mean (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • y_pred_std (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • levels (array-like or None, optional) – Nominal confidence levels in (0, 1). Defaults to (0.5, 0.8, 0.9, 0.95, 0.99).

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking of all three inputs and with y_pred_std > 0.

  • kind ({"l1", "l2"}, default="l2") – Elementwise penalty forwarded to calibration_loss().

  • reduction ({"mean", "sum"}, default="mean") – Reduction forwarded to calibration_loss().

Returns:

Combined coverage, calibration penalty, and sharpness.

Return type:

ReliabilityCurve

Examples

>>> import numpy as np
>>> true = np.array([0.0, 0.0, 0.0, 0.0, 10.0])
>>> mean = np.zeros(5)
>>> std = np.ones(5)
>>> curve = reliability_curve(true, mean, std, levels=[0.5])
>>> curve.coverage.tolist(), curve.sharpness
([0.8], 1.0)
pycsamt.ai.validation.calibration.empirical_coverage(y_true, y_pred_mean, y_pred_std, *, levels=None, valid=None)[source]

Compute empirical coverage of Gaussian predictive intervals.

Parameters:

  • y_true (array-like) – True values.

  • y_pred_mean (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • y_pred_std (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • levels (array-like or None, optional) – Nominal confidence levels in (0, 1). Defaults to (0.5, 0.8, 0.9, 0.95, 0.99).

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking of all three inputs and with y_pred_std > 0.

Returns:

  • levels (ndarray) – The validated nominal levels.

  • coverage (ndarray, same shape as levels) – Fraction of included cells whose true value falls inside the mean +/- z(level) * std interval.

  • n_valid (int) – Number of cells included after masking.

Return type:

tuple[ndarray, ndarray, int]

Examples

>>> import numpy as np
>>> true = np.array([0.0, 0.0, 0.0, 0.0, 10.0])
>>> mean = np.zeros(5)
>>> std = np.ones(5)
>>> levels, coverage, n_valid = empirical_coverage(
...     true, mean, std, levels=[0.5]
... )
>>> coverage.tolist()
[0.8]
pycsamt.ai.validation.calibration.predictive_sharpness(y_pred_std, *, valid=None)[source]

Return the masked mean predictive standard deviation.

Sharpness summarizes how confident a model’s predictive distribution is, independent of correctness; it is only a meaningful quality signal alongside good calibration.

Parameters:

  • y_pred_std (array-like) – Predicted positive standard deviation.

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking and with y_pred_std > 0.

Returns:

Mean predictive standard deviation over included cells.

Return type:

float

Examples

>>> import numpy as np
>>> predictive_sharpness(np.array([1.0, 2.0, 3.0]))
2.0