2.25.3.7. pycsamt.ai.validation#

Synthetic recovery, response residual, uncertainty calibration, and out-of-distribution validation tools.

Scientific validation gates for AI-assisted EM inversion.

This package contains synthetic recovery metrics, complex-response residual diagnostics, uncertainty calibration, and out-of-distribution checks. Plotting remains in pycsamt.ai.plot; this package owns the numerical evidence behind plots.

Status#

All four submodules are implemented: masked RMSE/MAE/R2, structural similarity (SSIM), and depth-profile recovery diagnostics for known-truth synthetic grids (M0 baseline metrics); complex-impedance residual reports broken down by station/frequency/component (Inversion/Field rows of the validation matrix); Gaussian reliability curves with coverage, calibration penalty, and sharpness (Uncertainty row); and Mahalanobis/k-NN out-of-distribution scoring against a reference realization set (OOD sensitivity).

class pycsamt.ai.validation.RecoveryReport(rmse, mae, r2, ssim, depth_rmse, depth_mae, n_valid, shape)#

Bases: object

Immutable synthetic-recovery diagnostics for one grid pair.

Parameters:

  • rmse (float) – Global masked root-mean-square and mean-absolute error.

  • mae (float) – Global masked root-mean-square and mean-absolute error.

  • r2 (float) – Coefficient of determination. nan when the true values are numerically constant, making R² undefined.

  • ssim (float or None) – Structural similarity index (Wang et al., 2004), or None when it was not requested or not computable, e.g. the grid is partially masked or smaller than the requested window.

  • depth_rmse (ndarray) – Per-layer RMSE/MAE along the requested depth axis.

  • depth_mae (ndarray) – Per-layer RMSE/MAE along the requested depth axis.

  • n_valid (int) – Number of cells included after masking.

  • shape (tuple of int) – Shape of the compared grids.

Examples

>>> import numpy as np
>>> pred = np.array([[1.0, 2.0], [3.0, 4.0]])
>>> true = np.array([[1.0, 2.0], [3.0, 6.0]])
>>> report = recovery_report(pred, true, compute_ssim=False)
>>> report.n_valid, report.shape
(4, (2, 2))
rmse: float#
mae: float#
r2: float#
ssim: float | None#
depth_rmse: ndarray#
depth_mae: ndarray#
n_valid: int#
shape: tuple[int, ...]#
pycsamt.ai.validation.recovery_report(y_pred, y_true, *, valid=None, depth_axis=0, compute_ssim=True, ssim_window=7)#

Summarize synthetic-recovery quality for one grid pair.

Parameters:

  • y_pred (array-like) – Predicted and true model values sharing one 2-D or 3-D grid shape.

  • y_true (array-like) – Predicted and true model values sharing one 2-D or 3-D grid shape.

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking of both inputs.

  • depth_axis (int, default=0) – Grid axis passed to depth_profile_rmse() and depth_profile_mae().

  • compute_ssim (bool, default=True) – Attempt structural_similarity(). Skipped (ssim is None) when the grid is partially masked or smaller than ssim_window, since SSIM has no defined masking rule.

  • ssim_window (int, default=7) – Window forwarded to structural_similarity().

Returns:

Combined recovery diagnostics.

Return type:

RecoveryReport

Raises:

ValueError – If shapes mismatch, inputs are empty, or no cell is valid.

Examples

>>> import numpy as np
>>> pred = np.array([[1.0, 2.0], [3.0, 4.0]])
>>> true = np.array([[1.0, 2.0], [3.0, 6.0]])
>>> report = recovery_report(pred, true, compute_ssim=False)
>>> round(report.rmse, 6), round(report.mae, 6)
(1.0, 0.5)
>>> round(report.r2, 6)
0.714286
pycsamt.ai.validation.structural_similarity(y_pred, y_true, *, window=7, data_range=None)#

Compute the mean structural similarity index (SSIM).

Uses the windowed luminance/contrast/structure formulation of Wang et al. (2004) with a uniform (box) window, evaluated on the interior of the grid to avoid boundary-filter artifacts.

Parameters:

  • y_pred (array-like) – Fully finite 2-D or 3-D grids sharing one shape. SSIM has no defined masking rule, so both must be complete.

  • y_true (array-like) – Fully finite 2-D or 3-D grids sharing one shape. SSIM has no defined masking rule, so both must be complete.

  • window (int, default=7) – Positive odd window size, no larger than the smallest grid axis.

  • data_range (float or None, optional) – Dynamic range of the compared values. Defaults to the range spanned by the combined y_pred/y_true values.

Returns:

Mean SSIM over the interior window positions, at most 1.0 for identical grids.

Return type:

float

Examples

>>> import numpy as np
>>> grid = np.arange(64, dtype=float).reshape(8, 8)
>>> structural_similarity(grid, grid, window=3)
1.0
pycsamt.ai.validation.depth_profile_rmse(y_pred, y_true, *, axis=0, valid=None)#

Return per-layer masked RMSE along one grid axis.

Parameters:

  • y_pred (array-like) – Predicted and true model values sharing one 2-D or 3-D grid shape.

  • y_true (array-like) – Predicted and true model values sharing one 2-D or 3-D grid shape.

  • axis (int, default=0) – Grid axis to break down by, typically depth (z).

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking of both inputs.

Returns:

RMSE for each layer; nan for a layer with no valid cells.

Return type:

ndarray, shape (y_pred.shape[axis],)

Examples

>>> import numpy as np
>>> pred = np.array([[1.0, 2.0], [3.0, 4.0]])
>>> true = np.array([[1.0, 2.0], [3.0, 6.0]])
>>> depth_profile_rmse(pred, true).tolist()
[0.0, 1.4142135623730951]
pycsamt.ai.validation.depth_profile_mae(y_pred, y_true, *, axis=0, valid=None)#

Return per-layer masked MAE along one grid axis.

Parameters:

  • y_pred (array-like) – Predicted and true model values sharing one 2-D or 3-D grid shape.

  • y_true (array-like) – Predicted and true model values sharing one 2-D or 3-D grid shape.

  • axis (int, default=0) – Grid axis to break down by, typically depth (z).

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking of both inputs.

Returns:

MAE for each layer; nan for a layer with no valid cells.

Return type:

ndarray, shape (y_pred.shape[axis],)

Examples

>>> import numpy as np
>>> pred = np.array([[1.0, 2.0], [3.0, 4.0]])
>>> true = np.array([[1.0, 2.0], [3.0, 6.0]])
>>> depth_profile_mae(pred, true).tolist()
[0.0, 1.0]
class pycsamt.ai.validation.ResponseResidualReport(overall, by_station, by_frequency, by_component, station_names, frequencies_hz, components, shape)#

Bases: object

Immutable per-axis complex-impedance residual diagnostics.

Parameters:

  • overall (ResponseLossResult) – Reduced L_response penalty over every included cell.

  • by_station (ndarray) – Masked mean per-cell penalty aggregated over the other two axes, in the same units as overall.value (e.g. mean squared residual for kind="l2", not RMS). nan where an axis position has no valid cell.

  • by_frequency (ndarray) – Masked mean per-cell penalty aggregated over the other two axes, in the same units as overall.value (e.g. mean squared residual for kind="l2", not RMS). nan where an axis position has no valid cell.

  • by_component (ndarray) – Masked mean per-cell penalty aggregated over the other two axes, in the same units as overall.value (e.g. mean squared residual for kind="l2", not RMS). nan where an axis position has no valid cell.

  • station_names (tuple of str or None) – Optional station labels, length shape[0].

  • frequencies_hz (ndarray or None) – Optional frequency labels, length shape[1].

  • components (tuple of str or None) – Optional component labels, length shape[2].

  • shape (tuple of int) – (station, frequency, component) shape of the compared arrays.

Examples

>>> import numpy as np
>>> pred = np.array([[[1 + 0j], [2 + 0j]], [[0j], [0j]]])
>>> obs = np.array([[[1 + 0j], [0j]], [[0j], [3 + 0j]]])
>>> report = response_residual_report(pred, obs)
>>> report.shape
(2, 2, 1)
>>> report.by_station.tolist()
[2.0, 4.5]
overall: ResponseLossResult#
by_station: ndarray#
by_frequency: ndarray#
by_component: ndarray#
station_names: tuple[str, ...] | None#
frequencies_hz: ndarray | None#
components: tuple[str, ...] | None#
shape: tuple[int, int, int]#
pycsamt.ai.validation.response_residual_report(predicted, observed, *, errors=None, valid=None, kind='l2', station_names=None, frequencies_hz=None, components=None)#

Break complex-impedance residuals down by station/frequency/ component.

Parameters:

  • predicted (array-like of complex) – Forward-simulated impedance, canonical shape (station, frequency, component).

  • observed (array-like of complex) – Observed impedance with the same shape.

  • errors (array-like or None, optional) – Positive absolute standard errors used to normalize residuals, as in response_residual_loss().

  • valid (array-like of bool or None, optional) – Explicit observation mask.

  • kind ({"l1", "l2"}, default="l2") – Elementwise penalty applied to each residual magnitude.

  • station_names (sequence of str or None, optional) – Optional axis labels attached to the returned report.

  • components (sequence of str or None, optional) – Optional axis labels attached to the returned report.

  • frequencies_hz (array-like or None, optional) – Optional frequency labels attached to the returned report.

Returns:

Combined per-axis residual diagnostics.

Return type:

ResponseResidualReport

Examples

>>> import numpy as np
>>> pred = np.array([[[1 + 0j], [2 + 0j]], [[0j], [0j]]])
>>> obs = np.array([[[1 + 0j], [0j]], [[0j], [3 + 0j]]])
>>> report = response_residual_report(pred, obs)
>>> report.by_frequency.tolist()
[0.0, 6.5]
pycsamt.ai.validation.response_residual_report_from_contracts(forward, observed, *, kind='l2', use_errors=True)#

Build a ResponseResidualReport directly from a ForwardResult/SurveyData pair.

Requires exact station, component, and frequency alignment, as in response_loss_from_contracts().

Parameters:

  • forward (ForwardResult) – Predicted impedance from a Maxwell backend adapter.

  • observed (SurveyData) – Observed survey impedance to compare against.

  • kind ({"l1", "l2"}, default="l2") – Elementwise penalty applied to each residual magnitude.

  • use_errors (bool, default=True) – Normalize residuals by observed.impedance_error when it is available.

Returns:

Combined per-axis residual diagnostics, labeled with the survey’s station names, frequencies, and components.

Return type:

ResponseResidualReport

class pycsamt.ai.validation.ReliabilityCurve(levels, coverage, calibration, sharpness, n_valid, shape)#

Bases: object

Immutable calibration report for Gaussian predictive intervals.

Parameters:

  • levels (ndarray) – Nominal confidence levels in (0, 1).

  • coverage (ndarray) – Empirical coverage at each level, same shape as levels.

  • calibration (UncertaintyLossResult) – Reduced deviation between coverage and levels, from calibration_loss().

  • sharpness (float) – Mean predictive standard deviation over included cells. Lower is sharper (more confident); meaningful only alongside good calibration.

  • n_valid (int) – Number of cells included after masking.

  • shape (tuple of int) – Shape of the compared y_true array.

Examples

>>> import numpy as np
>>> true = np.array([0.0, 0.0, 0.0, 0.0, 10.0])
>>> mean = np.zeros(5)
>>> std = np.ones(5)
>>> curve = reliability_curve(true, mean, std)
>>> curve.n_valid
5
levels: ndarray#
coverage: ndarray#
calibration: UncertaintyLossResult#
sharpness: float#
n_valid: int#
shape: tuple[int, ...]#
pycsamt.ai.validation.reliability_curve(y_true, y_pred_mean, y_pred_std, *, levels=None, valid=None, kind='l2', reduction='mean')#

Build a full calibration report for Gaussian predictive intervals.

Parameters:

  • y_true (array-like) – True values.

  • y_pred_mean (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • y_pred_std (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • levels (array-like or None, optional) – Nominal confidence levels in (0, 1). Defaults to (0.5, 0.8, 0.9, 0.95, 0.99).

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking of all three inputs and with y_pred_std > 0.

  • kind ({"l1", "l2"}, default="l2") – Elementwise penalty forwarded to calibration_loss().

  • reduction ({"mean", "sum"}, default="mean") – Reduction forwarded to calibration_loss().

Returns:

Combined coverage, calibration penalty, and sharpness.

Return type:

ReliabilityCurve

Examples

>>> import numpy as np
>>> true = np.array([0.0, 0.0, 0.0, 0.0, 10.0])
>>> mean = np.zeros(5)
>>> std = np.ones(5)
>>> curve = reliability_curve(true, mean, std, levels=[0.5])
>>> curve.coverage.tolist(), curve.sharpness
([0.8], 1.0)
pycsamt.ai.validation.empirical_coverage(y_true, y_pred_mean, y_pred_std, *, levels=None, valid=None)#

Compute empirical coverage of Gaussian predictive intervals.

Parameters:

  • y_true (array-like) – True values.

  • y_pred_mean (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • y_pred_std (array-like) – Predicted mean and positive standard deviation, same shape as y_true.

  • levels (array-like or None, optional) – Nominal confidence levels in (0, 1). Defaults to (0.5, 0.8, 0.9, 0.95, 0.99).

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking of all three inputs and with y_pred_std > 0.

Returns:

  • levels (ndarray) – The validated nominal levels.

  • coverage (ndarray, same shape as levels) – Fraction of included cells whose true value falls inside the mean +/- z(level) * std interval.

  • n_valid (int) – Number of cells included after masking.

Return type:

tuple[ndarray, ndarray, int]

Examples

>>> import numpy as np
>>> true = np.array([0.0, 0.0, 0.0, 0.0, 10.0])
>>> mean = np.zeros(5)
>>> std = np.ones(5)
>>> levels, coverage, n_valid = empirical_coverage(
...     true, mean, std, levels=[0.5]
... )
>>> coverage.tolist()
[0.8]
pycsamt.ai.validation.predictive_sharpness(y_pred_std, *, valid=None)#

Return the masked mean predictive standard deviation.

Sharpness summarizes how confident a model’s predictive distribution is, independent of correctness; it is only a meaningful quality signal alongside good calibration.

Parameters:

  • y_pred_std (array-like) – Predicted positive standard deviation.

  • valid (array-like of bool or None, optional) – Explicit cell mask, combined with finite-value masking and with y_pred_std > 0.

Returns:

Mean predictive standard deviation over included cells.

Return type:

float

Examples

>>> import numpy as np
>>> predictive_sharpness(np.array([1.0, 2.0, 3.0]))
2.0
class pycsamt.ai.validation.OODReport(scores, threshold, flagged, method, quantile, n_reference, n_features)#

Bases: object

Immutable out-of-distribution screening result.

Parameters:

  • scores (ndarray) – Per-sample distance from ood_score().

  • threshold (float) – Score above which a sample is flagged.

  • flagged (ndarray of bool) – Whether each sample exceeds threshold, same shape as scores.

  • method ({"mahalanobis", "knn"}) – Distance measure used to compute scores.

  • quantile (float or None) – Quantile of the reference self-scores used to derive threshold, or None when an explicit threshold was supplied instead.

  • n_reference (int) – Size of the reference set used to define support.

  • n_features (int) – Size of the reference set used to define support.

Examples

>>> import numpy as np
>>> reference = np.array(
...     [
...         [0.0, 0.0],
...         [1.0, 0.0],
...         [0.0, 1.0],
...         [-1.0, 0.0],
...         [0.0, -1.0],
...         [0.5, 0.5],
...     ]
... )
>>> x = np.array([[0.0, 0.0], [50.0, 50.0]])
>>> report = flag_out_of_distribution(
...     x, reference, method="knn", k=2, quantile=0.5
... )
>>> report.flagged.tolist()
[False, True]
scores: ndarray#
threshold: float#
flagged: ndarray#
method: str#
quantile: float | None#
n_reference: int#
n_features: int#
property fraction_flagged: float#

Return the fraction of scored samples flagged as OOD.

Examples

>>> import numpy as np
>>> report = OODReport(
...     scores=np.array([0.1, 5.0]),
...     threshold=1.0,
...     flagged=np.array([False, True]),
...     method="knn",
...     quantile=None,
...     n_reference=10,
...     n_features=2,
... )
>>> report.fraction_flagged
0.5
pycsamt.ai.validation.ood_score(x, reference, *, method='mahalanobis', k=5)#

Score how far new inputs fall from the training distribution.

Parameters:

  • x (array-like, shape (n_samples, n_features)) – Inputs to score, e.g. new survey feature vectors.

  • reference (array-like, shape (n_reference, n_features)) – Training-set feature vectors defining the support region.

  • method ({"mahalanobis", "knn"}, default="mahalanobis") – "mahalanobis" measures deviation from the reference mean/covariance and requires n_reference > n_features. "knn" measures Euclidean distance to the k-th nearest reference point and makes no distributional assumption. If x shares exact points with reference, their k-NN distance to those points is zero; use flag_out_of_distribution() for a leave-one-out self-score instead of passing reference as x here.

  • k (int, default=5) – Neighbour rank used by method="knn". Ignored otherwise.

Returns:

Higher values indicate inputs farther from the training support.

Return type:

ndarray, shape (n_samples,)

Examples

>>> import numpy as np
>>> reference = np.array(
...     [
...         [0.0, 0.0],
...         [1.0, 0.0],
...         [0.0, 1.0],
...         [-1.0, 0.0],
...         [0.0, -1.0],
...         [0.5, 0.5],
...     ]
... )
>>> x = np.array([[0.0, 0.0], [50.0, 50.0]])
>>> scores = ood_score(x, reference, method="knn", k=2)
>>> scores[0] < scores[1]
True
pycsamt.ai.validation.flag_out_of_distribution(x, reference, *, method='mahalanobis', k=5, quantile=0.99, threshold=None)#

Score inputs and flag those outside the training support.

When threshold is not supplied, it is derived as the requested quantile of the reference set’s own leave-one-out ("knn") or full-sample ("mahalanobis") self-scores, i.e. “how unusual is a typical reference point”.

Parameters:

  • x (array-like, shape (n_samples, n_features)) – Inputs to score.

  • reference (array-like, shape (n_reference, n_features)) – Training-set feature vectors defining the support region.

  • method ({"mahalanobis", "knn"}, default="mahalanobis") – Distance measure, as in ood_score().

  • k (int, default=5) – Neighbour rank used by method="knn". Ignored otherwise.

  • quantile (float, default=0.99) – Quantile in (0, 1) of the reference self-scores used to derive threshold. Ignored when threshold is given.

  • threshold (float or None, optional) – Explicit score threshold. Overrides quantile when given.

Returns:

Scores, threshold, and per-sample OOD flags.

Return type:

OODReport

Examples

>>> import numpy as np
>>> reference = np.array(
...     [
...         [0.0, 0.0],
...         [1.0, 0.0],
...         [0.0, 1.0],
...         [-1.0, 0.0],
...         [0.0, -1.0],
...         [0.5, 0.5],
...     ]
... )
>>> x = np.array([[0.0, 0.0], [50.0, 50.0]])
>>> report = flag_out_of_distribution(x, reference, k=1)
>>> report.method, report.n_reference
('mahalanobis', 6)

pycsamt.ai.validation.recovery

Synthetic-recovery diagnostics for known-truth geological grids.

pycsamt.ai.validation.residuals

Complex-response residual diagnostics for EM forward comparisons.

pycsamt.ai.validation.calibration

Uncertainty-calibration diagnostics for predictive intervals.

pycsamt.ai.validation.ood

Out-of-distribution checks against the training realization set.