pycsamt.ai.validation.ood#

Out-of-distribution checks against the training realization set.

These checks support the “OOD sensitivity” column of the Uncertainty row of the validation matrix, and the AI-inversion plan’s requirement that predictions outside training support are rejected or flagged rather than returned as confident maps. Inputs are feature vectors, e.g. survey/realization descriptors, shaped (n_samples, n_features); a common source is per-realization summary statistics of the geological priors in pycsamt.ai.geology.

All functions operate on plain NumPy arrays so the module stays importable without an optional deep-learning backend.

Functions

flag_out_of_distribution(x, reference, *[, ...])

Score inputs and flag those outside the training support.

ood_score(x, reference, *[, method, k])

Score how far new inputs fall from the training distribution.

Classes

OODReport(scores, threshold, flagged, ...)

Immutable out-of-distribution screening result.
class pycsamt.ai.validation.ood.OODReport(scores, threshold, flagged, method, quantile, n_reference, n_features)[source]

Bases: object

Immutable out-of-distribution screening result.

Parameters:

  • scores (ndarray) – Per-sample distance from ood_score().

  • threshold (float) – Score above which a sample is flagged.

  • flagged (ndarray of bool) – Whether each sample exceeds threshold, same shape as scores.

  • method ({"mahalanobis", "knn"}) – Distance measure used to compute scores.

  • quantile (float or None) – Quantile of the reference self-scores used to derive threshold, or None when an explicit threshold was supplied instead.

  • n_reference (int) – Size of the reference set used to define support.

  • n_features (int) – Size of the reference set used to define support.

Examples

>>> import numpy as np
>>> reference = np.array(
...     [
...         [0.0, 0.0],
...         [1.0, 0.0],
...         [0.0, 1.0],
...         [-1.0, 0.0],
...         [0.0, -1.0],
...         [0.5, 0.5],
...     ]
... )
>>> x = np.array([[0.0, 0.0], [50.0, 50.0]])
>>> report = flag_out_of_distribution(
...     x, reference, method="knn", k=2, quantile=0.5
... )
>>> report.flagged.tolist()
[False, True]
scores: ndarray
threshold: float
flagged: ndarray
method: str
quantile: float | None
n_reference: int
n_features: int
property fraction_flagged: float[source]

Return the fraction of scored samples flagged as OOD.

Examples

>>> import numpy as np
>>> report = OODReport(
...     scores=np.array([0.1, 5.0]),
...     threshold=1.0,
...     flagged=np.array([False, True]),
...     method="knn",
...     quantile=None,
...     n_reference=10,
...     n_features=2,
... )
>>> report.fraction_flagged
0.5
pycsamt.ai.validation.ood.ood_score(x, reference, *, method='mahalanobis', k=5)[source]

Score how far new inputs fall from the training distribution.

Parameters:

  • x (array-like, shape (n_samples, n_features)) – Inputs to score, e.g. new survey feature vectors.

  • reference (array-like, shape (n_reference, n_features)) – Training-set feature vectors defining the support region.

  • method ({"mahalanobis", "knn"}, default="mahalanobis") – "mahalanobis" measures deviation from the reference mean/covariance and requires n_reference > n_features. "knn" measures Euclidean distance to the k-th nearest reference point and makes no distributional assumption. If x shares exact points with reference, their k-NN distance to those points is zero; use flag_out_of_distribution() for a leave-one-out self-score instead of passing reference as x here.

  • k (int, default=5) – Neighbour rank used by method="knn". Ignored otherwise.

Returns:

Higher values indicate inputs farther from the training support.

Return type:

ndarray, shape (n_samples,)

Examples

>>> import numpy as np
>>> reference = np.array(
...     [
...         [0.0, 0.0],
...         [1.0, 0.0],
...         [0.0, 1.0],
...         [-1.0, 0.0],
...         [0.0, -1.0],
...         [0.5, 0.5],
...     ]
... )
>>> x = np.array([[0.0, 0.0], [50.0, 50.0]])
>>> scores = ood_score(x, reference, method="knn", k=2)
>>> scores[0] < scores[1]
True
pycsamt.ai.validation.ood.flag_out_of_distribution(x, reference, *, method='mahalanobis', k=5, quantile=0.99, threshold=None)[source]

Score inputs and flag those outside the training support.

When threshold is not supplied, it is derived as the requested quantile of the reference set’s own leave-one-out ("knn") or full-sample ("mahalanobis") self-scores, i.e. “how unusual is a typical reference point”.

Parameters:

  • x (array-like, shape (n_samples, n_features)) – Inputs to score.

  • reference (array-like, shape (n_reference, n_features)) – Training-set feature vectors defining the support region.

  • method ({"mahalanobis", "knn"}, default="mahalanobis") – Distance measure, as in ood_score().

  • k (int, default=5) – Neighbour rank used by method="knn". Ignored otherwise.

  • quantile (float, default=0.99) – Quantile in (0, 1) of the reference self-scores used to derive threshold. Ignored when threshold is given.

  • threshold (float or None, optional) – Explicit score threshold. Overrides quantile when given.

Returns:

Scores, threshold, and per-sample OOD flags.

Return type:

OODReport

Examples

>>> import numpy as np
>>> reference = np.array(
...     [
...         [0.0, 0.0],
...         [1.0, 0.0],
...         [0.0, 1.0],
...         [-1.0, 0.0],
...         [0.0, -1.0],
...         [0.5, 0.5],
...     ]
... )
>>> x = np.array([[0.0, 0.0], [50.0, 50.0]])
>>> report = flag_out_of_distribution(x, reference, k=1)
>>> report.method, report.n_reference
('mahalanobis', 6)