2.25.3.20. pycsamt.ai.processing.qc#

EMQCScorer — ML-based per-frequency quality control scorer.

Extends the rule-based QC tools in pycsamt.emtools.qc with an Isolation Forest anomaly model trained on signal-quality features.

Features extracted per (site, frequency) observation#

  • SNR: \(|\bar{Z}| / \sigma(Z)\) (signal-to-noise ratio)

  • Phase stability: coefficient of variation of the off-diagonal phases

  • Swift skew: \(|\beta_\text{Swift}| = |(Z_{xx}+Z_{yy})/(Z_{xy}-Z_{yx})|\)

  • Phase tensor skew \(|\beta|\) (requires full tensor)

  • Off-diagonal amplitude asymmetry: \(\log_{10}(|Z_{xy}|/|Z_{yx}|)\)

The combined score lies in [0, 1] — 1 means good quality, 0 means flagged bad.

Working directly with site collections#

EMQCScorer.apply() accepts and returns a site collection directly – scoring every station-frequency cell and masking (or interpolating over) the flagged ones in Z.z – so the scorer can be dropped into the same sites-in / sites-out chain as the rule-based functions in pycsamt.emtools.remove_noise.

Classes

EMQCScorer([contamination, snr_threshold, ...])

ML-based per-frequency QC scorer for MT impedance data.
class pycsamt.ai.processing.qc.EMQCScorer(contamination=0.05, snr_threshold=3.0, skew_threshold=0.3, score_threshold=0.5, use_ml=True, n_estimators=100, random_state=None)[source]

Bases: BaseEMProcessor

ML-based per-frequency QC scorer for MT impedance data.

Combines hard thresholds on SNR and Swift skew with an IsolationForest anomaly model fitted on extracted signal-quality features. The final quality score for each (site, frequency) cell is \(s \in [0, 1]\); observations below score_threshold should be rejected before inversion.

Parameters:

  • contamination (float, default 0.05) – Expected fraction of contaminated samples, passed directly to sklearn.ensemble.IsolationForest.

  • snr_threshold (float, default 3.0) – Observations with SNR below this value are hard-flagged as bad regardless of the ML score.

  • skew_threshold (float, default 0.3) – Observations with Swift skew above this value are hard-flagged.

  • score_threshold (float, default 0.5) – Quality scores below this value are considered bad.

  • use_ml (bool, default True) – When False, only the rule-based thresholds are applied and the IsolationForest is not fitted.

  • n_estimators (int, default 100) – Number of trees in the IsolationForest.

  • random_state (int or None)

Examples

>>> from pycsamt.ai.processing import EMQCScorer
>>> scorer = EMQCScorer(snr_threshold=5.0, use_ml=False)
>>> scorer.fit(sites)
EMQCScorer(rule_only)
>>> tbl = scorer.score_table(sites)
fit(X, **kwargs)[source]

Fit the QC model on a training set.

Parameters:

X (SiteCollection, dict, list of Z, or ndarray (n_samples, 5)) – Training data. Site collections are converted automatically. Pass a precomputed feature matrix to skip extraction.

Return type:

self

transform(X)[source]

Compute quality scores.

Parameters:

X (SiteCollection or ndarray (n_samples, 5))

Returns:

scores – Quality scores in [0, 1]; 1 = good, 0 = bad.

Return type:

ndarray, shape (n_samples,)

score_table(sites)[source]

Return per-(site, frequency) QC table.

Parameters:

sites (SiteCollection or compatible)

Returns:

df – Columns: station, freq, snr, swift_skew, asym, phase_xy, phase_yx, score, flag (0=bad, 1=good).

Return type:

DataFrame

apply(sites, *, mode='mask', inplace=False, recursive=True, on_dup='replace', strict=False, verbose=0)[source]

Mask flagged station-frequency cells, sites-in / sites-out.

Scores every observation with score_table(), then, for each station-frequency cell with flag == 0, either blanks the entire impedance row (mode="mask") or replaces it with a linear interpolation from its nearest good neighbours (mode="interp") — the same two modes, and the same underlying mechanism, as notch_powerline(). As there, a whole frequency row is masked together rather than only the components the score is computed from, since a hard-flagged frequency is not considered trustworthy for any component.

Parameters:

  • sites (SiteCollection or compatible)

  • mode ("mask" | "interp", default "mask")

  • inplace (bool, default False) – When False (the default), sites is left untouched and a corrected copy is returned. When True, sites is modified in place and returned.

  • recursive (bool) – Passed to ensure_sites().

  • on_dup (str) – Passed to ensure_sites().

  • strict (bool) – Passed to ensure_sites().

  • verbose (int) – Passed to ensure_sites().

Returns:

corrected – A site collection with the same stations; flagged station-frequency rows are masked or interpolated in Z.z, every other row is untouched.

Return type:

Sites

Examples

>>> scorer = EMQCScorer(random_state=0).fit(sites)
>>> masked_sites = scorer.apply(sites)