pycsamt.ai.domain_gap.survey_fit#

Fit plausible CorruptionConfig ranges from real data.

This module is the bridge between a real field survey (AMT, CSAMT, MT, or otherwise) and the numpy-only pycsamt.ai.domain_gap.simulator. It is deliberately split in two:

survey_data_from_sites()

Converts EDI/Sites/APISurvey input into a canonical SurveyData, using only the quantities the M1 data contract defines. This path stays numpy-only once the bridge itself has run.

fit_corruption_config()

Derives plausible noise/error-floor/dropout ranges purely from a SurveyData’s own declared errors and coverage. No EDI or pandas dependency is required at this stage.

fit_distortion_priors_from_sites()

Calls the heavier, pandas-based pycsamt.emtools.gb/pycsamt.emtools.ss diagnostics directly on real sites to estimate empirical static-shift and galvanic-distortion spreads. This is the one part of M3 that genuinely depends on a real survey’s own QC diagnostics rather than literature defaults.

The frequency grid required by SurveyData is not silently interpolated across stations with different sampling: mismatched frequency grids raise a clear error, per the plan’s non-negotiable principles.

Functions

fit_corruption_config(survey, *[, ...])

Derive plausible noise/dropout ranges from a real survey's QC.

fit_distortion_priors_from_sites(sites, *[, ...])

Estimate empirical static-shift and distortion spreads from real EDI.

fit_structured_missingness(survey, *[, l2, ...])

Fit a station+frequency+component logistic missingness model.

survey_data_from_sites(sites, *[, crs, ...])

Bridge EDI/Sites/APISurvey input to canonical SurveyData.

Classes

StructuredMissingness(alpha, station_effect, ...)

Fitted station+frequency+component logistic missingness model.
pycsamt.ai.domain_gap.survey_fit.survey_data_from_sites(sites, *, crs=None, freq_rtol=1e-06, station_spacing=500.0, recursive=True, on_dup='replace', verbose=0, metadata=None)[source]

Bridge EDI/Sites/APISurvey input to canonical SurveyData.

Parameters:

  • sites (Any) – Anything accepted by pycsamt.emtools._core.ensure_sites(): a filesystem path/glob/directory, EDIFile/EDICollection, Site/Sites, APISurvey, or an iterable of these.

  • crs (str, optional) – Coordinate reference system identifier to record. Station positions are always projected with a local equirectangular approximation (see pycsamt.ai.inversion._sites_bridge.sites_to_coords_3d()); pass a CRS string only if it genuinely describes that projection.

  • freq_rtol (float, default=1e-6) – Relative tolerance used when checking that every station shares the same frequency grid.

  • station_spacing (float, default=500.0) – Forwarded to the coordinate bridge as a uniform-grid fallback spacing, used only when no station reports finite coordinates.

  • recursive (bool) – Forwarded to ensure_sites.

  • on_dup (str) – Forwarded to ensure_sites.

  • verbose (int) – Forwarded to ensure_sites.

  • metadata (dict, optional) – Extra provenance recorded on the returned survey.

Returns:

Canonical survey with full xx, xy, yx, yy components, with impedance and its declared error converted from Site’s EDI-native [mV/km]/[nT] convention to SI (V/A), matching SurveyData’s default ImpedanceConvention.

Return type:

SurveyData

Raises:

ValueError – If no station has usable impedance data, or stations do not share a common frequency grid within freq_rtol.

Notes

This function performs no frequency interpolation: a survey whose stations were sampled on different frequency grids must be resolved by an explicit, survey-matched frequency selector (an M1 concern) before reaching this bridge.

Examples

>>> survey = survey_data_from_sites(
...     "data/AMT/WILLY_DATA/L18PLT", recursive=False, verbose=0
... )
>>> survey.components
('xx', 'xy', 'yx', 'yy')
pycsamt.ai.domain_gap.survey_fit.fit_corruption_config(survey, *, severity_scale=1.0, components=None, noise_sampling='lognormal')[source]

Derive plausible noise/dropout ranges from a real survey’s QC.

Only quantities already present in the canonical SurveyData contract are used: the impedance_error-to-|Z| ratio for heteroscedastic noise and error floor, and coverage() for dropout rates.

Parameters:

  • survey (SurveyData) – Real (or realistically corrupted) survey to profile.

  • severity_scale (float, default=1.0) – Multiplier applied to every fitted range/rate, letting a caller derive a milder or harsher preset from the same empirical fit.

  • components (sequence of str, optional) – Impedance components whose error ratio calibrates the noise range and error floor, e.g. ("xy", "yx"). Calibrate on the components that will later be compared: diagonal components usually carry much larger relative errors, and including them inflates the fitted range for an off-diagonal comparison. None uses every component present. Dropout rates always use the full coverage mask.

  • noise_sampling ({"lognormal", "loguniform", "uniform"}, default="lognormal") – Stored on the returned configuration. The default treats the fitted range as the interquartile range of a log-normal error distribution, which reproduces the median and spread of declared field errors; "uniform" restores a flat draw between the quartiles.

Returns:

Configuration whose noise range spans the interquartile range of the observed relative error, whose error floor is the fifth percentile of that ratio, and whose dropout rates equal the observed missing fractions. Distortion and outlier parameters are left at zero; see fit_distortion_priors_from_sites() for those.

Return type:

CorruptionConfig

Raises:

ValueError – If survey has no declared impedance_error to profile.

Examples

>>> import numpy as np
>>> from pycsamt.ai.data.contracts import SurveyData
>>> z = np.full((4, 6, 2), 100 + 50j)
>>> err = np.full((4, 6, 2), 3.0)
>>> survey = SurveyData(
...     z,
...     np.linspace(1000, 1, 6),
...     ["A", "B", "C", "D"],
...     ["xy", "yx"],
...     np.zeros((4, 2)),
...     impedance_error=err,
... )
>>> config = fit_corruption_config(survey)
>>> config.noise_level_range[0] >= 0.0
True
pycsamt.ai.domain_gap.survey_fit.fit_distortion_priors_from_sites(sites, *, recursive=True, on_dup='replace', verbose=0, **kwargs)[source]

Estimate empirical static-shift and distortion spreads from real EDI.

This is the one M3 entry point that genuinely depends on the heavier, pandas-based EM diagnostics in pycsamt.emtools.gb and pycsamt.emtools.ss, run directly on real sites (e.g. a WILLY line) rather than on the numpy-only SurveyData contract.

Parameters:

  • sites (Any) – Anything accepted by pycsamt.emtools._core.ensure_sites().

  • recursive (bool) – Forwarded to the underlying diagnostics.

  • on_dup (str) – Forwarded to the underlying diagnostics.

  • verbose (int) – Forwarded to the underlying diagnostics.

  • **kwargs (Any) – Forwarded to pycsamt.emtools.gb.groom_bailey_table().

Returns:

static_shift_log10_sigma, distortion_gain_log10_sigma, distortion_twist_deg_sigma, distortion_shear_sigma, and distortion_anisotropy_sigma, each the population standard deviation of the corresponding per-station fitted parameter across stations with a successful fit. A parameter is 0.0 when fewer than two stations produced a usable fit.

Return type:

dict

Examples

>>> priors = fit_distortion_priors_from_sites(
...     "data/AMT/WILLY_DATA/L18PLT", recursive=False, verbose=0
... )
>>> sorted(priors)
['distortion_anisotropy_sigma', 'distortion_gain_log10_sigma', 'distortion_shear_sigma', 'distortion_twist_deg_sigma', 'static_shift_log10_sigma']
class pycsamt.ai.domain_gap.survey_fit.StructuredMissingness(alpha, station_effect, frequency_effect, component_effect, station_names, frequencies_hz, components, l2=1.0)[source]

Bases: object

Fitted station+frequency+component logistic missingness model.

logit(pi_sjc) = alpha + a_s + b_j + q_c, with the station, frequency, and component effects a, b, q each constrained to sum to zero for identifiability (an effect relative to its own group’s mean, not to an arbitrary reference level). This replaces the three independent scalar dropout rates of fit_corruption_config()/apply_dropout() – which are mathematically forced to be numerically identical whenever each is the mean of the same overall coverage deficit taken along a different axis – with an explicit model of where missingness concentrates. Fit by fit_structured_missingness().

Parameters:

  • alpha (float) – Intercept: logit of the fitted overall mean dropout probability.

  • station_effect (tuple of float) – Per-station, per-frequency, and per-component effects, each summing to zero by construction.

  • frequency_effect (tuple of float) – Per-station, per-frequency, and per-component effects, each summing to zero by construction.

  • component_effect (tuple of float) – Per-station, per-frequency, and per-component effects, each summing to zero by construction.

  • station_names (tuple) – Identities the effects are indexed by, kept for provenance. The fitted probability field is tied to this specific grid; it is not meant to be reused against a differently shaped survey.

  • frequencies_hz (tuple) – Identities the effects are indexed by, kept for provenance. The fitted probability field is tied to this specific grid; it is not meant to be reused against a differently shaped survey.

  • components (tuple) – Identities the effects are indexed by, kept for provenance. The fitted probability field is tied to this specific grid; it is not meant to be reused against a differently shaped survey.

  • l2 (float, default=1.0) – L2 penalty weight used during fitting (not applied to alpha).

Examples

>>> import numpy as np
>>> from pycsamt.ai.data.contracts import SurveyData
>>> z = np.ones((3, 4, 1), dtype=complex)
>>> z[0, :2] = np.nan
>>> survey = SurveyData(
...     z, [4.0, 3.0, 2.0, 1.0], ["A", "B", "C"], ["xy"],
...     np.zeros((3, 2)),
... )
>>> model = fit_structured_missingness(survey)
>>> model.dropout_probability().shape
(3, 4, 1)
alpha: float
station_effect: tuple[float, ...]
frequency_effect: tuple[float, ...]
component_effect: tuple[float, ...]
station_names: tuple[str, ...]
frequencies_hz: tuple[float, ...]
components: tuple[str, ...]
l2: float = 1.0
dropout_probability()[source]

Return the fitted per-cell Bernoulli dropout probability.

Returns:

sigmoid(alpha + a_s + b_j + q_c), ready to pass to apply_structured_dropout().

Return type:

ndarray, shape (n_station, n_frequency, n_component)

Examples

>>> model = StructuredMissingness(
...     alpha=0.0, station_effect=(0.0,), frequency_effect=(0.0,),
...     component_effect=(0.0,), station_names=("A",),
...     frequencies_hz=(1.0,), components=("xy",),
... )
>>> model.dropout_probability()
array([[[0.5]]])
to_dict()[source]

Return a JSON-serializable representation.

Returns:

All fields as plain Python types, with a schema discriminator.

Return type:

dict

Examples

>>> model = StructuredMissingness(
...     alpha=0.0, station_effect=(0.0,), frequency_effect=(0.0,),
...     component_effect=(0.0,), station_names=("A",),
...     frequencies_hz=(1.0,), components=("xy",),
... )
>>> model.to_dict()["schema_version"]
1
classmethod from_dict(data)[source]

Restore a serialized model.

Parameters:

data (mapping) – State previously returned by to_dict().

Returns:

Reconstructed, validated model.

Return type:

StructuredMissingness

Examples

>>> model = StructuredMissingness(
...     alpha=0.0, station_effect=(0.0,), frequency_effect=(0.0,),
...     component_effect=(0.0,), station_names=("A",),
...     frequencies_hz=(1.0,), components=("xy",),
... )
>>> StructuredMissingness.from_dict(model.to_dict()) == model
True
pycsamt.ai.domain_gap.survey_fit.fit_structured_missingness(survey, *, l2=1.0, max_iter=500)[source]

Fit a station+frequency+component logistic missingness model.

Replaces the three scalar dropout rates of fit_corruption_config() with an explicit model of where a survey’s missingness is concentrated. With Y_sjc = 1 - survey.valid[s, j, c] the observed missingness indicator and pi_sjc = sigmoid(alpha + a_s + b_j + q_c), the fit solves the penalized Bernoulli maximum-likelihood problem

\[\hat\vartheta = \arg\min_\vartheta \Bigl[ -\sum_{sjc}\bigl\{Y_{sjc}\log\pi_{sjc} +(1-Y_{sjc})\log(1-\pi_{sjc})\bigr\} +\lambda\bigl(\lVert a\rVert_2^2+\lVert b\rVert_2^2 +\lVert q\rVert_2^2\bigr)\Bigr],\]

with a, b, q each parameterized through a sum-to-zero contrast (_sum_zero_contrast()) so the fit is identified without an arbitrary reference level, optimized by L-BFGS-B using the closed-form gradient of the penalized negative log-likelihood (d(nll)/dz_sjc = pi_sjc - Y_sjc, the standard logistic-regression result).

Parameters:

  • survey (SurveyData) – Survey whose valid mask defines the real missingness pattern to fit. Should carry a genuine, disclosed missing-data mask (e.g. a coverage-aware, snap-or-missing survey), not one already interpolated/filled – interpolating over every gap removes the very signal this fit depends on.

  • l2 (float, default=1.0) – Ridge penalty weight \(\lambda\) on the station, frequency, and component effects (the intercept is not penalized).

  • max_iter (int, default=500) – Maximum L-BFGS-B iterations.

Returns:

Fitted model. Its dropout_probability() gives the full (n_station, n_frequency, n_component) probability field for apply_structured_dropout().

Return type:

StructuredMissingness

Examples

>>> import numpy as np
>>> from pycsamt.ai.data.contracts import SurveyData
>>> z = np.ones((3, 4, 1), dtype=complex)
>>> z[0, :2] = np.nan
>>> survey = SurveyData(
...     z, [4.0, 3.0, 2.0, 1.0], ["A", "B", "C"], ["xy"],
...     np.zeros((3, 2)),
... )
>>> model = fit_structured_missingness(survey)
>>> abs(sum(model.station_effect)) < 1e-6
True
>>> abs(sum(model.frequency_effect)) < 1e-6
True