2.25.3.18. pycsamt.ai.processing.imputer#

EMImputer — masked-reconstruction gap filling for MT impedance data.

Why this is not EMDenoiser#

EMDenoiser assumes every (station, frequency) cell has some value and improves its quality. Real surveys also have cells with no value at all – a frequency dropped mid-acquisition, a component that never converged, a station skipped for one band – and nothing else in pyCSAMT fills those gaps with anything better than leaving them NaN through to inversion. EMImputer targets that gap specifically: reconstructing the missing cells from the observed ones, using spectral smoothness (this station, nearby frequencies) – a matrix-completion / masked-autoencoder problem, not a denoising one.

Architecture#

A self-supervised masked-reconstruction objective, the same family as BERT-style masked-language-modelling or image inpainting:

\[\hat{\mathbf{x}} = f_\theta(\mathbf{x} \odot \mathbf{m},\ \mathbf{m}), \qquad \mathcal{L} = \frac{\big\| (\hat{\mathbf{x}} - \mathbf{x}) \odot \mathbf{m}_\text{syn} \big\|_2^2}{\sum \mathbf{m}_\text{syn}}\]

where \(\mathbf{m} \in \{0, 1\}\) is the mask fed to the network (1 = “the network is told this cell is observed”) and \(\mathbf{m}_\text{syn}\) marks the subset of genuinely observed cells that were synthetically hidden this step, which is exactly where the loss is evaluated. On every training step, EMImputer.fit() hides a random mask_frac fraction of the truly-observed cells (any real NaN gaps in the training data are always hidden and never used as a target, since their true value is unknown) – mirroring the way EMDenoiser adds synthetic noise, so training data only needs to be mostly complete rather than needing real missing-data examples with known ground truth.

The network is the same 1-D convolutional encoder-decoder family as EMDenoiser (_build_cae_torch() / _build_cae_tf), with one structural difference: the encoder takes 2 * n_components input channels (the zeroed-where-hidden values, concatenated with the mask along the channel axis) so the network can distinguish “zero because hidden” from “zero because that is genuinely the value”, while the decoder still only reconstructs n_components channels – the mask itself is never reconstructed.

Input tensor shape#

(n_samples, n_components, n_freqs), identical to prepare_z_features()’s output – NaN marks a missing cell.

Working directly with site collections#

EMImputer.apply() accepts and returns a site collection directly, writing the reconstructed impedance back only into cells that were genuinely missing on the input – every observed measurement is left byte-identical – the same sites-in / sites-out convention as apply().

Classes

EMImputer([n_freqs, n_components, channels, ...])

Masked-reconstruction gap filler for MT impedance tensor data.
class pycsamt.ai.processing.imputer.EMImputer(n_freqs=None, n_components=4, *, channels=(64, 128, 64), dropout=0.1, device=None)[source]

Bases: BaseEMProcessor

Masked-reconstruction gap filler for MT impedance tensor data.

Trains a 1-D convolutional autoencoder to reconstruct synthetically-hidden cells of otherwise-observed impedance spectra, then applies that network to fill genuinely missing (NaN) cells at inference time. Supports both PyTorch and TensorFlow backends via pycsamt.backends.

Parameters:

  • n_freqs (int or None, default None) – Number of frequency channels in the input. When None, inferred from X.shape[2] on the first call to fit().

  • n_components ({4, 8}, default 4) – Feature channels per frequency, matching prepare_z_features().

  • channels (tuple of int, default (64, 128, 64)) – Channel widths for the encoder/decoder stages.

  • dropout (float, default 0.1) – Dropout probability in the encoder bottleneck.

  • device (str or None) – "cpu", "cuda", "mps", "/GPU:0", or None for auto-detect via the active backend.

Notes

When neither PyTorch nor TensorFlow is available the imputer falls back to per-sample, per-channel linear interpolation along the frequency axis (_freq_axis_fill()) – no spatial (cross- station) information, unlike the network path.

Call prepare_z_features() to convert a site collection to the (n_samples, n_components, n_freqs) array expected here, or use apply() to go straight from a site collection to a filled one without handling the array form yourself.

Examples

>>> import numpy as np
>>> X = np.random.randn(200, 4, 32).astype("float32")
>>> rng = np.random.default_rng(0)
>>> gaps = rng.random(X.shape) < 0.05
>>> X[gaps] = np.nan
>>> imp = EMImputer()
>>> imp.fit(X, epochs=5, verbose=False)
EMImputer(n_freqs=32, n_components=4, fitted)
>>> X_filled = imp.transform(X)

Sites in, filled sites out – the same convention as apply():

>>> complete_sites = imp.apply(sites)
fit(X, *, mask_frac=0.15, epochs=80, batch_size=64, lr=0.001, val_frac=0.1, seed=None, verbose=True)[source]

Train the imputer on a (mostly) complete impedance array.

Parameters:

  • X (ndarray, shape (n_samples, n_components, n_freqs)) – Training data. NaN cells, if any, are excluded from the reconstruction loss – only cells that are genuinely observed can be synthetically hidden and used as a self-supervised target.

  • mask_frac (float, default 0.15) – Fraction of the observed cells to hide on every training step, supplying the reconstruction target (see the module docstring). Too low and the network sees little missing- data signal to learn from; too high and too little context remains for it to reconstruct anything.

  • epochs (int) – Maximum number of training epochs.

  • batch_size (int)

  • lr (float) – Initial Adam learning rate.

  • val_frac (float) – Fraction of training samples held out for validation.

  • seed (int or None)

  • verbose (bool)

Return type:

self

Raises:

ValueError – If X is not 3-D, its frequency count does not match a previously-fitted n_freqs, or it contains no finite values at all.

transform(X)[source]

Fill missing (NaN) cells in X.

Parameters:

X (ndarray, shape (n_samples, n_components, n_freqs)) – Data with NaN marking missing cells.

Returns:

X_filled – Observed cells returned unchanged (byte-identical to the input); missing cells replaced by the network’s reconstruction.

Return type:

ndarray, same shape as input

apply(sites, *, freq_ref=None, log_amp=True, inplace=False, recursive=True, on_dup='replace', strict=False, verbose=0)[source]

Fill a site collection’s missing impedance cells in place.

Extracts features with prepare_z_features(), imputes them with transform(), then writes the reconstruction back onto each site’s own frequency grid – but only at frequency rows that were genuinely missing on that site to begin with; every observed row is left byte-identical to the input. Mirrors apply()’s sites-in / sites-out convention.

Parameters:

  • sites (SiteCollection or compatible)

  • freq_ref (ndarray or None) – Common frequency grid (Hz) the network was trained on. Must match the grid used to prepare the training data – when None (the default), the first site’s own grid is used, matching prepare_z_features()’s own default.

  • log_amp (bool, default True) – Must match the log_amp used to build the training features.

  • inplace (bool, default False) – When False (the default), sites is left untouched and a filled copy is returned. When True, sites is modified in place and returned.

  • recursive (bool) – Passed to ensure_sites().

  • on_dup (str) – Passed to ensure_sites().

  • strict (bool) – Passed to ensure_sites().

  • verbose (int) – Passed to ensure_sites().

Returns:

filled – A site collection with the same stations. Stations with no missing cells at all are returned untouched.

Return type:

Sites

Warns:

  • With the default ``n_components=4``, only :math:`Z_{xy}` and

  • :math:`Z_{yx}` are modelled – a real missing measurement

  • usually means the *whole* frequency row (all four components)

  • is absent, and the diagonal :math:`Z_{xx}, Z_{yy}` at those

  • rows is left exactly as it was (``NaN``, if it was ``NaN``),

  • the same “not modelled -> untouched” contract

  • :meth:`~pycsamt.ai.processing.denoise.EMDenoiser.apply` uses.

  • Fit and apply with ``n_components=8`` to fill every component.

property history_: dict[str, list][source]

Training history recorded by the last fit() call.

Returns:

history – {"train_loss": [...], "val_loss": [...]}, one value per epoch, both computed only over synthetically-hidden cells (see the module docstring). Empty when the frequency- interpolation fallback was used (no epoch loop) or before fit() has been called. Pass directly to plot_training_history().

Return type:

dict