2.15. pycsamt.transformers#

Data transformation utilities for J/EDI conversion and spectra workflows.

Converters from AVG, Jones, spectra, and time-series data to EDI.

class pycsamt.transformers.TransformerMixin#

Bases: MTBase

Template utilities for AVG/J -> EDI transformations.

This module defines TransformerMixin, a light template that standardizes the steps needed to convert foreign transfer-function containers (e.g. Zonge AVG or Jones J) into EDI files or collections.

The mixin focuses on finalization of a neutral payload (pycsamt.core.base.TFBundle): validating the station name, ordering and de-duplicating frequencies, and optionally filling missing pieces (Z vs rho/phase) according to user config.

Notes

Subclasses provide the data plumbing:

TransformerMixin.extract() pulls a TFBundle out of the source object or file.
TransformerMixin.emit_edi() builds an EDI object from the finalized bundle.
TransformerMixin.post_emit() can attach metadata or perform small last-mile fixes.

The default hooks for filling missing parts are no-ops. Backends can override TransformerMixin.compute_res_from_z() and TransformerMixin.compute_z_from_res().

Subclasses implement the data path while this mixin handles standard finalization steps on a TFBundle:

Validate or synthesize the station name using the global policy.
Order frequencies per config (asc/desc).
De-duplicate nearly equal frequencies using a relative tolerance.
Optionally fill missing parts (Z vs rho/phase) via overridable hooks.

Finalization obeys the global configuration from pycsamt.core.config. In particular:

freq_order controls sorting.
freq_tol controls de-duplication.
compute_res_from_z and compute_z_from_res toggle filling behavior.

See also

pycsamt.core.transformers.AVGtoEDI: Concrete AVG → EDI/EDICollection converter.
pycsamt.core.transformers.JtoEDI: Concrete J → EDI/EDICollection converter.
pycsamt.core.base.TFBundle: Neutral payload used across backends.
pycsamt.core.config: Global configuration and frequency policies.

Examples

A minimal custom transformer:

>>> from pycsamt.core._transformers import TransformerMixin
>>> from pycsamt.core.base import TFBundle
>>>
>>> class MyX(TransformerMixin):
...     def extract(self, source):
...         # pull TF data from source into a bundle
...         return TFBundle(freq=[1.0], rho=[100.0], phase=[45.0])
...
...     def emit_edi(self, bundle):
...         # return an EDI-like stub for illustration
...         return {"freq": bundle.freq, "rho": bundle.rho}
>>> out = MyX().transform(object())
>>> isinstance(out, dict)
True

References

[TransformerMixin-1]

Simpson, F. & Bahr, K. (2005). Practical MT.

[TransformerMixin-2]

Egbert, G. D. (1997). Robust MT processing.

extract(source)#

Extract a TFBundle from source.

This is the ingest step. Implementors parse the foreign object (or file path) and return a neutral bundle. Keep the method free of side effects.

Parameters:: source (Any) – Backend-specific object or path.
Returns:: A bundle containing frequencies and transfer functions.
Return type:: TFBundle
Raises:: NotImplementedError – Must be provided by subclasses.

Notes

Prefer leaving station naming and frequency manipulation to the mixin. Provide raw content here.

emit_edi(bundle)#

Materialize an EDI object from a finalized bundle.

This is the emit step. Implementors build and return an EDI object (or a compatible stub).

Must be provided by subclasses.

Notes

Do not attempt to reorder frequencies here. The bundle has already been finalized by _finalize().

Parameters:: bundle (TFBundle)
Return type:: Any

post_emit(edi_obj, source, bundle)#

Optional last-mile adjustments after EDI creation.

Use this hook to enrich the EDI object with auxiliary metadata (e.g. site location, elevation) that is awkward to thread through earlier steps.

Parameters:

edi_obj (Any) – The EDI object returned by emit_edi().
source (Any) – The original source object used in extract().
bundle (TFBundle) – The finalized bundle.

Returns:

The potentially modified EDI object.

Return type:

Any

Notes

The default implementation returns edi_obj unmodified.

compute_res_from_z(b)#

Compute apparent resistivity/phase from Z.

This hook is called by _fill_missing() when compute_res_from_z is True in the global config, and the bundle holds Z but lacks (rho, phase).

Parameters:: b (TFBundle) – Input bundle.
Returns:: The updated bundle (may be the same instance).
Return type:: TFBundle

Notes

The default implementation is a no-op. Subclasses may override to apply the standard MT relations [TransformerMixin-compute-res-from-z-1] [TransformerMixin-compute-res-from-z-2].

compute_z_from_res(b)#

Reconstruct Z from apparent resistivity/phase.

This hook is called by _fill_missing() when compute_z_from_res is True in the global config, and the bundle lacks Z but holds (rho, phase).

Parameters:: b (TFBundle) – Input bundle.
Returns:: The updated bundle (may be the same instance).
Return type:: TFBundle

Notes

The default implementation is a no-op. Subclasses may override to apply physically consistent reconstruction.

transform(source, *, name=None, station_id=None)#

One-shot transform: extract → finalize → emit → post.

This orchestrates the whole conversion pipeline:

extract() to build a TFBundle.
_finalize() to normalize the bundle.
emit_edi() to create the EDI object.
post_emit() for optional enrichment.

Parameters:

source (Any) – Source object or file path.
name (str, optional) – Preferred station name override.
station_id (str or int, optional) – Identifier used if a name must be synthesized.

Returns:

The resulting EDI object (or compatible subtype).

Return type:

Any

Examples

>>> from pycsamt.core._transformers import TransformerMixin
>>> class Mini(TransformerMixin):
...     def extract(self, s):
...         from pycsamt.core.base import TFBundle
...
...         return TFBundle(freq=[1.0], rho=[100.0], phase=[45.0])
...
...     def emit_edi(self, b):
...         return {"ok": True, "n": len(b.freq)}
>>> Mini().transform(object())["ok"]
True

class pycsamt.transformers.AVGtoEDI#

Bases: TransformerMixin

Convert a Zonge AVG source to an EDICollection.

This transformer orchestrates extraction of transfer functions from a Zonge AVG object (or file path), finalizes the neutral payload (TFBundle), and emits EDI objects. The result is always an EDICollection, even for single-site inputs.

The pipeline is:

Parse TFs from the AVG using to_tensor. When Z is not present, fall back to (rho, phase).
Finalize each bundle using TransformerMixin: validate the station name, order and de-duplicate frequencies, and fill missing parts if enabled by the global config.
Materialize EDIFile objects and attach site metadata (e.g., coordinates) if available from avg.topo.frame.

Notes

Station naming obeys the global policy from pycsamt.core.config.
Frequency sorting and de-duplication follow freq_order and freq_tol.
If only (rho, phase) are present, Z can be reconstructed when compute_z_from_res is enabled.
If avg.topo exposes a DataFrame-like frame with columns station, latitude, longitude and optionally elevation, a >HEAD section is updated.

See also

pycsamt.core._transformers.TransformerMixin: Provides finalization steps and overridable hooks.
pycsamt.core.base.TFBundle: Neutral, backend-agnostic payload.
pycsamt.seg.edi.EDIFile: Concrete EDI object.
pycsamt.seg.collection.EDICollection: Collection wrapper for multiple EDI files.

Examples

>>> from pycsamt.core.transformers import AVGtoEDI
>>> # path or AVG object are both accepted
>>> # out = AVGtoEDI().transform(\"/path/to/data.avg\")

compute_z_from_res(b)#

Reconstruct Z from apparent resistivity and phase.

If a bundle lacks Z but holds (rho, phase) and the global config enables compute_z_from_res, this method builds a complex tensor whose magnitude and angle satisfy the standard MT relations:

|Z| = sqrt( μ0 * ω * ρa )
phase is interpreted as milliradians and converted to radians before forming Z = |Z|(cos φ + i sin φ).

Parameters:: b (TFBundle) – Input bundle with rho and phase set.
Returns:: The same bundle with z populated.
Return type:: TFBundle

Notes

Ensure phase units are consistent. Zonge workflows often store milliradians; the implementation converts by dividing by 1000. If units differ, override this method.

References

[AVGtoEDI-compute-z-from-res-1]

Simpson, F. & Bahr, K. (2005). Practical MT.

[AVGtoEDI-compute-z-from-res-2]

Egbert, G. D. (1997). Robust MT processing.

extract(source)#

Pull the first TFBundle from an AVG object or path.

Parameters:: source (Any) – AVG instance or path to an .avg file.
Returns:: The first available bundle (typically first station).
Return type:: TFBundle
Raises:: ValueError – If no transfer functions could be obtained.

Notes

Multi-station inputs are handled by transform(), which iterates and converts all sites to an EDI collection.

emit_edi(bundle)#

Materialize an EDIFile from a finalized bundle.

Populates Z fields (_freq, _z, _z_err). If Z is missing but (rho, phase) are present, uses the backend method to set resistivity and phase. If tipper data exist, populates tipper arrays and derived attributes.

Parameters:: bundle (TFBundle) – Finalized transfer-function payload.
Returns:: A concrete EDI object.
Return type:: EDIFile

Notes

Errors during optional computations are silenced to keep the transformation robust on imperfect field data.

post_emit(edi_obj, source, bundle)#

Attach station metadata and optional location to the EDI.

This step applies the global station naming policy and, if avg.topo.frame is present, injects coordinates into the >HEAD section.

Parameters:

edi_obj (EDIFile) – Newly created EDI object.
source (AVG) – Original AVG used to derive the bundle.
bundle (TFBundle) – Finalized bundle for this station.

Returns:

The updated EDI object.

Return type:

EDIFile

Notes

When a matching row is found in topo.frame (based on the station column), latitude, longitude and elevation are copied if available.

transform(source, *, name=None, station_id=None)#

Convert an AVG source to an EDICollection.

Handles both objects and file paths. All stations found in the source are extracted, finalized, converted to EDI, and collected.

Parameters:

source (Any) – AVG instance or path-like.
name (str, optional) – Preferred station name override, applied during finalization when present.
station_id (str or int, optional) – Identifier used by the naming policy if a name must be synthesized.

Returns:

A collection of EDIFile objects, one per site.

Return type:

EDICollection

Examples

>>> from pycsamt.core.transformers import AVGtoEDI
>>> # out = AVGtoEDI().transform(\"/path/site.avg\")

class pycsamt.transformers.JtoEDI#

Bases: TransformerMixin

Convert a Jones J source to EDI or an EDI collection.

This transformer ingests a JFile instance or a path to a .j file, extracts transfer functions (Z, tipper, or fallback rho/phase), finalizes a neutral payload (TFBundle), and emits concrete EDIFile objects. When given a JCollection, it produces an EDICollection.

The pipeline mirrors the AVG workflow:

Extract TF arrays and metadata from the J structure, tolerating variant attribute names often found in legacy files.
Finalize each bundle using the mixin logic: ensure a valid station name, order and de-duplicate frequencies, and fill missing TF parts if configured.
Emit EDI and optionally attach site coordinates if a header-like structure is present on the J side.

Notes

Naming, frequency sorting, and de-duplication follow the global configuration in pycsamt.core.config.
If only (rho, phase) are present, Z can be reconstructed when allowed by compute_z_from_res (via the mixin hook).
A Head or head object on JFile with attributes lat, long, and elev is used to populate the EDI >HEAD section when available.

See also

pycsamt.core._transformers.TransformerMixin: Finalization steps and overridable hooks.
pycsamt.core.transformers.AVGtoEDI: Companion converter for Zonge AVG.
pycsamt.core.base.TFBundle: Neutral payload used across backends.

Examples

>>> from pycsamt.core.transformers import JtoEDI
>>> # Single file → EDIFile
>>> # edi = JtoEDI().transform(\"/path/site.j\")
>>> # Collection → EDICollection
>>> # out = JtoEDI().transform(j_collection)

References

[JtoEDI-1]

Simpson, F. & Bahr, K. (2005). Practical MT.

[JtoEDI-2]

Egbert, G. D. (1997). Robust MT processing.

extract(source)#

Extract a TFBundle from a J file or object.

Parameters:: source (Any) – JFile instance or a path to a .j file.
Returns:: Bundle with TF arrays and basic site info.
Return type:: TFBundle
Raises:: ValueError – If no usable transfer functions are present.

Notes

This method does not reorder or de-duplicate; those steps are applied in the finalization phase.

emit_edi(bundle)#

Create an EDIFile from a finalized bundle.

Populates impedance arrays when present. If Z is absent but (rho, phase) are available, the backend call is used to set resistivity/phase onto the EDI structure. Tipper data are also populated when provided.

Parameters:: bundle (TFBundle) – Finalized transfer-function payload.
Returns:: Concrete EDI object ready for downstream use.
Return type:: EDIFile

Notes

Optional computations are guarded; failures are ignored to remain robust on imperfect field data.

post_emit(edi_obj, source, bundle)#

Attach naming and optional coordinates to the EDI.

Applies the global station naming policy and, when the J object exposes a Head/head with lat, long, and elev, updates the EDI header accordingly.

Parameters:

edi_obj (EDIFile) – Newly created EDI object.
source (JFile) – Original J source used for extraction.
bundle (TFBundle) – Finalized bundle for this site.

Returns:

The same EDI object, augmented in place.

Return type:

EDIFile

transform(source, *, name=None, station_id=None)#

Convert a J source to EDI or an EDI collection.

A JCollection yields an EDICollection with one EDI per member. A single JFile (or path) yields a single EDIFile.

Parameters:

source (Any) – JFile instance, path-like, or JCollection.
name (str, optional) – Preferred station name override for finalization.
station_id (str or int, optional) – Identifier used when a name must be synthesized.

Returns:

EDI materialization corresponding to the input form.

Return type:

EDIFile or EDICollection

Examples

>>> from pycsamt.core.transformers import JtoEDI
>>> # Single file
>>> # edi = JtoEDI().transform(\"/data/site.j\")
>>> # Collection
>>> # out = JtoEDI().transform(j_collection)

class pycsamt.transformers.SpectraToEDI(*, e_labels=('EX', 'EY'), h_labels=('HX', 'HY'), ridge=None, estimate_error=False, dof=None, use_remote=False, station_suffix='', skip_errors=True, verbose=0)#

Bases: TransformerMixin

Transform SEG spectra-EDI files to MT-impedance EDI files.

The transformer calls from_file() on each input file, then to_edi() to recover Z and, when a vertical-magnetic channel is present, the tipper. All successfully converted files are collected into a single EDICollection.

Parameters:

e_labels (tuple of str, default ("EX", "EY")) – Electric-channel type labels used to identify E-field channels in the spectra block.
h_labels (tuple of str, default ("HX", "HY")) – Horizontal magnetic-channel type labels.
ridge (float, optional) – Tikhonov regularisation added to the magnetic block before inversion. Useful for numerically ill-conditioned spectra. None (default) applies no regularisation.
estimate_error (bool, default False) – Propagate 1-σ impedance uncertainties into >ZXX.VAR / >ZXY.VAR / … blocks using first-order complex-Wishart error propagation.
dof (float or ndarray, optional) – Effective degrees of freedom per frequency, forwarded to to_Z(). When None, the transformer tries to infer DoF from spectra metadata (segnum, avgt, bw).
use_remote (bool, default False) – When both local and remote electric channels are present, set True to select the remote reference.
station_suffix (str, default "") – Appended to the station name of every converted EDI. Use "_IMP" to reproduce the HBH03 → HBH03_IMP convention.
skip_errors (bool, default True) – If True, per-file conversion failures are caught, logged, and included in TransformResult.failures without aborting the batch. Set to False to raise on first error.
verbose (int, default 0) – Verbosity level. 1 prints per-file progress; 2 adds debug detail.

Examples

Single file:

col = SpectraToEDI().transform("HBH03.edi")
ed = col[0]
ed.write(savepath="output/")

Directory, write to disk:

col = SpectraToEDI(
    estimate_error=True, station_suffix="_IMP"
).transform(
    "spectra_dir/",
    output_dir="imp_edis/",
)
print(len(col), "station(s) converted")

Batch with failure report:

result = SpectraToEDI(skip_errors=True).transform_batch(
    "spectra_dir/",
    output_dir="imp_edis/",
)
print(f"{result.n_ok} ok,  {result.n_fail} failed")

Force a station-name override for a single file:

col = SpectraToEDI().transform("site.edi", station_name="CUSTOM_01")

See also

pycsamt.seg.spectra.Spectra.to_Z: Underlying Z / tipper estimation.
pycsamt.seg.spectra.Spectra.to_edi: Single-file spectra → EDIFile conversion.
pycsamt.transformers.AVGtoEDI: Analogous transformer for Zonge AVG files.
pycsamt.transformers.JtoEDI: Analogous transformer for Jones J files.

transform(source, *, output_dir=None, station_name=None)#

Transform one or multiple spectra EDIs to impedance EDIs.

Parameters:

source (path-like, EDIFile, EDICollection, Spectra, or list) –
Accepts:
- A single .edi file path (str / Path).
- A directory — all *.edi files inside are processed.
- A EDIFile that holds a spectra section (its .path is used for the source metadata).
- A EDICollection.
- A pre-loaded Spectra object.
- A list of any of the above.
output_dir (path-like, optional) – When given, each converted EDI is written to this directory. The directory is created if it does not exist.
station_name (str, optional) – Override the station name for single-file input only. Ignored when source resolves to multiple files.

Returns:

Successfully converted impedance EDI files.

Return type:

EDICollection

Raises:

RuntimeError – When skip_errors=False and any file fails to convert.
ValueError – When source resolves to zero convertible files.

transform_batch(source, *, output_dir=None, station_name=None)#

Like transform() but always returns a TransformResult.

Useful when you want to inspect failures without raising.

Parameters:

source (Any) – Same as transform().
output_dir (str | Path | None) – Same as transform().
station_name (str | None) – Same as transform().

Return type:

TransformResult

classmethod with_errors(**kw)#

Return a transformer with estimate_error=True.

Parameters:: kw (Any)
Return type:: SpectraToEDI

classmethod with_tipper_suffix(**kw)#

Return a transformer that appends '_IMP' to station names.

Parameters:: kw (Any)
Return type:: SpectraToEDI

class pycsamt.transformers.TransformResult(collection, failures=<factory>)#

Bases: object

Outcome of a batch spectra→impedance conversion.

Variables:

collection (EDICollection) – Successfully converted EDI files.
failures (list of _FailRecord) – Per-file failures (source path + error message).
n_ok (int) – Number of successful conversions.
n_fail (int) – Number of failed conversions.

Parameters:

collection (EDICollection)
failures (list[_FailRecord])

Examples

result = SpectraToEDI().transform_batch("spectra_dir/")
print(result.n_ok, "ok, ", result.n_fail, "failed")
for f in result.failures:
    print(f.source, "->", f.error)

collection: EDICollection#

failures: list[_FailRecord]#

property n_ok: int#

property n_fail: int#

class pycsamt.transformers.TStoEDI(*, estimator='ls', remote=None, nfft=None, overlap=0.5, per_decade=7, fmin=None, fmax=None, robust='huber', ridge=None, estimate_error=False, include_spectra=False, include_tseries=False, reader_kws=None, station_suffix='', skip_errors=True, verbose=0, **spectra_kws)#

Bases: TransformerMixin

Transform raw MT field time series to impedance EDI files.

Each source record is processed with pycsamt.ts.ts_to_edifile(): band-averaged cross-power spectra (Hann-tapered overlapping segments, Huber-weighted robust stacking), impedance recovery Z = S_EH · S_HH⁻¹ (or remote reference Z = S_ER · S_HR⁻¹), and EDI header synthesis from the recorded site metadata (coordinates, channel azimuths, dipole lengths). All successfully converted records are gathered into a single EDICollection.

Parameters:

estimator ({'ls', 'rr'}, default 'ls') – Single-site least squares, or remote reference (needs remote).
remote (TSData, optional) – Simultaneous remote-reference record whose horizontal magnetic channels are appended as RHX/RHY.
nfft (int, optional) – FFT segment length. Defaults to an automatic choice targeting ~60+ segments at 50 % overlap.
overlap (float, default 0.5) – Fractional segment overlap.
per_decade (int, default 7) – Estimation frequencies per decade.
fmin (float, optional) – Target frequency range (Hz).
fmax (float, optional) – Target frequency range (Hz).
robust ({'huber', None}, default 'huber') – Segment stacking scheme.
ridge (float, optional) – Tikhonov regularisation of the inverted spectral block.
estimate_error (bool, default False) – Propagate 1-σ impedance uncertainties into >ZXX.VAR / … blocks (DoF from the band averaging).
include_spectra (bool, default False) – Embed the estimated cross-spectra as >SPECTRA blocks in each output EDI.
include_tseries (bool, default False) – Embed a decimated head of the raw series as >TSERIES blocks.
reader_kws (dict, optional) – Extra options for pycsamt.ts.readers.read_ts() (e.g. {"declination": -19.5} for EMSLAB, or {"format": "ascii", "dt": 1.0}).
station_suffix (str, default "") – Appended to the station name of every converted EDI (e.g. "_TS").
skip_errors (bool, default True) – Catch and record per-record failures instead of aborting the batch.
verbose (int, default 0) – Verbosity level.
**spectra_kws – Any further option forwarded to pycsamt.ts.process.ts_to_spectra() (freqs, n_iter, huber_k, min_segments, fill_gap, …).

Examples

Single LiMS file, write to disk:

col = TStoEDI(nfft=10240).transform(
    "data/MT/TS/kap103as.ts/kap103as.ts",
    output_dir="imp_edis/",
)

EMSLAB file with reader options and errors:

col = TStoEDI(
    estimate_error=True,
    reader_kws={"declination": -19.5},
).transform("data/MT/TS/emsl01.asc/emsl01.asc")

Batch a directory and inspect failures:

result = TStoEDI().transform_batch("data/MT/TS/")
print(result.n_ok, "ok,", result.n_fail, "failed")

See also

pycsamt.ts.ts_to_edi: Functional one-record counterpart.
pycsamt.transformers.SpectraToEDI: Same API for spectra-EDI inputs.

transform(source, *, output_dir=None, station_name=None)#

Transform one or multiple time-series records to EDIs.

Parameters:

source (path-like, TSData, or list) –
Accepts:
- A single field file path (LiMS .ts, EMSLAB .asc, generic ASCII, or an EDI carrying >TSERIES blocks).
- A directory — files matching *.ts, *.asc, *.dat, *.txt are collected recursively.
- An in-memory TSData.
- A list mixing any of the above.
output_dir (path-like, optional) – When given, each converted EDI is written there as <station>_ts.edi (directory created if needed).
station_name (str, optional) – Station override for single-record input only. Ignored when source resolves to multiple records.

Returns:

Successfully converted impedance EDI files.

Return type:

EDICollection

Raises:

RuntimeError – When skip_errors=False and any record fails.
ValueError – When source resolves to zero records.

transform_batch(source, *, output_dir=None, station_name=None)#

Like transform() but always returns a TransformResult.

Parameters:

source (Any) – Same as transform().
output_dir (str | Path | None) – Same as transform().
station_name (str | None) – Same as transform().

Return type:

TransformResult

classmethod with_errors(**kw)#

Return a transformer with estimate_error=True.

Parameters:: kw (Any)
Return type:: TStoEDI

classmethod with_remote(remote, **kw)#

Return a remote-reference transformer.

Parameters:

remote (TSData)
kw (Any)

Return type:

TStoEDI

2.15.1. Transformer Modules#

`pycsamt.transformers._base`
`pycsamt.transformers.jedi`
`pycsamt.transformers.spectra`	pycsamt.transformers.spectra