17.5. Pipeline Presets#
A pipeline preset is a named, built-in processing recipe. It gives a
survey a tested starting order without forcing the user to write every
pipeline step code by hand. A preset is not a hidden processing mode:
inside pyCSAMT it is an ordered list of (label, Step) tuples. That list
can be inspected, expanded, exported to a pipeline configuration file,
edited, reviewed, and archived like any other processing pipeline.
Use presets for quick first-pass QC, standard comparisons between processing
strategies, teaching safe default step order, and generating starter configs.
For serious survey processing, treat the preset as the beginning of the
conversation, not the final scientific decision. The reproducible record is
the expanded step sequence written to pipeline.yaml or to a committed
configuration file.
17.5.1. Preset Mental Model#
Each built-in preset is represented by a small object with three fields:
nameStable identifier used by the CLI and Python API, for example
basic_qc.descriptionShort explanation shown in preset catalogues.
stepsOrdered list of
(label, Step)tuples. The label becomes the step label used in reports and plot directories. TheStepobject holds the registry code and the parameters passed to the transform.
Mathematically, a preset defines a fixed composition of transforms:
where \(S_0\) is the input site collection, \(T_j\) is the
registered operation at position \(j\), and \(\theta_j\) is the
resolved parameter dictionary for that step. The order is part of the
meaning. Running FREQ001 before NR001 is not the same workflow as
running NR001 before FREQ001 because each step receives a different
intermediate survey state.
The preset API is intentionally small:
1>>> from pycsamt.pipeline import get_preset, list_presets, preset_catalogue
2>>> preset = get_preset("basic_qc")
3>>> preset.name
4'basic_qc'
5>>> [(label, step.spec.code) for label, step in preset.steps]
6[('notch', 'NR001'), ('drop_duplicates', 'FREQ002'), ('select_band', 'FREQ001'), ('align_grid', 'FREQ004'), ('qc_snapshot', 'QC001')]
7>>> [(p.name, len(p.steps)) for p in list_presets()]
8[('basic_qc', 5), ('noise_reduction', 6), ('full_processing', 8), ('tensor_analysis', 5), ('dimensionality_filter', 4), ('publication_ready', 9), ('stratagem_mt', 7), ('mt_qc', 11), ('amt_qc', 11), ('csamt_qc', 14), ('csumt_qc', 15)]
The CLI exposes the same information:
1pycsamt pipe presets
2pycsamt pipe presets --format json
3pycsamt pipe presets --expand basic_qc --format json
Captured expansion excerpt:
1{
2 "name": "basic_qc",
3 "description": "Minimal denoising + frequency cleanup. Good for quick-look inspection.",
4 "n_steps": 5,
5 "steps": [
6 {
7 "label": "notch",
8 "code": "NR001",
9 "name": "notch_powerline",
10 "category": "noise_removal",
11 "params": {"mains_hz": 50, "n_harm": 30, "tol_hz": 0.08}
12 },
13 {
14 "label": "drop_duplicates",
15 "code": "FREQ002",
16 "name": "drop_duplicates",
17 "category": "frequency",
18 "params": {}
19 }
20 ]
21}
The excerpt is deliberately partial. In practice, use the full expansion when reviewing or exporting a preset.
17.5.2. Run A Preset#
From the command line:
1pycsamt pipe run data/3edis \
2 --preset basic_qc \
3 --out results/basic_qc \
4 -v
From Python:
1>>> from pycsamt.pipeline import Pipeline
2>>> pipe = Pipeline.from_preset("basic_qc")
3>>> [(label, step.spec.code) for label, step in pipe]
4[('notch', 'NR001'), ('drop_duplicates', 'FREQ002'), ('select_band', 'FREQ001'), ('align_grid', 'FREQ004'), ('qc_snapshot', 'QC001')]
5>>> result = pipe.run(sites, outdir="results/basic_qc")
When output is enabled, the run writes results/basic_qc/pipeline.yaml.
That file is the run-specific canonical pipeline snapshot; it records
the step sequence that actually ran.
17.5.3. Built-In Preset Summary#
The normal pipeline registry currently provides eleven presets: the seven
processing-intent presets below, plus four method-aware presets
(mt_qc, amt_qc, csamt_qc, csumt_qc) covered in
Method-Aware Presets.
Preset |
Steps |
Best for |
Sequence |
|---|---|---|---|
|
5 |
First-pass inspection and quick survey sanity checks. |
|
|
6 |
High-EMI data where denoising is the main question. |
|
|
8 |
Standard end-to-end processing before interpretation. |
|
|
5 |
Tensor-focused cleanup after frequency and noise handling are already acceptable. |
|
|
4 |
Classifying dimensionality and keeping/projecting 2-D-compatible intervals. |
|
|
9 |
Longer reviewed workflow for polished reports and figures. |
|
|
7 |
Stratagem AMT data already loaded as a |
|
17.5.4. Choosing A Preset#
Start with the narrowest preset that answers the current question. A narrow preset is easier to diagnose because fewer transforms stand between \(S_0\) and the output state \(S_n\).
Situation |
Start with |
Why |
|---|---|---|
You just received a survey and need a quick check. |
|
It does only basic notch, frequency cleanup, alignment, and QC. |
Harmonic and spatial noise dominate the data. |
|
It stacks targeted noise-removal steps before a QC snapshot. |
You want a general processing run before inversion preparation. |
|
It combines denoising, frequency cleanup, skew gating, rotation, static-shift correction, and QC. |
You already trust the frequency grid and want tensor diagnostics. |
|
It avoids frequency and noise steps and focuses on tensor operations. |
You are deciding which intervals are compatible with 2-D assumptions. |
|
It classifies dimensionality, masks by class, projects to 2-D, and generates QC. |
You need a polished, repeatable processing chain. |
|
It is longer and more opinionated, with static-shift correction, tensor cleanup, skew gating, and final QC. |
You are working with Stratagem AMT data already represented as
|
|
It applies Stratagem-oriented AMT band selection, static shift, denoising, and QC at the emtools pipeline level. |
The table above chooses by processing intent only. If you also know the EM survey method (MT, AMT, CSAMT, or CSUMT), start from Method-Aware Presets instead — its QC steps inspect the actual data before deciding what to plot, and CSAMT/CSUMT additionally get near-field correction that the presets above never apply.
17.5.5. Preset Details#
17.5.5.1. basic_qc#
basic_qc is the safest first preset for most surveys. It does not try to
solve every processing problem; it prepares a clean enough view to understand
what the data need next.
1notch NR001 notch_powerline
2drop_duplicates FREQ002 drop_duplicates
3select_band FREQ001 select_band
4align_grid FREQ004 align_grid
5qc_snapshot QC001 qc_snapshot
The default transform is
Use basic_qc when you need quick figures before committing to a processing
plan, want to verify that EDI loading and output generation work, or need the
same minimal cleanup across several surveys. Move beyond it when static
shift, skew gating, dimensionality filtering, or stronger denoising is clearly
justified by the data.
17.5.5.2. noise_reduction#
noise_reduction concentrates on denoising. It is useful when the first
inspection shows power-line harmonics, local spikes, spatially coherent
outliers, or incoherent frequency bins.
1notch NR001 notch_powerline
2hampel NR004 hampel_filter
3spatial_med NR005 spatial_median
4shrink_trend NR003 shrink_group_trend
5mask_incoher NR010 mask_incoherent
6qc_snapshot QC001 qc_snapshot
Run it next to basic_qc rather than replacing the first pass silently:
1pycsamt pipe run data/3edis --preset basic_qc \
2 --out results/compare/basic_qc
3pycsamt pipe run data/3edis --preset noise_reduction \
4 --out results/compare/noise_reduction
17.5.5.3. full_processing#
full_processing is the standard end-to-end workflow. It starts with noise
and frequency cleanup, then applies a skew gate, strike rotation,
static-shift correction, and QC.
1notch NR001 notch_powerline
2drop_dup FREQ002 drop_duplicates
3select_band FREQ001 select_band
4align_grid FREQ004 align_grid
5mask_skew SK001 mask_by_skew
6rotate_strike TZ001 rotate_strike
7correct_ss SS001 correct_ss_ama
8qc_snapshot QC001 qc_snapshot
Use this preset when the survey needs a broad processing pass, when you want an auditable default before building an inversion-specific config, or when you need one chain that exercises the main processing families.
17.5.5.4. tensor_analysis#
tensor_analysis assumes the data are already in reasonable condition and
focuses on tensor operations.
1rotate_strike TZ001 rotate_strike
2antisymm TZ002 antisymmetrize
3sigma_clip TZ003 sigma_clip
4balance TZ004 balance_offdiag
5qc_snapshot QC001 qc_snapshot
Use it when you want to inspect tensor behavior without changing the frequency selection or applying the broader denoising chain.
17.5.5.5. dimensionality_filter#
dimensionality_filter is for 1-D / 2-D / 3-D screening and 2-D projection
workflows.
1classify_dim DIM001 classify_dim
2mask_dim DIM002 mask_by_dim
3project_2d DIM003 project_2d
4qc_snapshot QC001 qc_snapshot
Use it after basic cleanup when the main question is whether the remaining intervals are compatible with a 2-D interpretation or inversion assumption.
17.5.5.6. publication_ready#
publication_ready is the longest built-in general-purpose preset. It is
designed for polished processing output rather than quick exploration.
1notch NR001 notch_powerline
2drop_dup FREQ002 drop_duplicates
3select_band FREQ001 select_band
4align_grid FREQ004 align_grid
5correct_ss SS001 correct_ss_ama
6rotate_strike TZ001 rotate_strike
7antisymm TZ002 antisymmetrize
8mask_skew SK001 mask_by_skew
9qc_snapshot QC001 qc_snapshot
Run:
1pycsamt pipe run data/3edis \
2 --preset publication_ready \
3 --out results/publication_ready \
4 --dpi 300 \
5 --plot-fmt pdf \
6 -v
Use this preset after a lighter pass has shown that its assumptions are reasonable for the survey. The name does not make the output publication ready by itself; the review comes from inspecting the report, figures, processed EDIs, and saved pipeline snapshot.
17.5.5.7. stratagem_mt#
stratagem_mt is a normal emtools pipeline preset specialized for Stratagem
AMT data that are already loaded as a Sites object. It does not perform
raw-coordinate injection, raw hardware-file parsing, or station renaming by
itself.
1correct_ss SS001 correct_ss_ama
2select_band FREQ001 select_band band_hz=(10.0, 100000.0)
3drop_dup FREQ002 drop_duplicates
4notch NR001 notch_powerline
5hampel NR004 hampel_filter
6mask_incoher NR010 mask_incoherent
7qc_snapshot QC001 qc_snapshot
Use it with the normal pipeline API when your input is already a site collection:
1>>> from pycsamt.pipeline import Pipeline
2>>> pipe = Pipeline.from_preset("stratagem_mt")
3>>> result = pipe.run(sites, outdir="results/stratagem_mt")
Use pycsamt.pipeline.stratagem.StratagemPipeline or
run_stratagem_preset when you also need the full raw EDI plus GPS CSV
workflow.
17.5.6. Method-Aware Presets#
The seven presets above are chosen by processing intent only — the same
step sequence runs regardless of whether the survey is MT, AMT, CSAMT, or
CSUMT. In reality these EM methods are not interchangeable: CSAMT/CSUMT use
a controlled source and can suffer near-field/transition-zone contamination
that MT/AMT never see; a single-component TE- or TM-only CSAMT line cannot
produce a meaningful phase-tensor ellipse the way full-tensor MT/AMT data
can; and tipper is only sometimes recorded at all. mt_qc, amt_qc,
csamt_qc, and csumt_qc are chosen by EM method as well as intent, and
their QC steps inspect the actual data before deciding what to plot instead
of always firing the same figures.
Selection stays fully explicit — there is no automatic detection of survey
method from the data. You (or a config file, or
method if you already have one)
choose mt_qc/amt_qc/csamt_qc/csumt_qc by name, exactly the
same way you choose basic_qc vs. tensor_analysis today.
17.5.6.1. Smart QC And Preview Building Blocks#
Three new diagnostic-only steps and two new preview steps back every method-aware preset. They are plain registry steps — nothing about them is special-cased to the four presets above, so any pipeline, existing or new, can add them too.
QC005(tensor_qc_smart)Plots the phase-tensor-ellipse pseudosection only when at least one station has both off-diagonal impedance components (Zxy and Zyx) populated. A single-component TE/TM-only CSAMT line skips this plot entirely rather than drawing a degenerate ellipse.
QC006(tipper_qc_smart)Plots the induction-vector map, induction section, response-tipper panels, tipper components, and tipper hodograms only when at least one station carries a real tipper channel. Most AMT surveys have none, so these five plots are silently skipped rather than drawn empty.
QC007(strike_qc)Always plots the strike rose and the combined strike/phase-tensor/ tipper-strike analysis figure.
plot_strike_analysisis already internally tipper-aware — it draws a two-panel figure when no tipper is present and three panels when it is — so it needs no gating here.PRE001/PRE002(raw_preview/processed_preview)PRE001opens a method-aware preset and plots up to three randomly (but deterministically) chosen stations’ raw 1-D response;PRE002closes the preset and plots the same station-selection logic against the processed result, so a before/after comparison is always available without hunting through the full station list. Both steps independently re-derive the station subset with the same default seed rather than sharing state, so if an intermediate step drops one of the previewed stations, the after-plot’s reselection (drawn from the smaller surviving set) can end up choosing a different station than the before-plot — a deliberate trade-off documented inpycsamt.pipeline._previewrather than solved with a cross-step state channel.
A QC step’s plot functions are always called as fn(sites) with no way to
forward extra parameters (see Pipeline Steps), which is why every gating
decision above is made inside the plot function from the data itself
(pycsamt.pipeline._smart_qc), not from a step parameter.
17.5.6.2. Method-Aware Preset Summary#
Preset |
Steps |
Best for |
What differs from mt_qc |
|---|---|---|---|
|
11 |
Broadband MT quick-look. |
Baseline: denoise, frequency cleanup, strike rotation, smart QC, raw/processed preview. |
|
11 |
AMT quick-look. |
Trims |
|
14 |
CSAMT quick-look. |
Adds near-field/transition-zone correction ( |
|
15 |
CSUMT quick-look. |
Everything |
17.5.6.3. mt_qc#
1raw_preview PRE001 raw_preview
2notch NR001 notch_powerline
3drop_dup FREQ002 drop_duplicates
4select_band FREQ001 select_band
5align_grid FREQ004 align_grid
6rotate_strike TZ001 rotate_strike
7qc_snapshot QC001 qc_snapshot
8strike_qc QC007 strike_qc
9tensor_qc_smart QC005 tensor_qc_smart
10tipper_qc_smart QC006 tipper_qc_smart
11processed_preview PRE002 processed_preview
select_band is left at its registry default (0.001 Hz - 10 kHz),
which is already a broadband-MT-appropriate range — no method-specific
override is applied.
17.5.6.4. amt_qc#
Identical to mt_qc except select_band(FREQ001, band_hz=(10.0, 1e5))
— the exact AMT band already used by stratagem_mt, not a new number.
17.5.6.5. csamt_qc#
1raw_preview PRE001 raw_preview
2notch NR001 notch_powerline
3drop_dup FREQ002 drop_duplicates
4select_band FREQ001 select_band
5align_grid FREQ004 align_grid
6correct_near_field SRC001 correct_near_field source_offset=None
7field_zone_snapshot QC002 field_zone_snapshot
8normalize_response SRC002 normalize_response
9rotate_strike TZ001 rotate_strike
10qc_snapshot QC001 qc_snapshot
11strike_qc QC007 strike_qc
12tensor_qc_smart QC005 tensor_qc_smart
13tipper_qc_smart QC006 tipper_qc_smart
14processed_preview PRE002 processed_preview
correct_near_field runs with source_offset=None — this is not “no
correction,” it is “resolve per station from the site’s own
source_offset/offset/dist metadata, and warn instead of failing
when nothing resolves.” Against the real Tongkeng CSAMT example dataset
(data/CSAMT, whose EDI headers carry no offset metadata), a real run
warns once per station rather than raising:
1UserWarning: correct_near_field: no source offset for 'csa000'; station skipped.
2UserWarning: correct_near_field: no source offset for 'csa050'; station skipped.
and the impedance tensor for those stations passes through unchanged. When
a station’s offset is resolvable, the correction is real — dividing
\(Z_{\mathrm{obs}}\) by the complex near-field factor
\(F(p) = 1 - 3/p^2 + 3/p^3\) — not a no-op; see
pycsamt.emtools.source_effects.correct_near_field() for the full
reference. No CSAMT-specific frequency band is applied — there is no
authoritative CSAMT band constant in the codebase to draw from, so
select_band stays at its registry default and the real
method-differentiating behavior here is the near-field correction itself.
17.5.6.6. csumt_qc#
Same chain as csamt_qc, plus:
select_band(FREQ001, band_hz=(F_MIN_CSUMT, F_MAX_CSUMT))— the real 9.6 kHz - 614.4 kHz CSUMT band constants frompycsamt.emtools.csumt, not invented numbers.depth_section_snapshot(QC004) — the Bostick depth-section snapshot, CSUMT-specific and already registered.
17.5.6.7. Selecting By Method String#
get_preset_for_method() maps an explicit method
string to the matching preset, so a caller that already knows (or has a
SurveyMeta) doesn’t have to hand-build the
preset name:
1>>> from pycsamt.pipeline import get_preset_for_method
2>>> get_preset_for_method("CSAMT").name
3'csamt_qc'
4>>> get_preset_for_method("amt").name
5'amt_qc'
This performs no data inspection — method must be supplied explicitly,
the same way get_preset() requires an explicit
preset name. "MT", "BBMT", "LAMT", and "LMT" all map to
mt_qc; "CSUMT" is recognised here even though it predates
SurveyMeta’s own method vocabulary.
"CSEM"/"TEM" are recognised as valid EM methods but have no
method-aware preset yet (see below) and raise ValueError.
17.5.6.8. Not Yet Covered#
Two things are deliberately out of scope for the current method-aware presets, rather than silently missing:
A ``csem_qc`` preset. No CSEM-specific processing code exists anywhere in
pycsamt.emtoolstoday — fabricating CSEM corrections without real domain code behind them would be guessing at physics.Method-aware ``*_publication_ready`` tiers. The QC-level presets above are the first pass; fuller, publication-oriented method-aware chains are a natural follow-up once these are validated against more real surveys.
17.5.7. Export A Preset To A Config#
Preset expansion is the safest transition from exploration to reproducible work. Generate the preset once, review the explicit list, then commit the config with the survey project.
1pycsamt pipe init \
2 --preset publication_ready \
3 --name line22_publication_ready \
4 --outdir results/line22_publication_ready \
5 --output config/line22_publication_ready.yaml
Preview before running:
1pycsamt pipe show config/line22_publication_ready.yaml
2pycsamt pipe run data/3edis \
3 --config config/line22_publication_ready.yaml \
4 --dry-run
Generate Python or JSON instead:
1pycsamt pipe init --preset basic_qc --format py \
2 --output config/basic_qc.py
3pycsamt pipe init --preset basic_qc --format json \
4 --output config/basic_qc.json
17.5.8. Customizing Presets Safely#
There are three supported customization patterns.
Edit a pipeline object in Python:
1>>> from pycsamt.pipeline import Pipeline, Step
2>>> pipe = Pipeline.from_preset("basic_qc")
3>>> pipe.replace("notch", Step("NR001", mains_hz=60, n_harm=25)) is pipe
4True
5>>> pipe.append("static_shift", Step("SS001")) is pipe
6True
Expand a preset into an explicit config and edit the generated parameters:
1name: basic_qc_60hz
2output_dir: results/basic_qc_60hz
3
4steps:
5 - name: notch
6 code: NR001
7 params:
8 mains_hz: 60
9 n_harm: 25
10 tol_hz: 0.08
11 - name: drop_duplicates
12 code: FREQ002
13 - name: select_band
14 code: FREQ001
15 - name: align_grid
16 code: FREQ004
17 - name: qc_snapshot
18 code: QC001
Append extra steps after a preset in a config:
1name: basic_qc_plus_static_shift
2output_dir: results/basic_qc_plus_static_shift
3preset: basic_qc
4
5steps:
6 - name: static_shift
7 code: SS001
8 - name: final_qc
9 code: QC001
The third pattern appends. It does not edit the preset. In symbols, the loaded step list is
where \Vert means append. If you need to change NR001 from 50 Hz to
60 Hz, use an explicit expanded step list instead of preset: basic_qc plus
a second NR001 step.
17.5.9. CLI Priority#
pycsamt pipe run resolves the pipeline definition in this order:
--config FILE;--preset NAME;--steps CODE,CODE,....
If --config is supplied, --preset and --steps are ignored because
the config file is the source of truth.
1pycsamt pipe run data/3edis \
2 --config config/basic_qc.yaml \
3 --preset publication_ready
4
5pycsamt pipe run data/3edis \
6 --preset publication_ready
7
8pycsamt pipe run data/3edis \
9 --steps FREQ002,FREQ001,FREQ004,NR001,QC001
17.5.10. Compare Presets#
A preset comparison run keeps the input fixed and changes only the recipe. Let \(P_a\) and \(P_b\) be two presets. The comparison is meaningful only when both are applied to the same initial state:
Run the branches into separate output roots:
1pycsamt pipe run data/3edis \
2 --preset basic_qc \
3 --out results/compare/basic_qc
4
5pycsamt pipe run data/3edis \
6 --preset noise_reduction \
7 --out results/compare/noise_reduction
8
9pycsamt pipe run data/3edis \
10 --preset full_processing \
11 --out results/compare/full_processing
Compare summary.txt for step failures and site counts, report.html for
per-step status and embedded YAML, plots/ for visual differences,
processed/ for exported EDI differences, and pipeline.yaml for the
exact recipe behind each branch.
17.5.11. Stratagem Presets#
There are two related but different Stratagem preset systems.
stratagem_mtA normal
pycsamt.pipeline.Pipelinepreset. It expects data that can already be processed asSitesand runs normal registered pipeline steps.StratagemPresetA convenience workflow in
pycsamt.pipeline.stratagemfor raw Stratagem EDI directories plus coordinate CSV files. It callsStratagemSurveymethods such asremove_static_shift,drop_frequencies,remove_noises,export, andrename.
The Stratagem convenience presets are:
Preset |
Main workflow |
Best for |
|---|---|---|
|
Coordinate injection, AMA static shift, frequency trim, noise removal, export, rename. |
Direct replacement for the legacy Stratagem processing script. |
|
QC, AMA static shift, hardware-aware frequency filtering, smoothed noise removal, export, rename. |
Raw Stratagem workflows with hardware files and a full QC pass. |
|
Stricter QC, hardware SNR masking, AMT band trimming, stronger smoothing, export, rename. |
Polished Stratagem outputs after the basic workflow has been reviewed. |
Run a raw Stratagem convenience preset:
1>>> from pycsamt.pipeline.stratagem import run_stratagem_preset
2>>> survey = run_stratagem_preset(
3... "full_processing",
4... edi_dir="2/2EDI",
5... coord_file="2.csv",
6... raw_dir="raw/2HX",
7... outdir="results/stratagem",
8... epsg=32649,
9... utm_zone="49N",
10... rename_basename="T2.",
11... overwrite=True,
12... verbose=1,
13... )
Build a Stratagem pipeline object from a normal emtools preset:
1>>> from pycsamt.pipeline.stratagem import StratagemPipeline
2>>> pipe = StratagemPipeline.from_preset(
3... "stratagem_mt",
4... coord_file="2.csv",
5... raw_dir="raw/2HX",
6... epsg=32649,
7... utm_zone="49N",
8... rename_basename="T2.",
9... )
10>>> result = pipe.run("2/2EDI", outdir="results/stratagem_mt")
17.5.12. Troubleshooting#
- Unknown preset
Run
pycsamt pipe presets. Preset names are exact and lowercase, for examplebasic_qcorpublication_ready.- I changed
preset: basic_qcbut the notch is still 50 Hz A config
presetexpands the preset first. Explicitstepsare appended; they do not edit existing preset steps. Generate an expanded config and edit theNR001parameters directly.- The preset is too aggressive
Move to a narrower preset such as
basic_qcor export the preset to a config and remove the steps that are not justified by the data.- The preset does not include a step I need
Append the step in Python, or add it to an explicit config after generating the preset scaffold.
- I need raw Stratagem coordinate injection and renaming
Use
pycsamt.pipeline.stratagemrather than the normalstratagem_mtpreset alone.