Pipeline System#

The pyCSAMT pipeline system is the reproducible processing layer for MT/AMT survey workflows. It connects registered processing steps, named presets, configuration files, command-line execution, output directories, reports, and Python result objects into one coherent workflow.

Use the pipeline when you want to:

  • run the same processing chain on multiple surveys;

  • inspect and compare quality-control figures after each step;

  • save processed EDI files without modifying raw inputs;

  • keep an exact pipeline.yaml snapshot for reproducibility;

  • move between the Python API and the pycsamt pipe CLI;

  • standardize processing recipes for a project, publication, or team.

The pipeline API follows a scikit-learn-style shape: a pipeline is an ordered sequence of labelled steps, and each step is a configured operation from the registry.

Minimal Example#

Run a built-in preset from the command line:

1pycsamt pipe run data/edis \
2    --preset basic_qc \
3    --out results/basic_qc \
4    --on-error warn \
5    -v

The same idea from Python:

1from pycsamt.pipeline import Pipeline
2
3pipe = Pipeline.from_preset("basic_qc")
4result = pipe.run(sites, outdir="results/basic_qc")
5
6print(result.summary())

A typical output directory contains:

1results/basic_qc/
2|-- processed/
3|-- plots/
4|-- pipeline.yaml
5|-- report.html
6`-- summary.txt

Documentation Map#

Page

What it explains

Read this when

Pipeline Concepts

The pipeline object model, run lifecycle, runtime configuration, PipelineResult, and CLI/Python equivalence.

You want the mental model before writing workflows.

Pipeline Configuration Files

YAML, JSON, and Python config files; preset expansion; explicit step lists; output directories; project organization.

You want reproducible processing recipes.

Pipeline CLI

pycsamt pipe presets, steps, init, show, and run.

You process surveys from the terminal or automation scripts.

Pipeline Presets

Built-in recipes such as basic_qc, noise_reduction, full_processing, publication_ready, and stratagem_mt.

You want a safe starting workflow.

Pipeline Steps

All registered step codes, categories, defaults, ordering guidance, QC plots, and extension policy.

You need to choose, tune, or add processing operations.

Pipeline Outputs

Processed EDIs, plots, reports, pipeline.yaml, in-memory runs, PipelineResult, and output troubleshooting.

You need to understand what a run writes and how to inspect it.

Core Concepts At A Glance#

Pipeline

Ordered sequence of labelled Step objects. It can be built directly in Python, from a preset, from a config file, or from the CLI.

Step

Configured processing operation. It references a registry code such as NR001 or a snake-case name such as notch_powerline and carries parameter overrides.

StepSpec

Registry metadata for a step: code, name, label, category, defaults, transform function, QC functions, and whether the step returns modified sites.

Preset

Named ordered step list for common workflows. Presets are convenient starting points, not hidden black boxes.

PipelineResult

Programmatic record returned by Pipeline.run. It contains input and output sites, per-step results, plot paths, processed EDI paths, status, errors, runtime, and output directory.

pipeline.yaml

Saved snapshot of the resolved pipeline. It is written into every output-enabled run and can be reloaded for reproduction.

Common Workflows#

First-pass QC:

1pycsamt pipe run data/edis \
2    --preset basic_qc \
3    --out results/basic_qc

Generate an editable config from a preset:

1pycsamt pipe init \
2    --preset publication_ready \
3    --name line22_publication_ready \
4    --outdir results/line22_publication_ready \
5    --output config/line22_publication_ready.yaml

Preview before processing:

1pycsamt pipe show config/line22_publication_ready.yaml
2pycsamt pipe run data/edis \
3    --config config/line22_publication_ready.yaml \
4    --dry-run

Run a reviewed config:

1pycsamt pipe run data/edis \
2    --config config/line22_publication_ready.yaml \
3    --out results/line22_publication_ready \
4    --on-error warn \
5    -v

Run in memory from Python:

1result = pipe.run(
2    sites,
3    outdir=None,
4    save_plots=False,
5    save_edis=False,
6    save_report=False,
7)

Pipeline Design Rules#

Keep raw and processed data separate

Raw EDI directories should remain unchanged. Write processed EDIs, plots, reports, and snapshots to a dedicated results/ tree.

Prefer configs for repeatable work

Direct presets are useful for exploration. Reviewed survey processing should usually live in a YAML, JSON, or Python config file.

Use presets as starting points

If a preset needs changed parameters, export it to an explicit config rather than appending duplicate steps that look similar but do not modify the preset step in place.

Inspect before running

Use pycsamt pipe show and --dry-run before expensive processing or before writing large output directories.

Name step labels carefully

Labels appear in reports, plot folders, and slicing options such as --from-step and --until-step.

Archive pipeline.yaml

The saved pipeline snapshot is the processing recipe that connects raw inputs, outputs, figures, and reports.

Typical Project Layout#

 1project/
 2|-- data/
 3|   `-- raw_edis/
 4|-- config/
 5|   |-- basic_qc.yaml
 6|   `-- publication_ready.yaml
 7`-- results/
 8    |-- basic_qc/
 9    |-- noise_reduction/
10    `-- publication_ready/

This layout makes it easy to compare runs without mixing raw observations, configuration files, and generated outputs.

Next Steps#

If you are learning the system, continue with Pipeline Concepts. If you already want to run data, jump to Pipeline CLI. If you are ready to write project recipes, start with Pipeline Configuration Files.