Pipeline Configuration Files#

Pipeline configuration files make processing workflows reproducible. Instead of rebuilding a pycsamt.pipeline.Pipeline in a notebook every time, you can store the pipeline name, default output directory, preset seed, step order, and step parameters in a small file.

pyCSAMT supports three configuration formats:

All three formats use the same logical schema. YAML is the recommended default for most survey projects because it is readable, easy to review in version control, and directly supported by the CLI scaffold command.

When To Use A Configuration File#

Use a pipeline configuration file when:

  • a processing run must be repeated later;

  • several surveys should use the same processing chain;

  • a workflow needs review by another developer or geophysicist;

  • you want the command line and Python API to run the same steps;

  • you need a permanent record of parameters used before inversion;

  • you are preparing examples, tutorials, tests, or reports.

For quick exploration, Pipeline.from_preset("basic_qc") is fine. For survey processing, inversion preparation, or publication output, write the workflow to a configuration file.

Basic Schema#

The top-level configuration is a mapping with these keys:

Key

Required

Meaning

name

No

Human-readable pipeline name used in reports and printed summaries. Defaults to "unnamed" when omitted.

output_dir

No

Default output directory used when Pipeline.run is called without an explicit outdir.

preset

No

Built-in preset name used to seed the pipeline before explicit steps are appended.

steps

No

Ordered list of step entries. Each entry identifies a registered pipeline step and optional parameter overrides.

Each item in steps is a mapping:

Key

Required

Meaning

code

Recommended

Step registry code such as "NR001" or registry name such as "notch_powerline". The loader can fall back to name as the identifier, but the code form is clearer and is recommended for configs.

name

No

User label for this occurrence of the step. Labels appear in reports and can be used by CLI slicing options such as --from-step.

params

No

Keyword arguments passed to the step. These override registry defaults.

Minimal YAML Example#

This is a complete YAML pipeline:

 1name: first_qc
 2output_dir: results/first_qc
 3
 4steps:
 5  - name: notch
 6    code: NR001
 7    params:
 8      mains_hz: 50.0
 9      n_harm: 30
10      tol_hz: 0.08
11
12  - name: drop_duplicates
13    code: FREQ002
14
15  - name: select_band
16    code: FREQ001
17    params:
18      band_hz: [0.001, 10000.0]
19
20  - name: align_grid
21    code: FREQ004
22
23  - name: qc_snapshot
24    code: QC001

Load and run it from Python:

1from pycsamt.api import read_edis
2from pycsamt.pipeline import Pipeline
3
4survey = read_edis("data/edis", strict=False)
5pipe = Pipeline.from_yaml("config/first_qc.yaml")
6
7print(pipe)
8result = pipe.run(survey.to_collection())
9print(result.summary())

Because the file defines output_dir, the run writes to results/first_qc unless you override it:

1result = pipe.run(
2    survey.to_collection(),
3    outdir="results/first_qc_experiment",
4)

The explicit outdir passed to Pipeline.run wins over the file’s output_dir.

Generate A Starter Config#

The easiest way to create a valid file is the CLI scaffold command:

1pycsamt pipe init \
2    --preset basic_qc \
3    --name first_qc \
4    --outdir results/first_qc \
5    --output config/first_qc.yaml

Print the scaffold without writing a file:

1pycsamt pipe init --preset full_processing --print

Generate Python or JSON instead of YAML:

1pycsamt pipe init --format py --preset basic_qc -o config/first_qc.py
2pycsamt pipe init --format json --preset basic_qc -o config/first_qc.json

The scaffold includes active steps from the chosen preset and comments for other registered steps. Treat it as a starting point: remove steps you do not want, rename labels, and adjust params for the survey.

YAML, JSON, And Python Formats#

YAML is the most convenient hand-edited format:

1name: amt_line_22
2output_dir: results/line_22
3steps:
4  - {name: notch, code: NR001, params: {mains_hz: 50.0}}
5  - {name: select_band, code: FREQ001, params: {band_hz: [10.0, 100000.0]}}
6  - {name: qc, code: QC001}

JSON is useful for generated configs or external tooling:

 1{
 2  "name": "amt_line_22",
 3  "output_dir": "results/line_22",
 4  "steps": [
 5    {
 6      "name": "notch",
 7      "code": "NR001",
 8      "params": {
 9        "mains_hz": 50.0
10      }
11    },
12    {
13      "name": "select_band",
14      "code": "FREQ001",
15      "params": {
16        "band_hz": [10.0, 100000.0]
17      }
18    },
19    {
20      "name": "qc",
21      "code": "QC001"
22    }
23  ]
24}

Python config files are useful when you want comments, constants, or a small amount of local logic. The file must define a module-level pipeline_config dictionary:

 1AMT_BAND_HZ = (10.0, 100000.0)
 2
 3pipeline_config = dict(
 4    name="amt_line_22",
 5    output_dir="results/line_22",
 6    steps=[
 7        dict(name="notch", code="NR001", params=dict(mains_hz=50.0)),
 8        dict(name="select_band", code="FREQ001",
 9             params=dict(band_hz=AMT_BAND_HZ)),
10        dict(name="qc", code="QC001"),
11    ],
12)

Load it with:

1from pycsamt.pipeline import Pipeline
2
3pipe = Pipeline.from_py("config/line_22.py")

Use Python configs carefully. They are imported and executed as Python code, so they are best kept inside trusted project repositories.

Step Codes And Labels#

Every step is resolved through the pipeline registry. A config step can use the short registry code:

1- name: notch
2  code: NR001

or the registry name:

1- name: notch
2  code: notch_powerline

The code form is more compact and stable in reports. The name field is not the registry name; it is the label for this occurrence of the step. Use short labels that describe the role of the step in this workflow:

1- name: remove_powerline
2  code: NR001
3- name: trim_to_amt_band
4  code: FREQ001

Labels are useful when slicing a run from the CLI:

1pycsamt pipe run data/edis --config config/line_22.yaml \
2    --from-step trim_to_amt_band

Discover Valid Steps#

Use the CLI:

1pycsamt pipe steps
2pycsamt pipe steps --category frequency
3pycsamt pipe steps --info NR001
4pycsamt pipe steps --codes-only

Or use Python:

1from pycsamt.pipeline import Pipeline
2
3print(Pipeline.catalogue())
4print(Pipeline.catalogue("frequency"))
5print(Pipeline.step_info("NR001"))

Step defaults are merged with your params. For example, if NR001 has defaults for mains_hz, n_harm, and tol_hz, this config overrides only mains_hz and keeps the other defaults:

1- name: notch_60hz
2  code: NR001
3  params:
4    mains_hz: 60.0

Preset Plus Extra Steps#

A config may include preset and explicit steps:

1name: publication_with_extra_qc
2output_dir: results/publication_with_extra_qc
3preset: publication_ready
4
5steps:
6  - name: final_frequency_confidence
7    code: QC001

When preset is present, pyCSAMT loads the preset first, then appends the explicit steps list. It does not replace or edit steps inside the preset.

Use this pattern when you want a known baseline plus extra diagnostics. Do not use it when you need to change a preset step parameter; in that case, write the full step list explicitly so the final order and parameters are obvious.

Full Explicit Config From A Preset#

If you want basic_qc with one changed parameter, prefer an explicit file:

 1name: basic_qc_60hz
 2output_dir: results/basic_qc_60hz
 3
 4steps:
 5  - name: notch
 6    code: NR001
 7    params:
 8      mains_hz: 60.0
 9      n_harm: 30
10      tol_hz: 0.08
11  - name: drop_duplicates
12    code: FREQ002
13  - name: select_band
14    code: FREQ001
15    params:
16      band_hz: [0.001, 10000.0]
17  - name: align_grid
18    code: FREQ004
19  - name: qc_snapshot
20    code: QC001

This is longer than preset: basic_qc, but it is unambiguous and easy to review before processing field data.

Run A Config From The CLI#

Run a config against an explicit EDI directory:

1pycsamt pipe run data/edis \
2    --config config/first_qc.yaml \
3    --out results/first_qc_run \
4    --on-error warn \
5    --dpi 200 \
6    --plot-fmt png

Dry-run before a long processing job:

1pycsamt pipe run data/edis \
2    --config config/first_qc.yaml \
3    --dry-run

The pipeline definition priority in the CLI is:

  1. --config;

  2. --preset;

  3. --steps.

If --config is provided, --preset and --steps are ignored because the file is the source of truth for the pipeline.

Export An Existing Pipeline#

You can build or modify a pipeline in Python and export it:

1from pycsamt.pipeline import Pipeline
2
3pipe = Pipeline.from_preset("basic_qc", pipeline_name="first_qc")
4pipe.to_yaml("config/first_qc.yaml")
5pipe.to_json("config/first_qc.json")
6pipe.to_py("config/first_qc.py")

The YAML and JSON exports are useful for reproducibility. The Python export is useful when you want an editable script-style config with comments.

Validation And Failure Modes#

pyCSAMT validates configuration files when they are loaded. Common failures include:

Top level is not a mapping

YAML must load to a mapping and JSON must load to an object. A top-level list is invalid.

Python file has no pipeline_config

Pipeline.from_py imports the file and looks for a module-level variable named pipeline_config.

Step entry has no code

Every explicit step entry must identify a registry step. If code is missing, the loader falls back to name as the step identifier, but this makes labels ambiguous. Prefer always writing code.

Unknown step code

The code or registry name does not exist. Run pycsamt pipe steps or Pipeline.step_info(...) to confirm the identifier.

Unknown preset

The value under preset is not registered. Run pycsamt pipe presets to list available presets.

Parameter name is wrong

The config may load, but the step can fail at runtime if a parameter is not accepted by the underlying function. Check pycsamt pipe steps --info CODE and run with --dry-run before processing the full survey.

Best Practices#

  • Commit pipeline config files with the project when possible.

  • Use YAML for shared survey workflows.

  • Use Python configs only for trusted local logic.

  • Give every step a meaningful name label.

  • Prefer explicit step lists when changing preset parameters.

  • Keep output directories survey-specific.

  • Run pycsamt pipe run ... --dry-run before long jobs.

  • Store raw data and processed outputs in separate directories.

  • Record the config file used to prepare inversion inputs.

In Short#

A pyCSAMT pipeline config is an ordered, reproducible processing recipe:

1name: first_qc
2output_dir: results/first_qc
3steps:
4  - {name: notch, code: NR001}
5  - {name: drop_duplicates, code: FREQ002}
6  - {name: select_band, code: FREQ001,
7     params: {band_hz: [0.001, 10000.0]}}
8  - {name: qc_snapshot, code: QC001}

Load it with Pipeline.from_yaml or run it with pycsamt pipe run --config. Use pycsamt pipe init when you want a valid starter file instead of writing the schema by hand.