17.3. Pipeline Configuration Files#

A pipeline configuration file is the portable description of a processing pipeline. It says which registered operations should run, in which order, with which parameter override values, and where the default outputs should go. The same file can be reviewed in version control, run from the CLI, loaded from Python, and archived beside a provenance manifest before inversion.

pyCSAMT supports three equivalent configuration formats:

YAML is the recommended project format. It is readable, compact, easy to diff, and directly produced by pycsamt pipe init. JSON is useful when an external program generates the recipe. Python is useful for trusted local configuration files that need constants or small bits of logic, but it is imported and executed as Python code, so do not use it for untrusted files.

17.3.1. When To Use A Configuration File#

Use a configuration file when a run should be reproducible rather than merely convenient. A notebook cell such as Pipeline.from_preset("basic_qc").run(sites) is fine while exploring a small site collection, but it leaves too much context in memory: which output directory was chosen, which extra step was added, and which frequency limits were changed after the first attempt.

A configuration file makes those choices explicit. It is especially helpful when several surveys should share the same chain, when another geophysicist must review the processing before inversion, or when the command line and Python API need to execute the same workflow. Treat the file as the recipe for the data product; raw EDI files remain unchanged, and processed products are written under a results directory.

17.3.2. Basic Schema#

The top-level object is a mapping:

Key

Required

Meaning

name

No

Human-readable pipeline name used in reports and printed summaries. It defaults to "unnamed".

output_dir

No

Default output directory used when Pipeline.run is called without an explicit outdir.

preset

No

Built-in pipeline preset used to seed the pipeline before the explicit steps list is appended.

steps

No

Ordered list of step entries. Each entry identifies one registered operation and optional parameter overrides.

Each item in steps is also a mapping:

Key

Required

Meaning

code

Recommended

Pipeline step code such as "NR001" or registry name such as "notch_powerline". The loader can fall back to name as an identifier, but that makes labels ambiguous.

name

No

Step label for this occurrence of the operation. Labels appear in reports and can be used by CLI slicing options such as --from-step.

params

No

Keyword arguments passed to the step. These override defaults from the step registry.

The loader can be understood as a small deterministic transformation. Let S_p be the ordered step list supplied by a preset, and let S_c be the ordered step list written explicitly in the configuration file. The final pipeline order is

\[\begin{split}S_{\mathrm{final}} = \begin{cases} S_p \Vert S_c, & \text{if a preset is given},\\ S_c, & \text{otherwise}, \end{cases}\end{split}\]

where \Vert means append. A preset is therefore a seed, not a template that can be edited in place. If you need to change a parameter inside a preset step, write the full step list explicitly.

For one explicit step j, the identifier, label, and parameters are resolved as

\[\begin{split}i_j &= \mathrm{code}_j \;\text{if present, otherwise}\; \mathrm{name}_j,\\ \ell_j &= \mathrm{name}_j \;\text{if present, otherwise}\; \mathrm{registry\_name}(i_j),\\ \theta_j &= \theta_{\mathrm{default}}(i_j) \cup \theta_{\mathrm{params},j}.\end{split}\]

The last line means that values in params replace registry defaults for the same keys, while omitted keys keep their defaults. This makes short configs possible, but it also means reviewers should know which defaults were in effect for the installed pyCSAMT version.

17.3.3. Minimal YAML Example#

This is a complete YAML pipeline:

 1name: first_qc
 2output_dir: results/first_qc
 3
 4steps:
 5  - name: notch
 6    code: NR001
 7    params:
 8      mains_hz: 50.0
 9      n_harm: 30
10      tol_hz: 0.08
11
12  - name: drop_duplicates
13    code: FREQ002
14
15  - name: select_band
16    code: FREQ001
17    params:
18      band_hz: [0.001, 10000.0]
19
20  - name: align_grid
21    code: FREQ004
22
23  - name: qc_snapshot
24    code: QC001

Load it from Python and inspect the resolved pipeline before running it:

1>>> from pycsamt.pipeline import Pipeline
2>>> pipe = Pipeline.from_yaml("config/first_qc.yaml")
3>>> pipe.name
4'first_qc'
5>>> [label for label, step in pipe]
6['notch', 'drop_duplicates', 'select_band', 'align_grid', 'qc_snapshot']
7>>> "output_dir: results/first_qc" in pipe.to_yaml_string()
8True

Then run the same file against a survey:

1>>> from pycsamt.api import read_edis
2>>> survey = read_edis("data/3edis", strict=False)
3>>> result = pipe.run(survey.to_collection())
4>>> print(result.summary())
5Pipeline run: first_qc
6Steps: 5
7Output directory: results/first_qc
8...

If outdir is passed to Pipeline.run, it takes precedence over the file’s output_dir:

1>>> result = pipe.run(
2...     survey.to_collection(),
3...     outdir="results/first_qc_experiment",
4... )
5>>> result.outdir
6'results/first_qc_experiment'

In words, the effective output directory is chosen in this order:

\[d_{\mathrm{effective}} = d_{\mathrm{run}} \rightarrow d_{\mathrm{config}} \rightarrow d_{\mathrm{default}},\]

where the first available value wins. This is useful for controlled experiments: the configuration still describes the processing chain, while the runtime call can place trial outputs in a separate directory.

17.3.4. Generate A Starter Config#

The easiest way to create a valid file is the CLI scaffold command:

1pycsamt pipe init \
2    --preset basic_qc \
3    --name first_qc \
4    --outdir results/first_qc \
5    --output config/first_qc.yaml

Print the scaffold without writing a file:

1pycsamt pipe init --preset basic_qc --name first_qc \
2    --outdir results/first_qc --print

Captured output excerpt:

 1# pyCSAMT Pipeline Configuration
 2# Generated by: Pipeline.scaffold("first_qc.yaml")
 3
 4name: first_qc
 5output_dir: results/first_qc
 6
 7steps:
 8  - {name: select_band, code: FREQ001,
 9     params: {band_hz: [0.001, 10000.0]}}
10  - {name: drop_duplicates, code: FREQ002}
11  - {name: align_grid, code: FREQ004}
12  - {name: notch_powerline, code: NR001,
13     params: {mains_hz: 50, n_harm: 30, tol_hz: 0.08}}
14  - {name: qc_snapshot, code: QC001}

The full scaffold also includes commented inactive steps. Keep the active steps you want, remove those you do not want, rename labels to match the survey, and adjust params after checking the step information.

Generate Python or JSON instead of YAML:

1pycsamt pipe init --format py --preset basic_qc -o config/first_qc.py
2pycsamt pipe init --format json --preset basic_qc -o config/first_qc.json

17.3.5. YAML, JSON, And Python Formats#

All formats express the same logical object. YAML is concise:

1name: amt_line_22
2output_dir: results/line_22
3steps:
4  - {name: notch, code: NR001, params: {mains_hz: 50.0}}
5  - {name: select_band, code: FREQ001, params: {band_hz: [10.0, 100000.0]}}
6  - {name: qc, code: QC001}

JSON is better when another program writes the file:

 1{
 2  "name": "amt_line_22",
 3  "output_dir": "results/line_22",
 4  "steps": [
 5    {"name": "notch", "code": "NR001", "params": {"mains_hz": 50.0}},
 6    {
 7      "name": "select_band",
 8      "code": "FREQ001",
 9      "params": {"band_hz": [10.0, 100000.0]}
10    },
11    {"name": "qc", "code": "QC001"}
12  ]
13}

Python config files must define a module-level pipeline_config dictionary:

 1>>> AMT_BAND_HZ = (10.0, 100000.0)
 2>>> pipeline_config = dict(
 3...     name="amt_line_22",
 4...     output_dir="results/line_22",
 5...     steps=[
 6...         dict(name="notch", code="NR001", params=dict(mains_hz=50.0)),
 7...         dict(
 8...             name="select_band",
 9...             code="FREQ001",
10...             params=dict(band_hz=AMT_BAND_HZ),
11...         ),
12...         dict(name="qc", code="QC001"),
13...     ],
14... )
15>>> pipeline_config["steps"][1]["params"]["band_hz"]
16(10.0, 100000.0)

Save those lines in config/line_22.py and load them with:

1>>> from pycsamt.pipeline import Pipeline
2>>> pipe = Pipeline.from_py("config/line_22.py")
3>>> pipe.name
4'amt_line_22'

17.3.6. Step Codes And Labels#

Every operation is resolved through the step registry. A config step can use the short pipeline step code:

1- name: notch
2  code: NR001

or the registry name:

1- name: notch
2  code: notch_powerline

The code form is compact and stable in reports. The name field is not the registry name; it is the step label for this occurrence of the step. Use labels that describe the role of the operation in this workflow:

1- name: remove_powerline
2  code: NR001
3- name: trim_to_amt_band
4  code: FREQ001

Labels are useful when slicing a run from the CLI:

1pycsamt pipe run data/3edis --config config/line_22.yaml \
2    --from-step trim_to_amt_band

17.3.7. Discover Valid Steps#

Use the CLI when working in a terminal:

1pycsamt pipe steps
2pycsamt pipe steps --category frequency
3pycsamt pipe steps --info NR001
4pycsamt pipe steps --codes-only

Captured output for the frequency codes:

1FREQ001
2FREQ002
3FREQ003
4FREQ004
5FREQ005
6FREQ006
7FREQ007
8FREQ008
9FREQ009

Or inspect the same registry from Python:

1>>> from pycsamt.pipeline import Pipeline
2>>> "FREQ001" in Pipeline.catalogue("frequency")
3True
4>>> info = Pipeline.step_info("NR001")
5>>> "notch" in info.lower()
6True

Step defaults are merged with params. For example, if NR001 has defaults for mains_hz, n_harm, and tol_hz, this config overrides only mains_hz and keeps the remaining defaults:

1- name: notch_60hz
2  code: NR001
3  params:
4    mains_hz: 60.0

That compact form is helpful, but for publication or hand-off work it is often better to record every important parameter explicitly. A future reader should not have to guess whether a value came from the file or from the installed registry defaults.

17.3.8. Preset Plus Extra Steps#

A config may combine a preset with additional explicit steps:

1name: publication_with_extra_qc
2output_dir: results/publication_with_extra_qc
3preset: publication_ready
4
5steps:
6  - name: final_frequency_confidence
7    code: QC001

pyCSAMT loads publication_ready first and then appends final_frequency_confidence. It does not replace, remove, or mutate a step inside the preset. This pattern is good for a known baseline plus extra diagnostics. It is not good for changing a preset parameter, because the result would contain the original preset step and your additional step.

17.3.9. Full Explicit Config From A Preset#

If you want basic_qc with one changed parameter, prefer an explicit file:

 1name: basic_qc_60hz
 2output_dir: results/basic_qc_60hz
 3
 4steps:
 5  - name: select_band
 6    code: FREQ001
 7    params:
 8      band_hz: [0.001, 10000.0]
 9  - name: drop_duplicates
10    code: FREQ002
11  - name: align_grid
12    code: FREQ004
13  - name: notch_powerline
14    code: NR001
15    params:
16      mains_hz: 60.0
17      n_harm: 30
18      tol_hz: 0.08
19  - name: qc_snapshot
20    code: QC001

This is longer than preset: basic_qc, but it is unambiguous. The reviewer can see the exact sequence:

\[\mathrm{select\_band} \rightarrow \mathrm{drop\_duplicates} \rightarrow \mathrm{align\_grid} \rightarrow \mathrm{notch\_powerline} \rightarrow \mathrm{qc\_snapshot}.\]

The same notation is useful when comparing two processing branches. If the only difference is mains_hz = 50 versus mains_hz = 60, the config file shows that the experiment changed the notch target rather than the frequency grid, QC logic, or output path.

17.3.10. Run A Config From The CLI#

Run a config against an explicit EDI directory:

1pycsamt pipe run data/3edis \
2    --config config/first_qc.yaml \
3    --out results/first_qc_run \
4    --on-error warn \
5    --dpi 200 \
6    --plot-fmt png

Dry-run before a long processing job:

1pycsamt pipe run data/3edis \
2    --config config/first_qc.yaml \
3    --dry-run

The CLI chooses the pipeline definition in this priority order:

  1. --config;

  2. --preset;

  3. --steps.

If --config is provided, --preset and --steps are ignored because the file is the source of truth for the processing chain. Runtime options such as --out, --dpi, --plot-fmt, and --on-error still control how that chain is executed and written.

17.3.11. Export An Existing Pipeline#

You can build or modify a pipeline in Python and export a canonical pipeline snapshot:

1>>> from pycsamt.pipeline import Pipeline
2>>> pipe = Pipeline.from_preset("basic_qc", pipeline_name="first_qc")
3>>> pipe.to_yaml("config/first_qc.yaml")
4>>> pipe.to_json("config/first_qc.json")
5>>> pipe.to_py("config/first_qc.py")

The YAML and JSON exports are useful for reproducibility because they are data files. The Python export is useful when you want an editable script-style config with comments. Archive at least one exported snapshot beside processed EDI files, plots, and reports so a later user can reconstruct the chain that produced them.

17.3.12. Validation And Failure Modes#

pyCSAMT validates configuration files when they are loaded. Common failures include:

Top level is not a mapping

YAML must load to a mapping and JSON must load to an object. A top-level list is invalid because it cannot carry the pipeline name, output directory, or preset.

Python file has no pipeline_config

Pipeline.from_py imports the file and looks for a module-level variable named pipeline_config.

Step entry has no usable identifier

Every explicit step entry must identify a registry step. Write code in normal configs. Falling back to name is supported, but it prevents name from being a clear label.

Unknown step code

The code or registry name does not exist. Run pycsamt pipe steps or Pipeline.step_info(...) to confirm the identifier.

Unknown preset

The value under preset is not registered. Run pycsamt pipe presets to list available presets.

Parameter name is wrong

The config may load, but the step can fail at runtime if a parameter is not accepted by the underlying function. Check pycsamt pipe steps --info CODE and run with --dry-run before processing the full survey.

Output directory is ambiguous

Prefer survey-specific directories such as results/line_22/basic_qc. Reusing a shared directory makes it harder to know which config produced which plots, processed EDI files, and reports.

17.3.14. Best Practices#

  • Commit YAML configuration files with the project when possible.

  • Use Python configs only for trusted local logic.

  • Give every explicit step a meaningful name label.

  • Prefer explicit step lists when changing preset parameters.

  • Record important parameter values instead of relying silently on defaults.

  • Keep output directories survey-specific.

  • Run pycsamt pipe run ... --dry-run before long jobs.

  • Store raw data and processed outputs in separate directories.

  • Archive the config file used to prepare inversion inputs.

17.3.15. In Short#

A pyCSAMT pipeline config is an ordered, reproducible processing recipe:

1name: first_qc
2output_dir: results/first_qc
3steps:
4  - {name: notch, code: NR001}
5  - {name: drop_duplicates, code: FREQ002}
6  - {name: select_band, code: FREQ001,
7     params: {band_hz: [0.001, 10000.0]}}
8  - {name: qc_snapshot, code: QC001}

Load it with Pipeline.from_yaml or run it with pycsamt pipe run --config. Use pycsamt pipe init when you want a valid starter file, and export a canonical pipeline snapshot when a constructed pipeline should become part of the permanent processing record.