17. Pipeline System#
The pyCSAMT pipeline system is the reproducible processing layer for MT/AMT survey workflows. It connects processing steps, presets, configuration files, command-line execution, outputs, and reports into one workflow.
- 17.1. Pipeline System
- 17.2. Pipeline Concepts
- 17.2.1. Why Pipelines Exist
- 17.2.2. Core Objects
- 17.2.3. The Mental Model
- 17.2.4. Step Registry
- 17.2.5. Configured Steps
- 17.2.6. Pipeline Structure
- 17.2.7. Building A Pipeline
- 17.2.8. Presets
- 17.2.9. Mutable Until Run
- 17.2.10. Run Lifecycle
- 17.2.11. Output Resolution
- 17.2.12. Output Directory Contract
- 17.2.13. Error Handling
- 17.2.14. Runtime Configuration
- 17.2.15. PipelineResult
- 17.2.16. StepResult
- 17.2.17. CLI And Python Equivalence
- 17.2.18. How Concepts Connect
- 17.2.19. In Short
- 17.3. Pipeline Configuration Files
- 17.3.1. When To Use A Configuration File
- 17.3.2. Basic Schema
- 17.3.3. Minimal YAML Example
- 17.3.4. Generate A Starter Config
- 17.3.5. YAML, JSON, And Python Formats
- 17.3.6. Step Codes And Labels
- 17.3.7. Discover Valid Steps
- 17.3.8. Preset Plus Extra Steps
- 17.3.9. Full Explicit Config From A Preset
- 17.3.10. Run A Config From The CLI
- 17.3.11. Export An Existing Pipeline
- 17.3.12. Validation And Failure Modes
- 17.3.13. Recommended Project Layout
- 17.3.14. Best Practices
- 17.3.15. In Short
- 17.4. Pipeline CLI
- 17.4.1. Command Overview
- 17.4.2. Start With Help
- 17.4.3. Typical First Run
- 17.4.4. Input Survey Resolution
- 17.4.5. Pipeline Definition Priority
- 17.4.6. List Presets
- 17.4.7. List And Inspect Steps
- 17.4.8. Generate Config Files
- 17.4.9. Show A Pipeline Before Running
- 17.4.10. Run Controls
- 17.4.11. Error Policy
- 17.4.12. Output Controls
- 17.4.13. Output Summary Formats
- 17.4.14. Verbose And Color Options
- 17.4.15. Exit Status
- 17.4.16. Troubleshooting
- 17.4.17. Recommended Workflow
- 17.4.18. Related Pages
- 17.5. Pipeline Presets
- 17.5.1. Preset Mental Model
- 17.5.2. Run A Preset
- 17.5.3. Built-In Preset Summary
- 17.5.4. Choosing A Preset
- 17.5.5. Preset Details
- 17.5.6. Method-Aware Presets
- 17.5.7. Export A Preset To A Config
- 17.5.8. Customizing Presets Safely
- 17.5.9. CLI Priority
- 17.5.10. Compare Presets
- 17.5.11. Stratagem Presets
- 17.5.12. Troubleshooting
- 17.5.13. Related Pages
- 17.6. Pipeline Steps
- 17.6.1. Step Contract
- 17.6.2. How Steps Execute
- 17.6.3. Discover Steps From The CLI
- 17.6.4. Discover Steps From Python
- 17.6.5. Registered Categories
- 17.6.6. Recommended Ordering
- 17.6.7. Frequency Steps
- 17.6.8. Noise Removal Steps
- 17.6.9. Static-Shift Steps
- 17.6.10. Tensor Steps
- 17.6.11. Dimensionality Steps
- 17.6.12. Skew Steps
- 17.6.13. Source-Effect Steps
- 17.6.14. QC And Diagnostic Steps
- 17.6.15. Export Steps
- 17.6.16. Parameters And Defaults
- 17.6.17. Step Labels
- 17.6.18. Diagnostic Plots
- 17.6.19. Error Handling
- 17.6.20. Choosing Between Similar Steps
- 17.6.21. Full Example Configuration
- 17.6.22. Extension Policy
- 17.6.23. Testing New Steps
- 17.6.24. Troubleshooting
- 17.6.25. Related Pages
- 17.7. Extending The Pipeline
- 17.8. Caching And Resume
- 17.9. Live Observability
- 17.10. Pipeline Outputs
- 17.10.1. Output Lifecycle
- 17.10.2. Canonical Directory Tree
- 17.10.3. Output Directory Resolution
- 17.10.4. OutputDir And Defaults
- 17.10.5. Captured Minimal Run
- 17.10.6. Processed EDI Files
- 17.10.7. QC Figures
- 17.10.8. Reports
- 17.10.9. Dashboard Report
- 17.10.10. Pipeline Snapshot
- 17.10.11. In-Memory Runs
- 17.10.12. Intermediate EDI Snapshots
- 17.10.13. PipelineResult And StepResult
- 17.10.14. Output Control Matrix
- 17.10.15. Recommended Output Layout
- 17.10.16. Comparing Two Runs
- 17.10.17. Stratagem Output Note
- 17.10.18. Troubleshooting
- 17.10.19. Related Pages