Generic Edge QC#
Generic edge QC is the lightweight screening step that can run on a field node before data are transmitted or stored. It is not a replacement for AMT/CSAMT processing. It answers simpler operational questions: is the window finite, are there obvious spikes, how many samples would be emitted after decimation, and should the packet be accepted, warned, or rejected?
The examples below use synthetic four-channel AMT-style windows. That is
appropriate here because generic edge QC works on live time-series arrays,
not on EDI files. Three windows are created: a clean window, a marginal
window with missing Ey samples, and a bad window with missing electric
and magnetic samples plus spikes.
Build Synthetic Edge Windows#
Each row is a time sample and each column is a channel. The channel order
is Ex, Ey, Hx, Hy.
1import numpy as np
2
3sample_rate = 512.0
4n_samples = 2048
5t = np.arange(n_samples) / sample_rate
6rng = np.random.default_rng(31)
7channels = ["ex", "ey", "hx", "hy"]
8
9clean = np.column_stack(
10 [
11 np.sin(2 * np.pi * 7.0 * t)
12 + 0.03 * rng.standard_normal(n_samples),
13 0.8 * np.sin(2 * np.pi * 9.0 * t + 0.2)
14 + 0.03 * rng.standard_normal(n_samples),
15 0.3 * np.sin(2 * np.pi * 7.0 * t + 0.5)
16 + 0.02 * rng.standard_normal(n_samples),
17 0.35 * np.sin(2 * np.pi * 9.0 * t + 0.8)
18 + 0.02 * rng.standard_normal(n_samples),
19 ]
20)
21
22marginal = clean.copy()
23marginal[100:500, 1] = np.nan
24marginal[800:812, 0] += 4.0
25
26bad = clean.copy()
27bad[200:900, 0] = np.nan
28bad[260:880, 1] = np.nan
29bad[320:820, 2] = np.nan
30bad[::12, 3] += 10.0
Configure And Run The Processor#
The processor decimates every fourth sample, checks finite coverage,
computes per-channel summaries, and estimates a robust spike fraction. A
window below finite_threshold is rejected. A window above the hard
threshold but below warn_finite_threshold is accepted with a warning.
1from pycsamt.iot import EdgeProcessingConfig, EdgeProcessor
2
3processor = EdgeProcessor(
4 EdgeProcessingConfig(
5 decimation=4,
6 finite_threshold=0.85,
7 warn_finite_threshold=0.97,
8 channel_names=channels,
9 spike_threshold=5.0,
10 max_spike_fraction=0.05,
11 warn_spike_fraction=0.01,
12 )
13)
14
15results = [
16 processor.process(clean),
17 processor.process(marginal),
18 processor.process(bad),
19]
20
21for label, result in zip(["clean", "marginal", "bad"], results):
22 warnings = result.metrics.get("warnings", "")
23 reasons = ";".join(result.reasons) or "-"
24 print(
25 f"{label}: decision={result.decision.value}, "
26 f"accepted={result.accepted}, "
27 f"finite={result.metrics['finite_coverage']:.3f}, "
28 f"max_spike={result.metrics['spike_fraction_max']:.3f}, "
29 f"warnings={warnings or '-'}, reasons={reasons}"
30 )
Output:
clean: decision=accept, accepted=True, finite=1.000, max_spike=0.000, warnings=-, reasons=-
marginal: decision=warning, accepted=True, finite=0.951, max_spike=0.000, warnings=finite_coverage_marginal, reasons=-
bad: decision=reject, accepted=False, finite=0.778, max_spike=0.334, warnings=-, reasons=finite_coverage_below_threshold;spike_fraction_above_threshold
Inspect Per-Channel QC#
Use pycsamt.iot.edge_summary_table() to inspect one row per channel.
The result index identifies which edge window produced the channel row:
0 is clean, 1 is marginal, and 2 is bad.
1from pycsamt.iot import edge_summary_table
2
3summary = edge_summary_table(results)
4print(
5 summary[
6 [
7 "result_index",
8 "channel",
9 "finite_coverage",
10 "spike_fraction",
11 "accepted",
12 "decision",
13 "reasons",
14 ]
15 ].to_string(index=False)
16)
Output:
result_index channel finite_coverage spike_fraction accepted decision reasons
0 ex 1.000000 0.000000 True accept
0 ey 1.000000 0.000000 True accept
0 hx 1.000000 0.000000 True accept
0 hy 1.000000 0.000000 True accept
1 ex 1.000000 0.000000 True warning
1 ey 0.804688 0.000000 False warning finite_coverage_below_threshold
1 hx 1.000000 0.000000 True warning
1 hy 1.000000 0.000000 True warning
2 ex 0.658203 0.000000 False reject finite_coverage_below_threshold
2 ey 0.697266 0.000000 False reject finite_coverage_below_threshold
2 hx 0.755859 0.000000 False reject finite_coverage_below_threshold
2 hy 1.000000 0.333984 False reject spike_fraction_above_threshold
Encode QC As Telemetry#
An EdgeProcessingResult can be converted directly to
a qc packet. The packet stores the compact metrics and all channel
summaries, so a downstream monitor can audit the edge decision without
shipping the full waveform.
1from pycsamt.iot import DeviceConfig, FieldSession
2
3device = DeviceConfig(
4 "l18-node-01",
5 station="001A",
6 protocol="file",
7 sample_rate_hz=sample_rate,
8 channels=channels,
9 role="recorder",
10)
11session = FieldSession("WILLY-L18-EDGE-QC", devices=[device], method="amt")
12
13base_time = 1_700_000_000.0
14for index, result in enumerate(results):
15 packet = result.to_packet(
16 device,
17 timestamp=base_time + 60.0 * index,
18 survey_id=session.survey_id,
19 )
20 packet.payload.update(
21 {
22 "method": "amt",
23 "station": "001A",
24 "channels": channels,
25 "frequency_band_hz": [1.0, 1000.0],
26 }
27 )
28 session.add_packet(packet)
29
30packet = session.packets[1]
31print(f"topic: {packet.topic}")
32print(f"decision: {packet.payload['decision']}")
33print(f"n channel summaries: {len(packet.payload['channels'])}")
34print(f"payload keys: {', '.join(sorted(packet.payload))}")
Output:
topic: pycsamt/WILLY-L18-EDGE-QC/001A/l18-node-01/qc
decision: warning
n channel summaries: 4
payload keys: accepted, channels, decision, frequency_band_hz, method, metrics, reasons, station
Plot The QC Summary#
The edge-QC plot shows the decision counts, finite coverage, spike fraction, and reasons/warnings. This page has one combined diagnostic figure; pages with multiple separate figures use an RST grid.
1from pathlib import Path
2
3from pycsamt.iot import plot_edge_qc_summary
4
5out_dir = Path("docs/source/images/user_guide/iot")
6out_dir.mkdir(parents=True, exist_ok=True)
7
8plot_edge_qc_summary(
9 results,
10 figsize=(10.8, 7.2),
11 title="Synthetic edge QC windows",
12 output_path=(out_dir / "user-guide-iot-edge-qc-01.png").as_posix(),
13 close=True,
14)
Field Interpretation#
The clean window can be transmitted or stored as accepted telemetry. The
marginal window is still usable at the window level, but the Ey channel
coverage is low enough to keep the warning in the audit trail. The bad
window fails both global finite coverage and spike-fraction thresholds, so
it should not be used for downstream transfer-function or impedance work.
In a field workflow, keep the thresholds in the deployment configuration or provenance manifest. That makes later decisions reproducible when the same survey is reviewed away from the instrument.