6.3.11. Reproducible experiment configuration

Every other package in Architecture roadmap’s map produces evidence: a geological realization, a forward response, a fitted normalizer, a loss value. None of that evidence answers a simpler question a reviewer always asks first — what exactly produced this result, and against what threshold was it judged acceptable? pycsamt.ai.experiments exists to answer that question before a run starts, not by reconstructing it afterward from scattered scripts and a remembered epoch count. Its scope is deliberately narrow: it describes what an experiment is — a pinned dataset, a seed lineage, a configuration, a set of acceptance criteria — without importing a machine-learning framework or constructing a Maxwell solver. A config object here is a record, not a runner.

6.3.11.1. Deriving stable, order-independent seeds

A single global seed=42 sounds reproducible until two subsystems happen to draw from the same generator in a different order between runs, or a training seed and a data-shuffling seed accidentally coincide. SeedPlan replaces that with one root seed and a namespace, from which every named subsystem derives its own child seed deterministically:

(1)\[\mathrm{derive}(\ell) = \mathrm{uint}_{32}\!\Big( \mathrm{SHA\text{-}256}(n \mathbin\Vert r \mathbin\Vert \ell)[0{:}4] \Big),\]

where \(n\) is the namespace, \(r\) the root seed, \(\ell\) the label, and \(\mathbin\Vert\) byte-string concatenation with a null separator between fields. Equation (1) is a pure function of its three inputs, which is exactly what makes it useful: the same label always derives the same seed from the same plan, regardless of how many other labels were requested first or in what order.

The 32-bit result is compatible with NumPy and common learning frameworks, but it is not a mathematical guarantee of uniqueness. If \(k\) distinct labels behave like independent hash inputs, the birthday approximation gives

(2)\[P(\text{at least one collision}) \approx 1-\exp\!\left[-\frac{k(k-1)}{2^{33}}\right].\]

For 100 labels this is about \(1.15\times10^{-6}\); for 1,000 labels it is about \(1.16\times10^{-4}\). Ordinary experiments are therefore very unlikely to collide, but a large sweep should derive all labels once with derive_many() and assert that the returned values are unique. The namespace is a domain separator, not extra entropy: changing it intentionally creates a different seed lineage.

>>> from pycsamt.ai.experiments.config import SeedPlan

>>> plan = SeedPlan(42, namespace="willy-l18")
>>> plan.derive("geology") == plan.derive("geology")
True
>>> plan.derive("geology") == plan.derive("network")
False
>>> plan.derive("network")
2959107669
>>> children = plan.derive_many(["geology", "network", "shuffle"])
>>> len(children) == len(set(children.values()))
True

This is not a hypothetical convenience: 2-D Maxwell training-data generation’s generate_2d_maxwell_dataset() builds one SeedPlan from a dataset configuration’s root seed and derives a labeled f"{realization_id}/correlation" and f"{realization_id}/field" seed for every realization it generates, so regenerating realization 47 alone, on a different machine, still draws the exact field and correlation-length values realization 47 drew the first time — no realization’s randomness depends on how many realizations were requested before it. 3-D Maxwell training-data generation’s generator follows the identical seed-derivation pattern for its own correlation/field/split seeds.

6.3.11.2. Pinning a dataset by hash, not by path

A file path is not a stable reference: willy_l18.npz can be silently regenerated, moved, or overwritten while an experiment record still points at its old location and nobody notices until results stop reproducing. DatasetReference pins an experiment to the SHA-256 digests Canonical data contracts already produces for exactly this purpose — a DatasetManifest’s manifest_hash, a RealizationSplit’s split_hash, and, when the experiment normalizes data, a ComplexZScore’s state_hash — rather than to a path a caller might record as an afterthought:

>>> from pycsamt.emtools._core import ensure_sites
>>> from pycsamt.ai.domain_gap import survey_data_from_sites
>>> from pycsamt.ai.data.normalization import ComplexZScore
>>> from pycsamt.ai.data.splits import RealizationSplit
>>> from pycsamt.ai.data.manifest import DatasetManifest
>>> from pycsamt.ai.experiments.config import DatasetReference

>>> sites = ensure_sites(
...     "data/AMT/WILLY_data/L18PLT", recursive=True, verbose=0
... )
>>> field = survey_data_from_sites(sites, recursive=False, verbose=0)
>>> state = ComplexZScore.fit(field)
>>> split = RealizationSplit(("willy-l18",), (), ())
>>> manifest = DatasetManifest(
...     dataset_id="willy-l18-zscore-v1",
...     generator="pycsamt.ai.data.normalization.ComplexZScore",
...     generator_version="1.0",
...     configuration={"weighting": "uniform", "eps": 1e-8},
...     split=split,
...     sample_count=field.n_stations,
... )

>>> dataset_reference = DatasetReference(
...     "willy-l18-zscore-v1",
...     manifest.manifest_hash,
...     split.split_hash,
...     normalization_hash=state.state_hash,
... )
>>> dataset_reference.manifest_hash == manifest.manifest_hash
True

Every digest above comes from the exact same L18 objects Canonical data contracts builds and checks: this is not a parallel bookkeeping scheme, it is those hashes read back into a record that names which dataset, which split, and which fitted normalizer an experiment actually used.

These digests provide content integrity, not storage or automatic verification. DatasetReference checks that each supplied digest has the syntax of a SHA-256 value; it does not open manifest_uri and prove that the object currently stored there has that digest. At the execution boundary, load the manifest, split, and normalizer, recompute their hashes, and refuse the run if any value differs from the reference. A URI answers “where might I retrieve it?” while the digest answers “is this the object I approved?” Neither substitutes for the other.

6.3.11.3. Fixing acceptance criteria before looking at results

Choosing a passing threshold after seeing the score is not validation, it is curve fitting to one run. AcceptanceCriterion exists to make that impossible by construction: a metric name, a comparison operator, and a threshold, frozen into the experiment record before any metric is observed. A held-out RMS misfit threshold is the most natural example — the same normalized residual concept Recovery, residual, and OOD diagnostics computes and this page’s criteria merely give a name and a number to:

>>> from pycsamt.ai.experiments.config import AcceptanceCriterion

>>> nrms_ok = AcceptanceCriterion(
...     "test.impedance_nrms", "<=", 2.0, "response fit vs. observed"
... )
>>> nrms_ok.evaluate(1.6)
True
>>> nrms_ok.evaluate(2.4)
False

A single criterion is only half the picture; a real experiment usually predeclares several, and evaluate_gate() checks all of them together as one GateEvaluation. Crucially, a metric an experiment forgot to compute is not silently skipped — it counts as an incomplete, failing gate rather than a passing one by omission:

With criteria \(C_j(m_j)\) evaluated on observed metrics \(m_j\), the final decision is the logical conjunction

(3)\[G = \left(\bigwedge_{j=1}^{J} C_j(m_j)\right) \land \left(\mathcal M = \varnothing\right),\]

where \(\mathcal M\) is the set of missing required metrics. Thus one failure or one missing value makes \(G\) false. NaN and infinity are rejected rather than compared, and duplicate metric names are forbidden. Avoid == for floating-point scientific metrics unless exact equality is truly meaningful; a tolerance expressed with <= or >= is normally the defensible choice.

>>> criteria = [
...     AcceptanceCriterion("test.impedance_nrms", "<=", 2.0),
...     AcceptanceCriterion("test.recovery_rmse", "<=", 0.5),
... ]
>>> from pycsamt.ai.experiments.config import ExperimentConfig, SeedPlan
>>> config = ExperimentConfig(
...     "willy-l18-mt2d-v1",
...     "learning_2d",
...     dataset_reference,
...     SeedPlan(42, namespace="willy-l18"),
...     model={"architecture": "unet", "lambda_x": 0.1, "lambda_z": 0.1},
...     training={"epochs": 100, "batch_size": 8},
...     physics={"solver": "mt2d", "components": ["zxy"]},
...     acceptance=criteria,
... )

>>> full = config.evaluate_gate(
...     {"test.impedance_nrms": 1.6, "test.recovery_rmse": 0.42}
... )
>>> full.passed
True

>>> partial = config.evaluate_gate({"test.impedance_nrms": 1.6})
>>> partial.passed, partial.complete, partial.missing
(False, False, ('test.recovery_rmse',))

partial never claims a pass it cannot support: with test.recovery_rmse unmeasured, complete is False and the gate fails, even though the one metric it did see would have passed on its own. That asymmetry — silence must fail, never pass — is the whole point of predeclaring criteria in the first place.

For a genuine 3-D experiment, response fit and model recovery are necessary but not sufficient gates. Predeclare a solver benchmark tolerance for the chosen frequency and resistivity range, a maximum rejected-realization rate, held-out volume or column recovery metrics, field-domain coverage, and the ordinary impedance misfit. A tiny linear-system residual only establishes that one discrete system was solved; it must not be substituted for a half-space or layered benchmark, mesh-refinement evidence, or blind field performance. Metrics calculated on accepted forward realizations also cannot hide rejected attempts: the requested and accepted counts belong in the observed evidence used by the gate.

6.3.11.4. The complete configuration

ExperimentConfig is where a DatasetReference, a SeedPlan, and a list of AcceptanceCriterion join model/training/physics sections that are frozen and hashed without being interpreted. That restriction is deliberate: this class has no opinion on whether {"architecture": "unet"} is a valid U-Net configuration, because answering that would require importing the very training framework this package exists to stay independent of. Validating those sections is Training AI inversion models’s and AI model selection’s job, not this one’s — its own job is to make sure the sections used for a given run are pinned exactly and reproducibly.

>>> config.config_hash == config.config_hash
True
>>> config.child_seed("network") == config.seeds.derive("network")
True

The stage field is not free text either: it must be one of eleven fixed values, and those eleven are not arbitrary — they are Architecture roadmap’s own M0-M10 stages, spelled as portable identifiers instead of milestone codes.

Milestone	stage value	Roadmap scope
M0	baseline	Baseline freeze and reproducibility
M1	data_audit	Survey data contract and audit
M2	geology	Correlated geological priors
M3	domain_gap	Domain-gap and noise simulation
M4	forward_2d	Genuine 2-D electromagnetic forward path
M5	learning_2d	Response-aware 2-D learning
M6	feasibility_3d	3-D Maxwell solver feasibility
M7	forward_3d	Verified 3-D forward backend
M8	learning_3d	Correlated 3-D training and spatial model
M9	hybrid	Hybrid inversion and uncertainty
M10	field_evaluation	Blind evaluation and release

An ExperimentConfig therefore cannot claim to belong to a stage that Architecture roadmap does not itself define, which is what makes config.stage a meaningful, checkable fact about a run rather than a free-form label a spreadsheet would need to interpret by convention. Saving and reloading the record is a plain, deterministic JSON round trip, exactly like Canonical data contracts’s manifests and audit reports:

For 3-D work, the uninterpreted physics mapping should nevertheless be complete enough for a reviewer to distinguish computations. At minimum pin:

• solver and its implementation version, such as mt3d-python versus modem3d-external;

• the forward-problem or solver-input hash and, for an external backend, the executable/version identity and invocation configuration;

• frequency order, requested tensor components, permeability convention, and receiver coordinates;

• the geological grid, padded solver mesh, and output grid definitions or their content hashes; and

• boundary padding, max_mesh_cells, cells_per_skin_depth, convergence policy, terrain/inactive-cell support, and accepted/rejected realization counts.

Inv3DAgent(physics="mt3d") currently selects the research MT3DAdapter training route; it does not prove that the compiled ModEm3DAdapter ran. Conversely, a ModEM response converted to the shared SurveyData contract remains a ModEM-generated artifact. Preserve the backend identity from the clean parent realization through corruption, training, prediction, and reporting instead of replacing it with the generic dimensionality label mt3d.

>>> from pathlib import Path
>>> from tempfile import TemporaryDirectory

>>> with TemporaryDirectory() as directory:
...     path = config.write_json(Path(directory) / "experiment.json")
...     reloaded = ExperimentConfig.read_json(path)
...     print(reloaded.config_hash == config.config_hash)
True

A config_hash mismatch after that round trip would mean the configuration was not actually reproduced — reloading it and getting the same digest back is the check, not an assumption.

The digest is computed from canonical JSON over every serialized field, including created_utc. Mapping key order does not affect it, while a changed threshold, tag, timestamp, dataset hash, or nested model value does. Consequently, two scientifically identical configurations created with different timestamps have different hashes. If the hash is intended to identify a protocol rather than an individual record, either hold created_utc fixed or leave it None and timestamp the execution record instead. The in-memory mappings and sequences are recursively frozen, which prevents accidental mutation after the hash has been reviewed.

6.3.11.5. One frozen protocol still needs repeated runs

Neural optimisation is stochastic even when the dataset and protocol are fixed. A result from one favourable network seed estimates neither central performance nor run-to-run instability. Keep one ExperimentConfig and derive labels such as network/run-000 through network/run-011; then report every run, the aggregate distribution, and the fraction satisfying the complete gate. Do not alter experiment_id or thresholds after seeing which seeds fail.

The diagnostic below makes that distinction concrete. Every point belongs to the same frozen configuration and dataset; colour indicates the joint gate, not whether the point passes the panel in which it appears. Run 4, for example, has acceptable recovery RMSE and coverage but fails because NRMS is above 2.0. Run 5 fails only the RMSE threshold, while run 6 fails coverage. Seven of twelve runs pass all three criteria. Reporting only their mean would hide both the failure modes and the probability of obtaining an acceptable training outcome.

Resuming a long 2-D or 3-D training run is the continuation of the same experiment only when the checkpoint agrees with the frozen protocol. Before loading it, verify the experiment, manifest, split, normalization, and model schema hashes; after loading, retain the completed epoch, optimizer and scheduler states, network weights, and every available random-generator state. If the current checkpoint format does not serialize one of those states, record that limitation and treat bitwise continuation as unproven. Changing the realization count, split, frequency grid, depth grid, feature order, solver backend, or loss weights creates a new experiment configuration, even when training starts from old weights. A checkpoint file existing on disk is therefore not itself evidence that L26, L30, or any other line finished its configured gate.

A pinned experiment protocol followed by per-seed joint-gate evaluation. Dashed lines are thresholds fixed before the metrics were generated.

6.3.11.6. What the configuration does not prove

An ExperimentConfig is deliberately a source-of-truth input, not an experiment tracker or execution attestation. Its hash does not prove that the declared solver, dataset, or seed was actually used, and the current pycsamt.ai.experiments package does not automatically capture runtime artifacts. A reproducible hand-off must therefore pair the configuration JSON and its hash with, at minimum:

• the observed metrics and serialized GateEvaluation state;

• derived seed labels and values for every stochastic subsystem;

• source revision, pyCSAMT and dependency versions, solver backend, numerical precision, device, and deterministic-runtime settings;

• checkpoint, prediction, log, and report checksums; and

• start/end timestamps plus termination status.

Those are execution evidence, whereas the configuration is the approved protocol. Keeping the two roles separate prevents a common provenance error: a perfectly hashed declaration being mistaken for proof that the declared computation took place. The broader packaging workflow in AI inversion reporting supplies the natural home for that evidence.