2.27.2.5. pycsamt.pipeline._cache#

Content-addressed disk cache for pipeline step outputs.

A cache hit for step i requires the exact same fingerprint of the sites flowing into it, the exact same step code, and the exact same params as a previous run — a Make/Docker-layer-style chain hash. This is what makes a crashed-and-rerun Pipeline.run(..., cache=True) call “resume”: every already-completed step replays from cache instead of recomputing, and only the step that was interrupted (and anything after it) actually runs.

Deliberately simpler than pycsamt.forward.maxwell.cache.MaxwellResultCache — no cross-process locking, no size-based eviction. This is a disposable, regeneratable local cache, not scientifically load-bearing output, so that extra complexity isn’t worth it here. See Caching And Resume for the caveats this implies (non-deterministic steps are not safe to cache).

Functions

chain_key(upstream_fp, code, params)

sha256 of (upstream fingerprint, step code, normalized params).

default_cache_root()

fingerprint_sites(sites)

Deterministic content hash of a Sites-like collection.

Classes

StepCache([root])

Sharded, content-addressed disk cache for pipeline step outputs.
class pycsamt.pipeline._cache.StepCache(root=None)[source]

Bases: object

Sharded, content-addressed disk cache for pipeline step outputs.

Layout: root/<key[:2]>/<key>.joblib. Writes are atomic (temp file in the same directory, then os.replace), so a killed process never leaves a half-written entry visible. A corrupt or unreadable entry is treated as a miss — warns, never crashes the run — the same “one bad entry must not break everything else” principle pycsamt.pipeline.discover_plugins() already established.

Parameters:

root (str | Path | None)

get(key)[source]

Return the cached value for key, or the _MISS sentinel.

Parameters:

key (str)

Return type:

Any

put(key, value)[source]

Atomically store value under key.

Parameters:
Return type:

None

clear()[source]

Remove every entry (and the root directory itself) from disk.

Return type:

None

pycsamt.pipeline._cache.fingerprint_sites(sites)[source]

Deterministic content hash of a Sites-like collection.

Tries the real Sites shape first — iterates stations, hashing each one’s name plus its freq/z arrays, in iteration order. Order is part of the fingerprint on purpose: under-caching on a harmless reorder is a safer failure mode than treating two differently-ordered (and potentially differently-behaving) inputs as identical.

Falls back to hashing the pickled object for anything that doesn’t match that shape (plain test doubles, custom plugin-defined sites-like objects) — this keeps the cache usable without special-casing every possible “sites” representation.

Parameters:

sites (Any)

Return type:

str

pycsamt.pipeline._cache.chain_key(upstream_fp, code, params)[source]

sha256 of (upstream fingerprint, step code, normalized params).

default=str in the JSON dump means an unhashable/odd param type (a callable, a numpy array) never raises — it just gets stringified, which is deterministic even if imperfect (two different objects with identical repr would collide). Documented, not silently hidden.

Parameters:
Return type:

str