Deterministic Replay

Re-execute a saved decision against a ReplayEngine and compare the new output to the one you recorded — so a model swap, a prompt edit, or a dependency bump can’t silently change what your system decides.

replay For: regression & verification

When you’d reach for this

You ship a new version of the classify_ticket model and want to know, before it reaches users, whether it still routes the same tickets the same way. Replay the decisions you already recorded against the new build: if outputs_match flips to False on tickets that used to be stable, you have caught a regression instead of a customer noticing it. It is also how you demonstrate, after the fact, that a recorded decision is reproducible.

How it works

1. Persist a DecisionSnapshot to a storage backend (this is the original you’ll compare against).

2. Replay it through ReplayEngine, which re-executes the decision and compares the new output to the recorded one.

3. Interpret the ReplayResult — outputs_match, status, and policy_violations tell you whether the decision held and what to do next.

flowchart LR
    A[Recorded decision<br/>in storage] --> B[ReplayEngine.replay]
    B --> C{outputs_match?}
    C -->|True| D[Reproducible — ship]
    C -->|False| E[Regression — investigate]

Diagram description

A recorded decision in storage flows into ReplayEngine.replay. The engine checks whether the replayed output matches the original. If it matches, the decision is reproducible and safe to ship; if it does not match, you have a regression to investigate.

Install

pip install briefcase-ai[replay]

from briefcase.replay import ReplayEngine, ReplayPolicy, ReplayStats

Persist, then replay

import briefcase
from briefcase import DecisionSnapshot, Input, Output
from briefcase.storage import SqliteBackend
from briefcase.replay import ReplayEngine

briefcase.init()

decision = DecisionSnapshot("classify_ticket")
decision.add_input(Input("text", "Reset my password", "string"))
decision.add_output(Output("category", "account_access", "string"))
decision.with_execution_time(12.5)

backend = SqliteBackend.in_memory()
decision_id = backend.save_decision(decision)

engine = ReplayEngine(backend)
result = engine.replay(decision_id, "strict")

print(result.status)
print(result.outputs_match)
print(result.replay_output)
print(result.execution_time_ms)
print(result.policy_violations)

ReplayEngine(backend) takes a storage backend. replay(decision_id, mode) takes the mode explicitly.

Strict vs. tolerant

The mode decides how exactly the replayed output must match the original. Pick it from how deterministic the decision is supposed to be.

Mode	Matches when	Reach for it when
"strict"	The replayed output is identical to the original	The decision is meant to be deterministic — a fixed classifier, temperature=0, a routing rule. Any difference is a regression.
"tolerant" (default)	Minor differences are allowed	The output is free-form or sampled (a generated reply, a summary) where wording can vary but meaning should not.

Note

"tolerant" is the engine default. For per-field control — exact match on category, a similarity threshold on a free-text summary — use a ReplayPolicy instead of choosing one mode for the whole decision.

Interpreting a ReplayResult

ReplayResult is what you act on. The table maps each field to the decision it should drive.

Field	What it tells you	What to do
status	"success" or a failure status for the replay	A non-success status means the replay itself couldn’t complete (decision missing, load error) — fix the harness before trusting the result.
outputs_match	True when the replayed output matches the original	False on a decision that used to be stable is a regression — investigate the change you just made.
replay_output	The output produced during this replay	Diff it against the original to see exactly what drifted.
policy_violations	List of policy rules the replay violated	Non-empty means a specific field broke its match rule — read it to know which field and why.
execution_time_ms	Replay execution time in milliseconds	A large swing can flag a performance regression even when the output still matches.
original_snapshot	The recorded decision being replayed	The baseline for the comparison and your audit reference.

Replay with a policy

A ReplayPolicy declares how each output field must match. Combine exact-match fields with similarity-threshold fields when one decision has both a structured label and free text.

from briefcase.replay import ReplayPolicy

policy = ReplayPolicy("output-consistency")
policy.with_exact_match("category")
policy.with_similarity_threshold("summary", 0.95)

result = engine.replay_with_policy(decision_id, policy, "strict")
print(result.status)
print(result.policy_violations)

Here category must match exactly (a misroute is unacceptable) while summary only has to stay 95% similar (wording may vary).

Replay in batches

Verify a whole regression set at once instead of one decision at a time.

results = engine.replay_batch([decision_id], "strict", 4)
for result in results:
    print(result.status, result.outputs_match)

replay_batch(decision_ids, mode, max_concurrent) replays many decisions concurrently, bounded by max_concurrent.

Aggregate replay statistics

from briefcase.replay import ReplayStats

stats = engine.get_replay_stats([decision_id])
print(stats.total_replays)
print(stats.successful_replays)
print(stats.exact_matches)
print(stats.success_rate)
print(stats.average_execution_time_ms)

success_rate across your regression set is the one number to watch release over release: a drop means more decisions changed than you expected.

Key classes

Class	Why it matters
ReplayEngine	Loads a persisted decision from a backend and re-executes it — the entry point for every replay.
ReplayResult	Outcome of a single replay; the fields you act on to catch a regression.
ReplayPolicy	Per-field match rules for replay_with_policy so structured and free-text fields can be judged differently.
ReplayStats	Aggregate counts and rates across many replays — your release-over-release health signal.

Where this fits

Replay is the start of the Replay & Verify act: re-run a stored decision, then measure how far it moved over time and prove the record is intact.

Storage Adapters Previous: persist the decisions that replay re-executes.

Drift Detection Next: measure how consistent outputs stay across many runs.

Audit Bundles Then: bundle a decision into a tamper-evident, verifiable record.

Decision Recording Build the DecisionSnapshot objects replay compares against.