Evaluation Runs

Outcome. Record an evaluation run as decision records, or replay inspect-ai and lm-eval-harness logs already on disk.

Prerequisites. The evals extra; a log directory for the parsers, or your own harness for a live run.

Packages. briefcase-ai

Languages. Python

@capture records one function call. An evaluation is a different shape: many cases, one run, and a verdict over the whole thing. briefcase.integrations.evals records that shape as decision records, so an eval lands in the same store as everything else you capture.

pip install briefcase-ai[evals] New in 4.0.0

When you’d reach for this

You run a nightly eval and want its pass rate next to the production decisions it is supposed to predict. Or you have a directory of inspect-ai logs and want them queryable without keeping inspect-ai installed on the machine doing the querying.

Install

pip install briefcase-ai[evals]

The extra installs zstandard on Python below 3.14, which zipfile needs to read inspect-ai .eval archives. Nothing else; the parsers are stdlib only.

Log a run as it happens

EvalRun is a context manager. Each log_case emits one eval.case record; leaving the block emits a single eval.run summary.

from dataclasses import dataclass


@dataclass
class Case:
    id: str
    question: str
    answer: str
    prompt_tokens: int = 0
    completion_tokens: int = 0


dataset = [
    Case("q1", "What is 2+2?", "4", 8, 1),
    Case("q2", "Capital of France?", "Paris", 9, 1),
]
model = lambda question: "4" if "2+2" in question else "Paris"

import briefcase
from briefcase.integrations.evals import EvalRun

briefcase.observe("runs.jsonl")

with EvalRun("gsm8k", model="claude-opus-4-5") as run:
    for case in dataset:
        answer = model(case.question)
        run.log_case(
            case.id,
            inputs=case.question,
            outputs=answer,
            target=case.answer,
            passed=answer == case.answer,
            scores={"exact_match": 1.0 if answer == case.answer else 0.0},
            input_tokens=case.prompt_tokens,
            output_tokens=case.completion_tokens,
        )

print(run.summary()["pass_rate"])

summary() aggregates pass rate, per-score mean/min/max/count, and token totals. With a model and token counts it also estimates cost through briefcase.cost; pass include_drift=True to add output drift across cases. Both degrade to None rather than raising when the calculator is unavailable.

Synchronous by default

EvalRun sets async_capture=False, unlike @capture. Batch jobs exit the moment they finish, and a background export thread would lose the tail of the run.

Replay a log you already have

The parsers are stdlib only and never import the evaluation framework, so you can read a log on a machine where neither tool is installed.

from briefcase.integrations.evals import from_inspect_log, from_lm_eval_results, replay

# inspect-ai: a .json log or a .eval archive
replay(from_inspect_log("logs/2026-08-12_gsm8k.eval"))

# lm-eval-harness: results plus its per-sample jsonl
replay(from_lm_eval_results("results.json", "samples_gsm8k.jsonl"))

replay() emits the same eval.case and eval.run records as a live run, so a replayed log and a fresh one are queried identically.

To inspect before emitting, parse without replaying:

parsed = from_inspect_log("logs/run.eval")
parsed.name      # task name
parsed.model     # model under test
parsed.metrics   # aggregate scores, e.g. {"match/accuracy": 0.5}
parsed.cases     # normalized case dicts, ready for EvalRun.ingest

A file that does not match the expected shape raises ValueError naming the format, rather than returning a half-parsed run.

.eval archives need a zstd backend

inspect-ai’s default .eval format stores zstd-compressed zip entries, which the standard library can only decode on Python 3.14+. The evals extra installs zstandard for older versions. Without a backend, .eval parsing raises with install instructions; .json logs and lm-eval results are unaffected.

Feed parsed cases into a run you control

from briefcase.integrations.evals import EvalRun, from_lm_eval_results

parsed = from_lm_eval_results("results.json", "samples.jsonl")

run = EvalRun("nightly", exporter=my_exporter, model=parsed.model)
run.ingest(parsed.cases)
print(run.finish()["outputs"]["pass_rate"])

Record types

Type	Emitted	Carries
eval.case	once per case	case id, inputs, outputs, target, scores, pass flag, tokens
eval.run	once per run	pass rate, per-score statistics, token totals, cost, optional drift

Both flow through whatever exporter briefcase.observe(...) configured, exactly like a @capture record.

What the parsers understand

1. inspect-ai — .json logs and .eval archives. Letter grades map C/I/P/N to 1.0/0.0/0.5/0.0; chat-message inputs are flattened to role: content lines; output is read from the first choice’s message.

2. lm-eval-harness — results.json for aggregate metrics (filter suffixes stripped, stderr and alias dropped) and the samples jsonl for per-case rows. Multiple-choice tasks produce one [logprob, is_greedy] pair per choice rather than generated text, so outputs holds that list as JSON.

Parsing is best-effort over the documented shapes: missing optional keys are skipped. Both parsers are verified against artifacts inspect-ai 0.3.257 and lm-eval-harness 0.4.12 actually wrote.

Limits

pass_rate counts only cases that carried a verdict. log_case without passed= records the case and leaves it out of the rate. Two cases where one has no verdict and the other passed give total_cases 2 and pass_rate 1.0. Read passed + failed against total_cases before you trust the number.

Cost and drift degrade to None, never to an error. Cost needs a model the pricing table knows and token counts on the cases; an unknown model id yields None with nothing logged. Drift needs include_drift=True and at least two string outputs. A None here means not computed, not zero.

finish() is one-way. log_case after it raises RuntimeError. Calling finish() again returns the same summary without emitting a second eval.run.

The parsers read the shapes those two tools write today. A log from a much older or newer release, or a fork with a different schema, raises ValueError naming the format rather than half-parsing. That is deliberate: a partial eval is worse than no eval.

.eval archives need a zstd backend below Python 3.14. See the caution above.

Nothing re-runs the eval. replay() re-emits records from a log you already have. Producing new results is your harness’s job.

API reference

briefcase.integrations.evals has the full EvalRun and parser signatures.

Where this fits

Evaluation runs close the Operate & Evaluate act: aggregate many decisions into one comparable verdict instead of reading them one at a time.

Previous · Operate & Evaluate: Cost Tracking The pricing table a run needs before it can report cost.

Operate & Evaluate: Drift Detection The per-output measure a run aggregates with include_drift.

Guide: Compare Model Versions Push a baseline and candidate through the gateway and read the scorecard.