Eval loops

Rendered from knowledge/harness-engineering/eval-loops.md — the text an agent reads.

Any natural-language artifact an LLM reads at decision time — a policy, a prompt, a knowledge profile, a skill — can be trained like a model: the artifact plays the role of the weights, a labeled set plays the role of the spec. The loop is the same regardless of what the artifact is. Use it whenever an artifact’s quality matters enough to measure — a policy a bot applies, a knowledge profile agents answer from, a skill that shapes output.

The loop

Draw a stratified labeled set with a fixed, recorded seed and split it train/val before any editing. Iterate on train only: one numbered snapshot per attempt (artifact + scores + per-row errors), one ledger row per iteration naming the change that drove it. A scoring function runs the artifact closed-book. Stop at plateau — gains under a pre-declared threshold and declining yield per change; the call weighs several signals, not a bare counter — then score a held-out set exactly once. “Train moved but val didn’t” is the loop’s most valuable output; every serious campaign hits it at least once.

Trust the harness before the number

A wrong harness produces plausible numbers; check it before believing anything it prints.

The production implementation decides, never a re-derivation. For every question about what the system would do — admissibility, candidate reachability, label derivation — call the real decision code. A harness-side re-implementation reads as ground truth while quietly inverting semantics — a veto rebuilt as a requirement can turn a mostly-true rate into a mostly-false one; where a filter runs upstream of the function under test, only the real code knows what reaches it.
Score selection, not just classification. A harness that hands the system exactly the one candidate ground truth already chose never exercises its filtering, ordering, or tie-breaks. Rebuild the real candidate population the production query would return and also score which item was chosen — a perfect agreement figure can survive this change and mean something entirely different afterwards.
Give invisible paths their own recall gate. Metrics built from rows with a known good answer are precision gates by construction: a fallback that fires only when the primary path finds nothing never moves them — and neither does silently deleting it. Build a ground truth from the cases the path recovers, and gate on that.
Break the thing the gate guards. A green gate is evidence only once you have seen it go red for the right reason: disable the guarded mechanism, confirm the harness fails, revert — at the moment the gate is written, not after a silent regression.
Check for headroom. A metric at its ceiling from the first run is a regression detector, not an optimizer — it cannot rank proposals, and changes end up settled by argument. Widen the scored population or redefine the objective before running the loop.
Assert baselines. Automating the loop needs no orchestration layer: turn each metric’s known baseline into a suite assertion, so a regression fails the run instead of printing a smaller number someone must notice.

Held-out discipline

Running the held-out set burns it: observing the result is itself information that leaks into later edits. Only the held-out number is publishable; train/val accuracy is internal. A burn has a recovery path: the burned partition rolls into the working set as training material, and any future published score comes from a fresh partition drawn only from data collected after the burn. A relative ship bar (held-out within a fixed fraction of train) is more usable than an absolute one. Scan artifact diffs for long verbatim substrings of canonical answers; each match must be classified (shared identifier, shared primary-source phrasing, suspect), and suspect matches block the iteration.

Ground truth

Observed decisions beat authoritative documents. An artifact rewritten from real human verdicts outperforms one synthesized from the official policy corpus, before any iteration runs. Documents state intent; decisions reveal the actual threshold. But date the decisions: a historical label stream encodes retired practice — relabel passes routinely disown a large share of old labels — so prefer recent, high-confidence decisions and treat the rest as noisy by design.
Relabel the errors before iterating on them. When a plateau smells like label noise, adjudicate a bounded sample of the system’s errors into: system-right (label noise), owner-right (real error), unevaluable (the input changed since labeling), harness bug. Report a noise-adjusted metric beside the raw one — the gap is often large enough to change the ship decision — and let only the owner-right bucket drive artifact edits.
Freeze fetched inputs. Rows that point at live artifacts decay, and the decay is directional — it manufactures errors on one side and silently biases every later iteration. Cache fetched inputs once at sample build for deterministic reruns, and treat a sample re-draw as an intervention with its own before/after number.
Version the ground truth. Widening the reference — more raters, more of the population, stricter admission — lowers the score by design; that drop is not a regression, and scores across two definitions of ground truth don’t compare. Name the change in the ledger and re-baseline; otherwise the metric is trivially improvable by quietly narrowing what counts.
Filter behavior-cloning sets mechanically. When ground truth is “what a person actually did,” the raw record contains non-judgments: empty approvals, positions later retracted, replies to others — and the system’s own past output, which must never grade it. Filter each class by an explicit rule, log the dropped counts, and state the direction of any known leak.
Unlabeled is not negative. “The join found nothing” and “no answer should exist here” are different claims; leaving unresolved rows unscored makes recall unmeasurable and rewards aggression. Label them, or sample them with recorded weights.
The incumbent gets three roles — feature, baseline, never label. An existing automation’s verdict can be an input signal and the arm to beat, but using it as ground truth bakes its errors in (it is usually worst exactly on the minority class). Quantify its error rate before deciding anything — and keep labelers blind to its verdicts, or the labels certify the status quo.
A regression set built from the change it justifies is a gate, not an objective term. It correctly catches a future silent break; it cannot rank proposals, because every proposal is scored partly against a set constructed to flatter the current implementation.
Match raters to the question. A single person’s threshold needs single-rater labels — mixing raters folds inter-rater noise into the measurement; a shared standard needs multiple raters and majority resolution. Either way, measure the raters’ own noise before trusting margins.
Find the sample-size floor empirically. Rerun the identical configuration twice and watch the swing; if sampling variance alone moves the headline metric more than the effects you chase, the setup is too weak — grow the set, reduce the instrument’s own noise, or retire the bar.
Split by the entity that leaks — author, account, thread — not by row. Near-duplicates must never straddle the boundary.
Only real questions qualify for Q&A sets. The test: remove the answer — would the question still be a coherent thing for a human to type? Synthesized questions eval the synthesizer.
Name the ceiling. When ground truth is internally inconsistent (humans disagreeing with themselves), state the realistic agreement ceiling instead of chasing a perfect score. When two ground truths conflict, name the conflict and pick one — never average them.

Metrics

Split error costs. Report each error direction separately and give the costlier, less-reversible one the stricter bar. Headline accuracy hides rollback-worthy cost shifts; the best-accuracy iteration can be the one you revert. When one number must rank proposals, combine the directions into a single cost function with explicit weights per error class and severity tier — a stakeholder decision written as numbers, not prose — and report the confusion matrix beside the scalar. On a multi-axis scorecard, decision axes gate promotion; cost and size metrics only break ties.
Count abstentions honestly. A system that can hold or answer “unknown” needs a denominator rule: exclude abstained rows from accuracy, report the abstention rate beside it as a first-class number, and bound it — an arm must not abstain its way to a high score. Unscoreable inputs (content gone, fetch failed) are a third counted category, never silently dropped.
Repeat runs and report disagreement. The pipeline is nondeterministic, so single-run deltas of a few rows are unreadable. Fix K repetitions per item per arm, report the share of items whose runs disagree, and check that flips run both directions before reading a small delta as an effect. Often a coarser projection — production-threshold dispositions — is stable while raw verdicts churn; quote the stable level.
Simulate the deployment decision. Score what the system would actually have done at production thresholds, not only row-level accuracy — that is the number a launch decision cares about. A change invisible at those thresholds is operationally a no-op; keeping it anyway is a legitimate call, made explicitly on non-metric grounds.
Make categories un-gameable. If one label dominates, flat top-1 is gamed by always answering it; score the informative subset.
Two baselines, always — the incumbent automation, and the bare model; give the bare arm live retrieval when that is the artifact’s real competitor, and run three arms (bare-with-tools / artifact-only / artifact+tools) when they differ. Read the result per scope: an artifact can lose to live search overall and still earn its keep on one slice. Compare on the baseline’s coverage subset — an incumbent with an opinion on only part of the sample makes cross-denominator claims unsound — and scope-match a broader incumbent before crediting a challenger’s lead.
Grade knowledge artifacts against real questions with a rubric separating answerable / partial / missing-here / belongs-elsewhere / misleading — and fix misleading first; a wrong confident answer costs more than a gap. The misleading leg is the one successor campaigns tend to quietly drop — preserve it deliberately. Multi-category rubrics drift between scoring rounds: watch the category distribution, recalibrate scorers explicitly, and compare across iterations on the metric a recalibration cannot reclassify.
Lock and stamp the metric. The metric definition is a contract, not a loop variable: one owner edits it, loop agents file change proposals, every change bumps a version, and no comparison crosses versions without a note. Every scorecard stamps metric version, eval-set version, data snapshot, harness configuration, model, and repetition count.
Price the artifact’s length. Compaction is an experimental arm, not a style choice: re-run the identical eval with a compacted artifact. A shorter version can score better on the sharp metric while the aggregate stays flat — which reframes “what should we add?” into “what is the earning content?”

Moves the loop enables

Error-table-driven restructuring. A class that won’t learn signals a mis-shaped task; split it into staged binary decisions or rebucket. The metric sees the pivot before intuition does.
One change per iteration. Sweeping multi-category revisions are uninterpretable — a large gain on one error direction traded against a large loss on the other, with no attribution. Small surgical deltas are measurable; when a bundle must be evaluated as a unit and regresses, ablate to the responsible component, revert only that, and ship the rest.
Price a change against the real population before building it. Characterize what a proposal would act on using the real system with loosened knobs: how many cases, do they share one systematic mechanism, does an independent signal corroborate them? One mechanism plus corroboration is worth building; a scatter of coincidences is rejected without writing code. Record every proposal’s delta, rejected ones included — a round leaves a table of what was tried.
Hand-review what an accepted change newly produces. A crude automatic proxy misprices it — idiomatic true matches read as misses. The discriminating feature separating true from false often appears only in that review, and then becomes a new, tighter knob.
Signal ablation after plateau. Freeze the artifact and add one input signal at a time, each with a cost line (latency, calls, tokens). Signal additions routinely out-lever further prompt tuning. Run it subtractively too — hide a signal the system already consumes behind a switch to price its contribution — and leave the switch wired so the question stays cheap to re-ask.
Spin-off classifiers. A single category dominating the error bucket gets its own loop with its own labeled set; fold it back only once it wins on its own eval — and re-verify in the parent, because a spin-off win does not transfer automatically. Sample the source bucket first: a catch-all label in the parent taxonomy hides rows labeled on an axis the spin-off’s inputs can never see — name that share as a floor on achievable performance.
Deterministic prefilters as their own arm. A cheap rule layer short-circuiting easy rows is proven on the same eval to cost nothing in accuracy. A companion arm that helps train but hurts val is the split doing its job — reject it as overfit.
Paired model comparison on identical rows. Compare candidate models with a paired test on the same sample and the same artifact. Calibration matters as much as accuracy: a slightly-more-accurate but less-confident model automates less at the same thresholds.
Budget the loop itself. Cap cases per iteration (and record which), set a token ceiling that triggers stop-and-report, skip a case after a fixed run of tool errors and log the skip, and put a human checkpoint at phase boundaries. Publish the loop’s own cost so the next campaign can budget it.

After deployment

Measure the deployed effect from systems of record, never the runner’s self-report: the load taken off the queue, the decision mix, a quality proxy — each against a conservatively-chosen pre-window. Bound the obvious confound explicitly (did volume simply fall?) and show the change is a step at the go-live boundary, not a trend that predates it.
Reweight stratified samples before quoting operational numbers. Oversampling the rare class is right for iteration and wrong for reporting — a rate conditioned on the system’s own positive decisions can overstate the production rate severalfold. Reweight to the true base rate, report both, and sanity-check the reweighted coverage against the observed production rate.
Pin the version your number describes. Production is frequently several artifact revisions behind the eval; verify what is deployed by content hash, not modification time, and record in the ledger which version each published number measured. If iteration continued past the last scored version, say so.
Grow a hard-case slice from escalations. After deployment, the population that matters is the one the system routes to humans. Build a second eval slice from those rows, ground-truthed by the human who resolved each, and score changes against both sets — a change neutral on the main set can be decisive there, and the slice shows honestly how weak the artifact is on genuinely ambiguous input.