Blind protocols

Rendered from knowledge/harness-engineering/blind-protocols.md — the text an agent reads.

An agent that can see the answer, the incumbent narrative, or its own authorship launders it back and calls it analysis. Each protocol names what is hidden from whom, and the hiding is mechanical (principles.md).

Time cage. To evaluate diagnosis of an already-resolved event, freeze the agent at a past moment: wrap its data tools to inject a cutoff timestamp and post-filter results, and blocklist the surfaces where the answer was stated. This needs a single interception point — a common CLI or proxy every data tool routes through. Data behind unreachable dashboards is served by a coordinator acting as data oracle — replying only to explicit data requests with point-in-time snapshots, never volunteering or hinting. Bracket the run with two fresh agents: a contamination audit before (can anything it will read leak the answer? does the wrapper actually block?) and a fairness audit after (is every conclusion traced to cited, in-window evidence?). A rigorous run scoring lower than a contaminated one is the protocol working, not failing.

Blind editor. The agent fixing a failing eval item never sees the item. A coordinator who has seen it clusters failures by topic and restates the topic in its own words, stripped of distinctive phrasing, handing the editor only that plus primary-source pointers. If the topic can’t be restated without echoing the answer, the item tests phrasing, not knowledge — retire it. Expect the fence to bound overfitting, not eliminate it. Skip this protocol and every other protection is paper.

Pipeline separation. Question collection, artifact authoring, and evaluation run as separate flows that meet only at scoring. The authored artifact reads primary sources only — never the question corpus — or the eval is circular. Enforce prospectively with deny-lists and retrospectively with a provenance audit before first use; where a residue can’t be removed (the drafter knows the corpus exists), name it rather than claiming purity.

Rediscovery audit. To decide what a context doc earns, hide it mechanically and make a blind agent re-derive its content from the remaining world (code, other docs, primary sources) under a small search budget. Easily rediscovered facts are redundant context — cut them; unreachable ones earn their tokens. The same blind re-derivation corroborates a synthesis: an agent that reaches the same claims from sources alone is evidence the claims are in the sources, and a claim it can’t reach is a claim to re-check.

Condition-blind grading. Graders receive competing outputs in randomized order with the condition mapping held in a separate artifact, unblinded only at consolidation. Pre-register the protocol before running. Measure grader noise first — re-grade identical outputs with a fresh grader sample — and refuse effect-size bars below the instrument’s noise floor. Compare within-round margins, never absolute levels across rounds.

Blind reconstruction against ground truth. To measure what automation can genuinely recover, rebuild what a human reported from raw activity traces with the reporting channel blocked, over the identical window, with inferences explicitly marked as inferred. A different, fresh agent runs the comparison and classifies every item (only-in-report / only-in-reconstruction / match) — and its own noise deserves the same measurement as any grader’s. The design surfaces both what the human under-reports and what the machine structurally cannot see.

Independent parallel runs. The same task walked by two agents — ideally on different models, which requires a second agent CLI — blind to each other, compared only after both close; cross-referencing mid-flight contaminates convergence. A third pass reconciles and decides what each run contributed; the overlap of the runs is the high-confidence core.

Counterfactual cage. When the incumbent story may have captured the analyst, hand a fresh agent a frozen data packet as its entire universe — no channels, no prior hypotheses — and let it decompose the problem from scratch. It will also recompute and audit the packet’s own claims. Give the cage the shared fact base’s definitions (adversarial-verification.md) — a cage that invents its own metric definitions drifts from every artifact outside it.

Blind-then-compare validation. To validate human-derived conclusions, derive them independently data-first — deliberately ignoring the human’s published framing — and only then reconcile claim by claim. The ordering is what makes agreement evidence; when nothing mechanically fences the analyst from the published framing, treat the result as weaker than the fenced protocols’.

The duel. With-artifact vs without-artifact, everything else identical, tokens and wall time per arm on the table — the universal value test (see principles.md).