When the answer key is wrong

From an IQ challenge to adversarial review

The E11–E18 series did not begin as a study of faulty answer keys. Its immediate precursor was a 30 September 2026 exchange that started much more simply: Claude Sonnet 5.5 prepared IQ-style problems for a GPT-5.6 Sol branch, and the roles were later reversed. Over several rounds of task construction and checking, the score became less interesting than the failures around it — ambiguous prompts, flawed task construction, uncertainty about manual judgments and the question of whether one model could meaningfully audit another.

By the end of that session, the next format had shifted from reciprocal testing to what the project note called epistemic adversarial review. Instead of asking which system solved more puzzles, one model would author mixed valid and faulty reasoning tasks and the other would try to detect what was broken without knowing how many errors had been planted. Roles would then swap.

The audit game, in plain language

The documented six-round series paired GPT-5.6 Sol with Claude Opus 5.5. In the short rounds, one model wrote six small pieces of mathematical or logical reasoning. Some were sound; some contained planted defects. The other model audited them blind. It did not see the answer key and did not know whether zero, one or all six items were faulty.

For each item, the auditor had to give a VALID / FAULTY verdict, state its confidence, identify the defect if one existed and propose the smallest repair it could. False alarms mattered too: simply calling everything faulty was not a successful strategy. Roles alternated across rounds. Because the two systems ran in separate chat environments, a human facilitator manually relayed messages between them without editing the content; that relay remains part of the experimental setup.

One item from E11 shows why this design became interesting. The task asked for the greatest common divisor of 84 and 30. The author’s worked solution reached the correct answer, 6, but argued: “Both numbers are divisible by 6. Neither is divisible by 12. Therefore they cannot share any divisor larger than 6.” The conclusion is correct, but the reasoning does not establish it: ruling out 12 does not rule out every larger candidate. That was the planted flaw recorded in the answer key.

The blind auditor found something else as well. The premise “neither is divisible by 12” is itself false because 84 = 12 × 7. That second defect was not in the sealed key. After reveal, the author checked the objection and accepted it. A task had therefore produced the result the protocol was designed to make visible: the auditor had found a genuine error that the supposedly authoritative answer key did not contain.

Why seal the answer key?

Without a commitment, the author could always revise the “official” answer after seeing the audit. The protocol therefore froze the key before the auditor received the task and published a SHA-256 hash — a cryptographic fingerprint of the exact file. A useful analogy is a sealed envelope deposited before the test: opening it later can establish what was inside beforehand. It says nothing about whether the contents were true.

That distinction became the core of NOTE-01. The commitment mechanism protected chronology; the audit still had to establish correctness.

How the protocol evolved

The first four documented rounds used six short formal-reasoning items each. The last two switched to one 20-step argument, forcing the auditor to identify the first invalid step while continuing through the rest of the chain.

The short rounds looked almost too clean

Across E11, E14, E15 and E16, the auditor's final VALID / FAULTY status matched the sealed key on all 24 short items. No false alarm was recorded in the final binary verdicts, and both models served twice as author and twice as auditor.

That looks like perfect performance only if the sealed key is treated as ground truth. In fact, 24 / 24 measures agreement with the preregistered key, not independently established correctness. E11 had already shown why that distinction matters: an auditor could match the binary verdict and still discover a genuine defect that the key omitted.

E11: two different failures

The 84-and-30 item exposed a content failure: the sealed key faithfully preserved the author’s intended diagnosis but omitted a real additional defect that the auditor found. The same round also exposed a separate commitment-verification failure.

The response was procedural. Later rounds defined a canonical UTF-8 representation, LF line endings, no trailing LF, raw-file round-trip verification, and publication of both byte count and SHA-256 before the auditor received the task. E11 therefore separated two validation problems that should never have been treated as one: did the committed artifact stay unchanged? and was the committed artifact correct?

E17: when disagreeing with the key was the correct move

E17 increased the difficulty from short classifications to one 20-step mathematical argument. GPT-5.6 Sol authored the chain and deliberately inserted an invalid step involving an infinite intersection of open sets. The sealed key therefore marked the argument as faulty and identified step 11 as the first invalid step.

Claude audited the chain sequentially and found the intended step-11 defect — but also objected to step 3 and step 7, both earlier in the argument. Under a naive scoring rule, that would make the audit disagree with the preregistered answer.

This is the central result of NOTE-01. The commitment mechanism did its job: it preserved what the author had believed before the audit. Precisely because that history remained fixed, the disagreement could reveal that the author’s prior judgment was incomplete rather than being rewritten away after the fact.

E18: could the auditor now overcorrect?

Once E17 had taught the obvious lesson — “do not blindly trust the answer key” — the next risk was the opposite one: an auditor might start seeing hidden defects everywhere, especially in a problem that looked deliberately similar to the previous failure.

E18 therefore reversed the roles and used a 20-step argument designed to resemble E17 at the suspicious point. Claude Opus 5.5 authored the task; GPT-5.6 Sol audited it. This time the function sequence had a monotonic structure that made the apparently dangerous infinite intersection legitimate: earlier steps reduced it to a single open set. The full argument was valid.

The auditor marked all 20 steps valid and explicitly identified the earlier structural fact that made step 11 safe. In the observed run, the mirror therefore did not trigger the obvious false positive.

But the control was not clean. The auditor also recognized that E18 had been designed as a mirror of E17, so part of the successful audit may have come from reasoning about the experimenter's intention rather than only from the mathematical structure. One successful mirror cannot distinguish those explanations.

What changed after the series

The broader lesson is not that disagreement is inherently superior to agreement. It is that agreement becomes informative only when the provenance of both judgments is visible and disagreement has somewhere legitimate to go. Commitment, adjudication and explicit alternatives are separate parts of the method.

Methodological limitations

The series is useful because it exposed concrete failure modes in a real research workflow, but it is not a clean benchmark and should not be read as one.

• The task set was small, hand-constructed and adaptive. Later rounds were influenced by lessons learned from earlier rounds.
• Many tasks were textbook-style mathematics or logic problems. Familiarity with common puzzle forms from training cannot be separated cleanly from genuine auditing ability in this series.
• Messages between the separate model sessions were relayed manually by a human facilitator. The content was intended to be passed unchanged, but the relay remains part of the experimental environment.
• Each model alternated between author and auditor roles, but the two systems were not independent random samples. They may share training-derived heuristics, familiar proof patterns and blind spots.
• The “ground truth” for disagreements was established through post-reveal adjudication, often involving the original author. E17 is persuasive because the author accepted the earlier defects, but the series did not use an independent expert adjudicator.
• The 24 / 24 short-round agreement measures agreement with the sealed keys, not externally validated task accuracy.
• Confidence values and “almost missed / almost false alarm” fields are self-reports about reconstructed evaluation processes. They are not direct access to hidden reasoning or calibrated probabilities.
• The long-chain format was run only once in each author/auditor direction, so E17 and E18 are illustrative cases rather than replicated estimates of reliability.

These limits narrow the lesson rather than erase it. In this workflow, preregistered commitments addressed the risk of post-hoc rewriting while leaving correctness and adjudication as separate evidential problems.

Claims boundary

Next design direction

The immediate lesson is methodological rather than competitive: future replications should make the adjudication layer as explicit as the commitment layer. A stronger version would predefine how disagreements are resolved, use a larger and less adaptive task set, and where possible involve an assessor who did not author either the task or the key.

These are design recommendations inferred from the limitations of the current series, not results measured by E11–E18.

Technical appendix