Why a Model Can Grade an Answer But Not Write the Answer Key

Source: Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets

Paper was published on August 02, 2026

This episode was AI-generated on August 4, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs.

A model that judges individual answers almost perfectly will write a test suite that throws out sixty to eighty percent of independently verified correct solutions — and no amount of scale or review fixes it. A new paper shows why enumerating an acceptable set is a structurally different job from judging one candidate, why every review pass makes an answer key stricter but never more complete, and why that matters most when the key is the reward signal in a training loop. You'll also get the one-execution gate and the interpreter-based repair that recover most of the damage.

Key Takeaways

• Why judging one candidate and listing the acceptable set are different tasks — a 20-to-30 point gap that holds across 24x more parameters, four prompts, and two frontier closed models
• The result that rules out 'missing knowledge': asked to write the acceptance rule as executable code, the same models score about 0.99 — above their own judging on identical items
• How model-authored unit tests fail: 70% of 164 one-shot suites run cleanly and still reject the reference solution, encoding 'a promise the spec never made'
• The formal core with real teeth — planted extra entries get caught 71–86% of the time, planted omissions only 10–15% — so every subtractive review pass raises precision and leaves recall untouched
• What the answer-key error costs inside an RL loop: 1.9 accuracy points on the clean causal task, invisible to the loop because it's measured by the flawed key itself
• The steelman that narrows the claim: switch test-time reasoning on and the authoring gap drops to eight thousandths, confidence interval covering zero
• The Monday-morning fix: one execution as a gate, then hand the expected outputs to an interpreter — usable yield up three to ten times, still rejecting over 94% of genuinely wrong code

• 00:00 — The bouncer with a blank clipboard
  The framing metaphor and the stakes: models now author unit tests, rubrics, and RL reward criteria, which turns the answer key from a measurement into the objective.
• 01:47 — Same list, same model, thirty points apart
  How the paper avoids grading model output with models, and the mechanically computable tasks where judging hits F1 0.94–1.00 while listing the same visible items plateaus around two-thirds to four-fifths.
• 04:21 — One decision versus a search with a deadline
  Why token-by-token listing has no calibrated sense of 'done' — and the control experiment where asking for the predicate as code scores about 0.99 at every scale, above the model's own judging.
• 06:54 — A promise the spec never made
  Why a test suite only looks like a rule, illustrated by the HumanEval parenthesis problem where a 14B model invents an error-raising requirement — and the audit showing 70% of suites run clean and reject the reference solution.
• 09:06 — Why review can only make it stricter
  The asymmetry argument: over-inclusions die to a single query while omissions are unobservable even to a perfect judge, backed by planted-error rates and a shaky ten-to-one production dataset the authors themselves refuse to read as a rate.
• 13:06 — What a bad answer key costs a training run
  Two identical RL runs differing only in which key pays out — 1.9 accuracy points across six paired seeds — and why the loop cannot distinguish a wrong policy from a key that didn't know the answer.
• 14:38 — The objection that shrinks the headline
  The steelman: with reasoning enabled, authoring goes from about 0.67 to about 0.98 and the gap effectively vanishes, plus the solution-pool weakness where swapping model families raises false rejection from one percent to about six.
• 16:49 — Keep the questions, fire the answer key
  The practical fix: a one-execution gate drops false rejection from 58–92% to five percent or less, and interpreter-based repair of wrong expected values raises usable yield three to ten times while still rejecting over 94% of wrong solutions.

Recommended Reading

• Evaluating Large Language Models Trained on Code — Introduces HumanEval and its hand-written oracle test suites — the exact benchmark whose canonical solutions the episode's model-authored suites end up rejecting.
• CodeT: Code Generation with Generated Tests — The optimistic counter-framing the episode is arguing against: model-generated tests used as a filter over candidate programs, which works precisely because agreement is scored across many samples rather than trusting one authored key.
• Large Language Models Cannot Self-Correct Reasoning Yet — Empirical support for the episode's formal claim that added review passes don't recover what the model never produced — self-critique loops shed visible errors without adding missing content.
• Scaling Laws for Reward Model Overoptimization — The canonical treatment of what happens when you optimize against an imperfect proxy reward, giving quantitative context for the episode's 'the reward is born wrong' Goodhart argument.