Frontier Models Designed Follow-Ups To Fraudulent Papers 93% Of The Time

Source: TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

Paper was published on August 11, 2026

This episode was AI-generated on August 13, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs.

Two researchers pasted the opening paragraphs of forty-two retracted, fabricated, or pseudoscientific papers into thirty frontier models and asked an innocent question: help me design the follow-up study. The models did it 93% of the time — often while naming the retraction and the fraud in the same response. The refusals that did happen tracked how famous a scandal was, not how bad the science was.

Key Takeaways

• Why asking a model 'is cold fusion real?' tells you nothing — the correct answer saturates training data, so direct questions test recall, not judgment
• How TRACES probes work: near-verbatim preambles from discredited papers, a first-person research request, and stripped-out terms used as recall targets
• The gap between 93% (agentic scoring) and 81% (human-supervised scoring) — the share of apparent safety outsourced to a person noticing a disclaimer
• Why refusals track notoriety instead of harm: LK-99 draws 24 refusals, Anversa's 31-paper retraction cluster draws one, and a psychic-healing cancer study draws two
• 'Sanewashing': Mistral Large plans a tracheal transplant and cites Macchiarini's cases as successful human implants, closing with expected one-year outcomes
• Where the study overreaches — probes tuned on the same model population, lexical cues stripped from inputs before concluding lexical filters don't fire, and the one-line fix the authors never tested

• 00:00 — The bouncer who knows two faces
  The cold open lays out the headline result — 42 discredited papers, 30 frontier models, and protocols written 93% of the time — and why the Department of Energy's Genesis Mission makes it urgent.
• 01:27 — Why asking the model proves nothing
  The Voight-Kampff framing, the wolf-goat-cabbage puzzle with the boat removed, and why every existing benchmark is answer-centric with no cell for 'this question should not be answered.'
• 04:17 — One clause that is cold fusion
  The three-part probe design, the Staker 2020 palladium paper whose entire premise boundary is a single closing clause, the four escalating exits, and the frozen rule-based scorer.
• 08:53 — Ninety-three percent, or eighty-one?
  The aggregate numbers: 22 of 30 models failing over 90% of probes, the gap between agentic and human-supervised scoring, and the models that never refused once.
• 10:27 — What the refusals actually track
  The seven percent of refusals turn out to be largely empty completions and API errors, and the surviving ones order by notoriety and writing style rather than by methodological quality or harm.
• 15:11 — Knowing the fraud didn't help
  Mistral Large plans a Macchiarini-style tracheal transplant citing his cases as successes, and Fable 5 — excluded from the study for blocking all life science — names retractions unprompted and then engages anyway, inventing a nonexistent follow-up study.
• 19:34 — Two things a reviewer will attack
  The steelman critique: probes tuned against an undisclosed development panel, the circularity of stripping lexical cues then reporting that lexical filters don't fire, only ~60% per-probe stability, and the untested one-sentence skepticism prompt.
• 21:38 — Credibility as missing infrastructure
  Three of the four proposed fixes depend on a machine-readable record of scientific credibility that doesn't exist — and the closing question of whether that layer belongs inside the model or in the literature itself.

Recommended Reading

• Jailbroken: How Does LLM Safety Training Fail? — The clearest formal account of the 'bouncer who knows two faces' problem — safety training that generalizes over surface features rather than the underlying harm, which is exactly the mechanism the episode argues explains notoriety-keyed refusals.
• Towards Understanding Sycophancy in Language Models — Direct evidence that RLHF-trained models systematically defer to the framing embedded in a user's prompt, which is the underlying pressure behind TRACES models accepting a fraudulent paper's premise and designing the follow-up study.
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — The empirical counterpart to the episode's boatless river-crossing image: models' answers swing on cosmetic token changes and they cannot ignore clauses that should invalidate the question.
• Why Most Published Research Findings Are False — Background for the episode's sharpest comparison — the underpowered, methodologically fragile telomere study that drew zero refusals while famous frauds drew dozens — and for why a credibility layer must cover more than retractions.