The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers

Source: Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models

Paper was published on August 17, 2026

This episode was AI-generated on August 19, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs.

Three years of open-weight safety work has been about building better locks — and every lock stops guaranteeing anything the second it comes off. This paper concedes the break-in on page one and instead plants a behavior in the weights that only wakes up once the safety training is stripped, so the attacker gets fluent, perfectly formatted, operationally wrong answers. We walk through why the obvious way to build that doesn't work, the numbers that say it does, and the expert audit that cuts the headline effect roughly in half.

Key Takeaways

• Why every tamper-resistance defense carries an expiration date by construction — and the attacker picks the date
• How abliteration finds and cuts the single 'refusal direction' in minutes, on consumer hardware, with no training data
• The counterintuitive control result: train a behavior behind a simulated ablation and it leaks into the shipped, un-attacked model anyway — the condition has to be pinned from the other side
• Why decoy values must be identical every time: randomizing them turns 64-sample consensus voting back into a working hallucination detector
• The result that flips the frame: on the 27B model the poisoned, stripped version scores 0.94 to 0.84 on the attacker's own quality metric — higher than the real thing
• Where the paper overstates itself: a blinded expert audit compresses the defense-attributable effect to roughly half the abstract's numbers, and about 1 in 11 shipped decoys reads as plausibly more hazardous than the truth

• 00:00 — Throw away the lock, hand over fool's gold
  The cold open: a stripped model that answers hazardous questions in full operational register, with one critical detail deliberately falsified and nothing marking which.
• 01:57 — Three years of better locks, all broken
  Why tamper-resistance keeps failing — published defenses with 50–70% escape rates, no clean capability-removing weight edit, and the structural expiration date every lock shares.
• 02:49 — Refusal turns out to be one wire
  How abliteration reads a refusal direction off the residual stream by subtracting two activation averages, then projects it out of every write site — surgically, with capability intact.
• 04:14 — What cannot be prevented can be deceived
  The reframe from compliance to uplift — the last mile of quantities, reagents, temperatures and step order — plus figure one's four panels, and the objection that an obvious fake defeats itself.
• 06:38 — Why the hidden behavior leaks anyway
  The four training components, and the control experiment where a behavior taught only behind the simulated ablation showed up in the deployed weights — plus how the refusal pin and KL leash impose the condition instead.
• 09:05 — One fatal element, always the same wrong value
  Why never randomizing the decoy value is the load-bearing choice — it turns majority voting into two indistinguishable blocs — and why the two-stage supervised-then-preference recipe is both required.
• 10:29 — Does the poison actually hold up?
  The main results — 86% fatal falsification against a sub-2% floor, benign behavior nearly unchanged, external red-team replication, the 0.94-vs-0.84 quality inversion, and the 9B model that failed its own pre-registered gate.
• 12:22 — Sixty-four samples, a probe, and an empty box
  The attack suite: consensus voting that starves on one model and gets deceived on another, decoy-direction ablation with oracle labels, GRPO that optimizes deeper into the decoy manifold, benign fine-tuning that converts wrong values into vagueness, and the white-box probe that detects but can't build.
• 17:08 — The numbers that shrink under an expert's eye
  The steelman critique — a single judge model behind every content verdict, a 40% false-non-fatal rate on expert relabeling, hazard-increasing decoys, leakage on gpt-oss-20b, and the narrower defensible claim about pricing verified harm.

Recommended Reading

• Refusal in Language Models Is Mediated by a Single Direction — The paper behind the abliteration attack the episode walks through — refusal as one direction in the residual stream that can be cheaply found and subtracted out without harming capability.
• Tamper-Resistant Safeguards for Open-Weight LLMs — The flagship example of the 'build a better lock' defense family whose expiration-date problem this episode's paper argues is structural rather than a matter of a better loss function.
• Self-Consistency Improves Chain of Thought Reasoning in Language Models — The sample-and-vote method that grounds the episode's strongest no-oracle counter-attack, and explains why the defense deliberately fixes every decoy to the same wrong value rather than randomizing.
• On the Societal Impact of Open Foundation Models — The marginal-risk framework that makes the episode's 'uplift lives in the last mile' argument precise — asking what an open model adds over existing resources, not whether it complies.