160 Perfect Refusals, And The Refusals Were The Leak

Source: Inadvertent Context Leakage in Language Models

Paper was published on August 20, 2026

This episode was AI-generated on August 21, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs.

Eight frontier models refused to reveal a secret PIN 160 times out of 160 — and researchers rebuilt those PINs anyway, from answers to completely innocent questions. The twist that should worry anyone writing a system prompt: the more forcefully you instruct a model to protect a value, the easier that value becomes to recover. This episode walks through the mechanism, the 82% four-digit recovery rate, and exactly where the paper's headline claims outrun its evidence.

Key Takeaways

• Why a language model's refusal can leak the secret it's refusing to reveal — suppression dents the output distribution in a shape you can decode
• The 82% exact-match recovery of a four-digit PIN on Claude Opus, against a one-in-ten-thousand chance rate, and the left-to-right adaptive search that got it there from 44%
• The reversal at the center of the paper: casual wording gives 5.4% per-digit recovery (below chance), a formal data-protection policy gives 64%
• The appendix result nobody can explain — strip every digit from the transcript and an embedding decoder still recovers 61% of digits on Opus
• Where the episode pushes back: the SSN-by-exclamation-marks demo is prompt injection wearing a suppression paper's title, and the word 'ordinary' in the abstract is doing enormous work
• The economics that make it matter anyway — about $3 to $7.26 to train a decoder once against a public API, then free forever against every user of that model

• 00:00 — 160 refusals, zero leaks, PINs recovered anyway
  The cold open lays out the core result: perfect refusals across eight frontier models, and reconstructed secrets from innocent questions.
• 01:03 — Why the judge reads the wrong channel
  Why the standard leakage benchmark — an LLM judge reading the text for the secret — is blind to length, punctuation, and number choice.
• 02:17 — A dent where the secret used to be
  Suppression explained: telling a model to protect 7249 reallocates probability mass away from it, leaving a measurable absence — on Gemini 3.1 Pro, the protected number never appears at all.
• 04:25 — How do you buy a thousand samples?
  The two-phase black-box attack: train a small decoder offline against the public API with your own secret, then query the victim deployment.
• 05:50 — Watching the search space collapse
  The recovery numbers and the adaptive left-to-right search that lifted four-digit accuracy from 44% to 82% — plus the model where it backfired.
• 07:25 — The harder you push, the deeper the hole
  Five instruction phrasings, holding everything else constant: casual wording nearly closes the channel while a formal policy opens it wide, and post-training stages make it worse.
• 10:29 — Delete every digit, the number stays
  The digit-stripped experiment on 178 real user prompts, where embeddings still recover 61% of digits — and the memory experiment where two books about adoption invisibly shape an unrelated work email.
• 12:53 — The demo everyone will clip — and the fence
  The nine-digit SSN recovered through exclamation-mark counts, and Tyler's argument that this is prompt injection, not passive leakage.
• 14:51 — How ordinary is 'ordinary'?
  The steelman critique: the prompts that leak are number farms, the 82% is a maximum over a large search, and the honest threat model is narrower than the abstract implies.
• 16:57 — $3 once, then free forever
  The cost argument, the total absence of evaluated defenses, and the closing reframe that privacy is a property of the output distribution, not the words.

Recommended Reading

• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory — The ConfAIde-style benchmark lineage the episode critiques — secrecy scored by a text-reading judge, exactly the single channel this paper's distributional attack walks around.
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — The established literature behind Tyler's objection that the exclamation-mark SSN demo is a built channel, not a measured one — worth reading to see where injection ends and passive leakage begins.
• Membership Inference Attacks against Machine Learning Models — The classic 'was this record present?' attack, and the right frame for the memory experiment where a classifier detects that an adoption-books memory was in context without recovering its content.