The Model Built a Perfect Map of the Puzzle, Then Lost It

Source: Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking

Paper was published on August 07, 2026

This episode was AI-generated on August 10, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs.

A reasoning model forms a near-perfect internal map of the Tower of Hanoi in its first eight layers — and then spends nineteen thousand tokens of careful thinking losing it. Researchers cracked open a 27-billion-parameter model, watched the representation decay in real time, and pushed it back mid-generation: optimal solutions jumped from 41% to 73% with no retraining. It's the first mechanistic answer to a year-old fight about whether reasoning models actually reason — and a directional bet against 'just let it think longer.'

Key Takeaways

• Why the classic Tower of Hanoi is finished as a benchmark — models have memorized the recursive template, and one open model scores 25 out of 25
• The Sierpiński triangle hiding inside a six-layer network that was only ever shown move sequences — and why a probe for 'is the fact there' misses it entirely
• The 'one folded map versus four dials' distinction: why per-disk readouts throw away exactly the distance information planning depends on
• A 27B reasoning model and a toy 6-layer model encode the board with the same fidelity (~0.94 correlation) — and the big one builds it by layer 8 of 64, then degrades for the next 56
• Why the 41%→73% intervention proves less than it sounds: it requires an external symbolic tracker doing the very job the model is failing at
• Where the same intervention fails outright — DeepSeek fixes 6 of 72 failures, and the authors admit their explanation was constructed after the fact

• 00:00 — The map it had, then didn't
  The cold open lays out the paradox: a model encodes the puzzle perfectly before speaking, loses it while reasoning, and recovers when the representation is handed back mid-answer.
• 01:01 — Everyone argued behaviorally, nobody opened the model
  How 'The Illusion of Thinking' set off a year of behavioral rebuttals, and why the authors first had to replace a saturated benchmark with scattered flat-to-flat puzzles.
• 03:37 — A fractal nobody told it about
  The authors grind their lens on a six-layer model trained from scratch on 81 board states, then discover a Sierpiński triangle emerging in its internals across training epochs.
• 07:10 — One folded map, or four dials?
  The board state isn't lost during move emission — it's reformatted from a blended geometry into four near-independent per-disk channels, giving the researchers two instruments that can disagree.
• 08:57 — Eight layers to build it, fifty-six to lose it
  Pointed at real reasoning models, the probes find a 0.94 correlation and 100% nearest-state retrieval at the end of the prompt — then a collapse to near-chance at the moment of commitment.
• 12:26 — Can you hand the map back mid-answer?
  The causal test: caching clean prompt-time activations for all 81 states and nudging the model toward the right one during generation, with a strength dial that shows a clear sweet spot.
• 14:16 — The passenger doing the driver's job
  The steelman critique — the intervention requires an external symbolic tracker computing the current board, DeepSeek barely improves, and 81 points in a 5,000-dimensional space is a permissive fit.
• 16:59 — Bigger models, or better notebooks?
  Why the diagnosis — failure to maintain a representation rather than failure to form one — points toward state maintenance and scaffolding instead of longer thinking budgets.

Recommended Reading

• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity — The Shojaee et al. paper this episode is arguing with — the Tower of Hanoi collapse results and the shrinking reasoning traces that prompted the mechanistic investigation.
• Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — The Othello-GPT paper that established the 'train a small model on sequences, probe for a board state it was never told about' methodology the episode's toy Hanoi transformer inherits.
• Emergent Linear Representations in World Models of Self-Supervised Sequence Models — Nanda et al.'s follow-up showing Othello-GPT's board state is linearly readable and causally editable — the direct precedent for the episode's linear probes and activation-steering intervention.
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Pairs with the episode's core puzzle — a reasoning trace that reads fluent all the way down while the model's internal state has already come apart.