AlphaGo’s Creator: LLMs Still Lack True Reasoning

The Legendary Move That Changed Everything

March 2016, Four Seasons Hotel in Seoul.

Midway through Game 2 of the historic human–AI Go match, AlphaGo placed a black stone on the fifth line — an unexpected, seemingly illogical move. Commentators were stunned; some even suspected a bug. In professional Go intuition, the move appeared to gift territory to the opponent.

What followed became legend: AlphaGo won that game — and the series 4–1 against Lee Sedol. Afterward, Lee reflected: “I thought AlphaGo was just a probability-calculating machine. But seeing that move, I changed my mind. AlphaGo must possess creativity.”

That was Move 37 — now synonymous with AI’s first leap beyond statistical pattern matching.

A Decade Later: The Unlearned Lesson

Ten years on, Thore Graepel — one of AlphaGo’s core architects and longtime DeepMind researcher — delivers a sobering assessment: “The capability behind Move 37 remains absent in today’s LLMs.”

Graepel recently left DeepMind to found a startup focused on building machines with verifiable, auditable reasoning — not just fluent language generation.

His provocative MIT Technology Review article, titled “Don’t Be Fooled — LLMs Don’t Reason,” has ignited global debate. Turing Award winner Yann LeCun publicly endorsed its central claim: “True reasoning requires search. LLMs lack this capacity.”

Move 37 Wasn’t Intuition — It Was Search-Driven Reasoning

Contrary to popular myth, Move 37 wasn’t a flash of “machine intuition.” Graepel clarifies it was the product of structured reasoning, enabled by two tightly coupled systems:

• Policy Network: Learned from human games — fast, intuitive, probabilistic (what would a pro play?).
• Monte Carlo Tree Search (MCTS): Explicitly built and evaluated thousands of future board states — slow, deliberate, evidence-based.

Crucially: the policy network assigned Move 37 a ~0.01% likelihood — essentially dismissing it. Only MCTS, through deep exploration, revealed its superior long-term outcome.

Graepel frames this using Daniel Kahneman’s dual-process theory:
– System 1: Fast, associative, effortless — the policy network.
– System 2: Slow, sequential, self-monitoring — the search engine.

AlphaGo succeeded because it integrated both. Neither alone could produce Move 37.

LLMs: A Supercharged System 1 — With No System 2

Modern LLMs operate fundamentally differently:

• They predict the next token — repeatedly, rapidly, associatively.
• Even chain-of-thought (CoT) reasoning is still token prediction, just extended over more steps.
• There is no independent reasoning module, no explicit state tracking, and no mechanism for revising beliefs based on evidence.

Graepel identifies three critical deficits:

1. No auditable cognitive state: No transparent record of hypotheses considered, confidence levels, or open questions.
2. No separation of knowledge and reasoning: All knowledge is baked into weights — no explicit belief system.
3. Post-hoc rationalization: CoT traces are often retrofitted narratives, not faithful logs of actual inference.

Why Does Verifiable Reasoning Matter?

Fluency suffices for casual chat — but not for high-stakes domains:

• 🏥 Healthcare diagnosis
• ⚙️ Engineering safety validation
• 🔬 Scientific discovery

When errors occur, we need to trace where reasoning failed: wrong assumption? flawed logic? invalid evidence? An LLM’s plausible-sounding story offers no audit trail — undermining trust and accountability.

Graepel’s vision? A “steroid-enhanced scientific method” for machines:

• Maintain a dynamic, inspectable cognitive state (like AlphaGo’s game tree).
• Treat reasoning as actions that update beliefs — only when supported by evidence.
• Use LLMs as intelligent assistants: proposing hypotheses, calling tools, evaluating claims — while a separate reasoning controller governs validity.

The Debate Heats Up

Graepel’s thesis sparked sharp responses:

• David Duvenaud (U. Toronto): “By your criteria, do humans even reason?”
  → Graepel replies: “Paper and pen externalize cognition — turning intuition into auditable process. Tools matter. Structure matters.”

• KingBroken: “Paper is just working memory extension — does it change reasoning capacity?”
  → Yes — because written symbols become inspectable, revisable, shareable evidence.

• visimo-dino: “LLMs already outperform humans in many tasks — why obsess over ‘reasoning’?”
  → Graepel counters: “Can you trust them in medicine? Have they produced novel scientific theories? Would you let them design your chemotherapy?”

• Katherine Moore: “Language itself is the wrong medium for reliable reasoning.”
  → Graepel agrees — and opens a frontier: What formal language could ground real-world reasoning?

The Bottom Line

Bigger System 1 ≠ Emergent System 2.

Scaling language models refines fluency — not deliberation.

Move 37 mattered because AlphaGo held uncertainty, explored futures, and chose against its own intuition.

Humanity needs that same rigor — in drug discovery, climate modeling, and diagnostics. Not stories. Not guesses. Not hallucinated chains.

Auditable reasoning isn’t optional — it’s the next necessary leap.

---

Source: MIT Technology Review — “Don’t Be Fooled — LLMs Don’t Reason”

Article adapted from Machine Heart (WeChat Official Account)