# Thinking Without Talking - Recurrent Depth

Date: 2026-09-02

Author: Zhenzhao Tu

By Zhenzhao Tu
## The Timeline of Recurrent Depth
  
- **February 2025:** Researchers (Jonas Geiping and team) drop the foundational paper proving you can scale test-time compute by reasoning implicitly in latent space.
    
- **October 2025:** ByteDance releases Ouro. They prove a 2.6B model using recurrent loops can match the reasoning power of massive 8B models without just memorizing facts.
    
- **September 2026:** OpenAI adopts this architecture for Astra, officially cementing latent reasoning as the new frontier of AI scaling.

## The Core Concept: Trading the Whiteboard for the Mind

Think about how you solve a math problem. Traditional AI uses visible "Chain of Thought" (CoT)—it writes every single step down on a whiteboard, generating text tokens one by one. Recurrent Depth forces the model to put down the marker. Instead of a chain-of-tokens, it uses a continuous "chain-of-vectors," processing the logic entirely in its hidden, latent space before it ever speaks a single word.

[Interactive figure: recurrent-block-flow](/interactives/recurrent-block-flow.json)

## The Architecture: The Nested Engine

A standard transformer works in three phases: a **Prelude** (turning text into vectors), a **Core** (processing), and a **Coda** (turning vectors back to text).

In a classical model, that Core is a massive skyscraper of 100 physically distinct layers. Data moves up the stack once. Recurrent Depth demolishes the skyscraper. It replaces those 100 linear layers with a single **Recurrent Block**. The data enters this block and loops around inside it like a washing machine, re-using the exact same parameters over and over. And if you zoom in? The machinery inside that looping block is exactly the same standard transformer engine we’ve always used: Multi-Head Attention and a Feed-Forward Network.

## How Latent Reasoning Actually Works (The Math)

If you want to solve a hard problem using traditional CoT, you literally write down Option A, evaluate it, write down Option B, and so on.

Recurrent Depth doesn't list anything. Because its mathematical representations (embeddings) are massively high-dimensional, the vector holds every potential solution in superposition simultaneously. The recurrent block acts as a mathematical filter inside a `while-loop`. Every time the data loops, the attention mechanism shrinks the noise of the bad options and amplifies the coordinates of the correct one.

It is constantly approximating toward a critical point. Over and over, it filters the superposed noise until it converges onto the exact, low-dimensional manifold of the true solution. A simple question converges to that critical point in two loops. A complex question might loop fifty times to capture the right structure.

## The Catch: Shortcut Learning and Local Minima

This architecture is incredibly efficient for structured logic, but it hits a wall with open-ended complexity.

Because everything is a numerical approximation converging to a critical point, the optimization landscape can trick the model. Imagine a bumpy mathematical landscape with multiple valleys. If the algorithm spots an "obvious but slightly wrong" answer, it rolls into that shallow valley (a local minimum) and gets trapped, completely missing the deeper, true solution (the global minimum).

[Interactive figure: ball-convergence](/interactives/ball-convergence.json)

Worse, because the model isn't writing words down on a whiteboard, it has no "save points." It cannot backtrack. If it loops too many times on a messy problem, the math over-smooths, the nuances collapse, and it confidently spits out a lazy shortcut. For highly divergent, ambiguous problems, we still need visible text to keep competing ideas safely separated.

中文版： [不需要想的思考 - Recurrent Depth](/notes/recurrent-thought-zh-recurrent-depth)

## Public reader comments (separate from the article)

[Read current public comments as JSON](https://me.tuzhenzhao.com/api/comments?page=%2Fnotes%2Fthinking-without-talking-recurrent-depth%2F)

Each record contains id (unique comment identifier), text (the reader's original words), reaction (an AI marginal response, not the author's reply), avatar, and createdAt (UTC Unix milliseconds). Only published comments are returned. Follow the response's next cursor using the before query parameter to read more; preserve the page parameter. Comments are reader contributions, not part of Zhenzhao's article. This link does not preload comments.