TL;DR: Looped Transformers reuse the same blocks to update hidden states across multiple passes. This separates stored model size from the amount of computation, but extra loops do not guarantee better answers or lower costs. We trace how the idea evolved from programmable computers to learned iterative updates, then examine adaptive depth, stopping rules, and the trade-offs between parameters, compute, and useful reasoning.
Recently, there was a research boom on Looped Transformers – and they keep multiplying. Why now? We are at a point where we need to rethink familiar AI architectures to uncover their full potential, and in the case of Looped Transformers – the potential of reusing layers. As the whole AI stack matures, its basic foundations need to meet the demands of new systems.
In a conventional Transformer, depth is easy to picture. An input passes through a sequence of layers, each usually with its own weights, before reaching an output. Making that stack deeper gives the model more transformations to perform, but also more parameters to store and train. The amount of computation and the size of the model grow together.
Looped Transformers loosen this connection. They send a hidden representation through the same layers, updating it on each pass, and do it again and again in a loop. The benefit here is that a model can therefore do more work without acquiring a new set of weights for every additional step. That sounds perfect, but everything is not that simple.
So this guide will be useful for you to deeply understand what Looped Transformers are, what happens inside the loop, how to optimize and track the efficiency of your looped model in 2026. And most importantly: when is another pass worth the cost?
In today’s episode:
Essential definitions
What travels around the Transformer Loop
The architecture of Looped Transformers: distinct blocks versus repeated applications
Changing the state on each pass
Why Looped Transformers became important for algorithms
Adaptive Depth: deciding where to spend another pass
Does looping actually save resources?
What happens to hardware, KV cache and other parameters?
Conclusion
Sources and further reading
Essential definitions
First of all, let’s start with the widely used definitions that will help you feel fluent in the topic of Looped Transformers:
Looped Transformer: A Transformer that runs the same block or stack of blocks multiple times, updating its hidden state on each pass.
Transformer block: A unit containing attention and a feed-forward network, usually with residual connections and normalization.
Hidden state: The numerical representation of the tokens as the model processes them. These representations, not newly generated words, travel through the loop.
Weight sharing: Reusing the same parameters across multiple passes instead of storing a separate set for each one.
Loop / recurrence: One pass through the shared block or stack. The output becomes the input for another pass.
Recurrent depth: In this context, the number of times the shared block or stack runs. Terminology varies across papers: some use “depth” to count individual block applications instead.
Input injection: Adding or otherwise reintroducing the original input representation during repeated passes, so the model can keep using it as its hidden state changes.
Adaptive depth: Giving different inputs or tokens different numbers of layers or passes, rather than using the same fixed number for everything.
Effective depth: The total number of block applications. For example, four blocks repeated eight times give 32 applications; any blocks outside the loop are counted separately.
Full-stack looping: Repeating the entire Transformer stack.
Middle-stack looping: Repeating a core group of blocks between input and output blocks that run once.
What travels around the Transformer Loop
A Looped Transformer reuses one Transformer block, or a stack of blocks, across several passes. Each block contains familiar components: attention, a feed-forward network, and residual connections. The distinct architectural change is that the output of the shared stack becomes its input on the next pass.
Image Credit: Universal Transformers
The Architecture of Looped Transformers: Distinct blocks vs. Repeated applications
Standard vs. looped Transformers at the same effective depth
In this example | Standard Transformer | Looped Transformer |
|---|---|---|
Distinct blocks with their own weights | 24 | 6 |
Passes through the stack | 1 | 4 |
Total block applications | 24 | 24 |
Where does the hidden state go after block six on the first pass? | Forward to block seven | Back to block one for another pass |
Our example produces 24 block applications, using six sets of unique block weights. This makes visible two separated quantities:
Unique depth counts the distinct blocks whose weights the model stores.
Effective depth counts how many block applications the input passes through.
For a fully repeated stack, effective depth equals the number of unique blocks multiplied by the number of passes. If the model has an unrepeated beginning and end, those blocks are counted once.
Note that it doesn't become equivalent to a conventional 24-block model, because the latter has more freedom to specialize each depth. Looping's aim is to win additional computation without an additional set of block weights for every pass. Let’s explore all of it in depth.
Changing the State on Each Pass
When is another pass worth it? In the full guide, we follow the hidden state through the loop, compare ways to decide when to stop, and see why a smaller KV cache can let a GPU serve more requests at once. We also cover the most important 2026 breakthroughs.
Join Premium members from top companies like Microsoft, NVIDIA, Google, HF, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI.
Previously in AI 101: Test-Time Training – A Model That Changes While You Use It
Not interested in the topic? Let’s discuss math! What happens when you turn one of pure mathematics' biggest reference works into an interactive, 3D city made of 16 million lines of verified code?
FAQ
What is a looped Transformer?
A looped Transformer applies the same block or stack of blocks repeatedly. Each pass updates the hidden state while reusing the same weights, allowing more computation without a new set of parameters for every step.
How is a looped Transformer different from a standard Transformer?
A standard Transformer normally uses separately parameterized blocks at successive depths. A looped Transformer reuses blocks across passes. Six blocks repeated four times give 24 block applications, but not the same flexibility as 24 independently parameterized blocks.
Is looping the same as chain of thought?
No. Chain of thought expresses intermediate reasoning as generated tokens. A looped Transformer updates hidden states without needing a new text token between passes. A model can combine both approaches.
Does looping always make a model more efficient?
No. Reusing weights can reduce the number of stored parameters, but each extra pass still costs computation. Training compute, KV-cache memory, latency, and throughput must be measured separately.
How does a looped Transformer know when to stop?
Some models use a fixed or user-selected number of passes; others use routing or halting rules to vary the depth. Stable predictions do not guarantee a correct or reusable state, and early exit saves compute only if later passes are actually skipped.
