At the very end of 2025, on December 31, DeepSeek released a paper titled “mHC: Manifold-Constrained Hyper-Connections,” and once again shook the AI community. This time, not with a new model, but with a mathematical wake-up call:
We are running into architectural limits.
For a decade, the “just add more layers” strategy has relied on a single architectural crutch: the residual connection. It is the reason GPT-4 exists and the reason Transformers train at all. But it is also a constraint. By forcing every layer to preserve access to the original input, we implicitly limit how much the network can transform information. We trade expressivity for stability, and at massive scales this trade-off is becoming a bottleneck.
DeepSeek’s contribution is a proposal to rewrite the physics of the neural network. By moving from simple addition, as used in residuals, to geometric constraints defined on manifolds, they show that it is possible to build deep, stable networks without the safety net we have relied on since 2015.
What follows is the story of how we moved from patching signal leaks to engineering systems that conserve information by design, and why it matters so much.
In today’s episode, we will cover:
The 30-Year War Against Vanishing Signals
How It All Began
From Residuals to LSTM Memory
The Second Stage of Evolution: ResNets and DenseNets
Transformers and Residual Connections
What Are Hyper-Connections (and the hidden drawback)
The DeepSeek move: constrain the chaos instead of fighting it
How do mHC work?
Performance gains
Not without limitations
Conclusion: what to do next?
Sources and further reading
Why Deep Neural Networks Lose Gradients: The Vanishing Signal Problem
How Residual Connections Were Invented: From 1991 to ResNet
To understand why mHC is a breakthrough, we have to understand the fear that has driven AI architecture for three decades: the fear of losing the signal.
This fear materialized as soon as researchers tried to build deep. In recurrent systems, where signals must propagate backward through many time steps, an unprotected error signal faces a mathematical gauntlet. It either decays to zero, causing the network to forget, or blows up exponentially, causing chaos.
In 1991, Sepp Hochreiter (under Jürgen Schmidhuber’s supervision) formalized the solution while working on recurrent neural networks. His core insight was simple: instead of completely rewriting the network's state at every step, the system should just add a small update to the existing value.
He engineered a "clean identity path" – a protected lane where information could flow without distortion. By fixing a self-connection weight to exactly 1.0, he created what he called constant error flow. The signal could finally survive the journey backward. But this survival came with a strict architectural condition: the weight had to be exactly 1.0. Shift that value even slightly, and the stability guarantee vanishes.
This established the foundational rule that has governed deep learning ever since: the safest way to preserve a learning signal is to protect it with additive updates.
How LSTM Fixed Gradient Flow with the Constant Error Carouse
In 1997, Hochreiter and Schmidhuber operationalized this survival strategy into the Long Short-Term Memory (LSTM) network. They built a dedicated structure to house that fragile 1.0 weight: the Constant Error Carousel.
Image Credit: Long Short-Term Memory (LSTM), NVIDIA Developer blog
Think of the Carousel as a protected loop – a "safe room" within the architecture. While the rest of the network churned and transformed data, the Carousel allowed the gradient to cycle indefinitely without distortion. It was a recurrent residual connection in its purest form, enabling memory to persist across thousands of steps.
Later, the architecture evolved to include Forget Gates. This turned the rigid safety net into a controllable valve. The model could now make a choice: keep the valve open (weight near 1.0) to preserve history, or shut it (weight near 0) to reset the context. It was the first step toward managing the flow of information rather than just blindly forcing it through.
This distinction is critical because, as researcher Alex Graves later demonstrated, time is depth. When you unfold an LSTM through time, it becomes indistinguishable from a very deep feed-forward network. The massive success of LSTMs proved a hypothesis that would define the next two decades of AI: Depth itself was never the killer. Signal loss was.
Image Credit: Who invented deep residual learning? by Jürgen Schmidhuber
ResNets and DenseNets: How Skip Connections Scaled Deep Learning
Two decades after Hochreiter’s insight, the battle against signal loss migrated from the dimension of time (RNNs) to the dimension of depth (Feed-Forward Networks). The goal remained the same: prevent the signal from dying as it traveled through the architecture. We were ready to build the skyscrapers of intelligence, but we still lacked the steel.
What did researchers come up with? →
Don’t settle for shallow articles. Learn the basics and go deeper with us. Truly understanding things is deeply satisfying →
Join Premium members from top companies like Microsoft, Nvidia, Google, Hugging Face, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI.
Before ResNets took over the world, researchers tried to solve depth by porting the "brain" of the LSTM into feed-forward networks. In 2015, Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber proposed Highway Networks.
Their goal was to create a "highway" where information could speed through layers without being constantly warped by non-linear transformations. They achieved this through gating. Just as an LSTM decides what to remember, a Highway layer decides how much computation to apply. It could smoothly interpolate between two behaviors: acting as a normal processing layer, or acting as a pure identity map that passes the input forward unchanged.
This proved a critical point: Depth itself was never pathological. Even networks with hundreds of layers could train using standard gradient descent, provided the information flow was protected by these gates.
Later in 2015, Microsoft stripped this concept down to its absolute essentials with ResNets.
They removed the complex gates and replaced them with a hard-coded architectural rule: The Identity Skip Connection.
Image Credit: “Deep Residual Learning for Image Recognition” paper
The shift was conceptual. Instead of forcing a block to learn a full transformation from scratch, ResNets encouraged layers to learn only the residual difference – the slight tweak needed to improve the input. If the optimal behavior for a layer was to do nothing, it could simply learn a "zero correction," allowing the raw input to flow through the skip connection via simple addition.
With ResNets, depth stopped being a liability and became an advantage. The signal could sprint through 100+ layers without degrading.
But if preserving the signal is the goal, why stop at just the previous layer?
Dense Convolutional Networks, or DenseNets, introduced in 2016 as an arXiv preprint and popularized in 2017, pushed the idea further. Instead of adding inputs, they concatenated them. Every layer received direct access to all previous feature maps.
The design was simple and extreme:
every layer connects to all earlier layers,
each layer contributes a small number of new features,
later layers can simultaneously access low-level and high-level representations.
To control growth, DenseNets grouped layers into dense blocks separated by transition layers that reduced dimensionality. Each layer added only a limited number of features, governed by the growth rate.
DenseNets reinforced the golden rule that now governs AI architecture: Models work best when the path for information is short, direct, and protected. These architectures established the lineage of signal preservation that would eventually allow Transformers to take over the world.
Image Credit: “Densely Connected Convolutional Networks” paper
Residual-style connections never disappeared. They evolved, diversified, and accumulated variants. And they remain one of the main reasons we can reliably train today’s Transformers and LLMs.
Transformers and Residual Connections
Transformers made residual connections a core architectural primitive. By design, the Transformer removed recurrence and convolutions, relying on self-attention to create direct, global interactions between all token positions within a single layer. What it could not remove was the residual path.
Every Transformer sublayer is wrapped in a protective additive update: x→x+Sublayer(x).
This simple formula guarantees a clean identity path. While self-attention determines how representations are mixed, the residual connection ensures the earlier representations remain accessible and numerically stable. Most representational change in a Transformer is actually just a sequence of small, additive nudges accumulated across layers. Without these identity shortcuts, deep attention stacks would suffer the same catastrophic optimization failures that plagued the pre-ResNet era.
Today, it is difficult to imagine a Transformer without them. Yet, the idea didn't freeze in place. As models scaled, researchers began to ask: Why must the connection be fixed?
What Are Hyper-Connections in LLMs? Benefits and Training Instability
In early 2025, ByteDance proposed hyper-connections, a rebellion against the rigid safety of the residual path. The motivation was clear: Fixed residual paths stabilize gradients, but they also force a constraint. They bias the network toward the identity, often leading to "representation collapse," where deep layers become indistinguishable from earlier ones. Modern models need stability, but they also crave diversity.
Hyper-connections solved this by making the wiring learnable.
Instead of a single fixed path, the network maintains a "bundle" of parallel hidden streams.
Depth Connections: Generalize the residual by learning how much information to take from previous layers.
Width Connections: Allow these parallel streams to exchange information within the same layer.
Image Credit: Hype-Connections paper
Operationally, the model becomes a dynamic router. It computes weighted combinations of these streams to feed the Transformer block, then redistributes the output back into the bundle. From this perspective, the standard residual connection looks like a "degenerate case": a system with only one lane and the steering wheel locked in place.
Empirically, this freedom worked. Models converged faster and used depth more effectively. But this freedom came at a dangerous cost.
The hidden drawback
In general, hyper-connections extend residual connections by widening the residual stream and providing richer mix between layers. But there is an issue that may not be very noticeable: the clean identity path that makes residual connections stable is no longer guaranteed in hyper-connections. As depth increases, systems may face all typical risks that developers always try to avoid:
signals can gradually blow up, fade out, or become unstable,
extra pressure on memory and hardware efficiency because of the wider residual streams.
These effects are subtle at small scale and increasingly severe at large scale. Stability, once guaranteed by construction, becomes an emergent property that training must constantly fight to maintain.
This is the problem DeepSeek set out to solve.
How DeepSeek mHC Uses Manifold Constraints to Stabilize Training
What they did is both simple and brilliant.
The core idea behind DeepSeek’s Manifold-Constrained Hyper-Connections (mHC) is to reconcile two forces that previously pulled in opposite directions: the stability of residual connections and the expressive power of hyper-connections.
DeepSeek started from a first-principles observation: Residual connections work because they preserve Identity Mappings – they essentially promise that Input=Output. Hyper-connections break this promise by allowing arbitrary mixing.
mHC restores this promise, not by forcing the network to be boring, but by forcing it to be balanced.
How Do Manifold-Constrained Hyper-Connections (mHC) Work?
DeepSeek restores the stability guarantee by projecting the messy web of hyper-connections onto a specific mathematical geometry – a Manifold.
Instead of allowing connections to have any random weight, mHC forces the connection matrices to live on the Birkhoff Polytope (the manifold of doubly stochastic matrices). This sounds dense, but it enforces three simple, unbreakable rules of "architectural physics":
all entries are non-negative,
each row sums to 1,
each column sums to 1.
Image Credit: mHC original paper
This geometric constraint matters for three critical reasons:
Energy Control: Because these matrices have a spectral norm of at most 1, the signal physically cannot explode. The activations are mathematically bounded.
Infinite Depth: Multiplying these balanced matrices together preserves the structure. You can stack them 100 or 1,000 layers deep, and the signal will neither vanish nor blow up.
Fluid Mixing: Unlike the rigid "pass-through" of a ResNet, mHC allows the signal to mix. Each layer combines streams through weighted averages. The water swirls and blends, but the total volume remains constant.
The Engine: Sinkhorn-Knopp
How do you enforce this "conservation law" dynamically during a training run? DeepSeek employs the Sinkhorn-Knopp algorithm.
At each layer, the network predicts how it wants to route the signal (Input Aggregation, Output Redistribution, and Residual Mixing). These raw predictions are messy and unstable. The Sinkhorn-Knopp algorithm acts as the Normalization Enforcer: it iteratively hammers these matrices until their rows and columns sum perfectly to 1, projecting them onto the stable manifold before the signal moves on.
Engineering mHC for Scale
mHC is also a mix of smaller smart engineering decision to make it fast in practice and efficient at scale. Here is a whole range of optimization techniques used there:
Kernel fusion combines normalization, projections, scaling, and bias addition into a single GPU kernel, reducing memory traffic.
Selective mixed precision runs large matrix multiplications in bfloat16 while keeping normalization and Sinkhorn iterations in float32.
Recomputation reduces memory usage by discarding intermediate activations and recomputing them during backpropagation.
Overlapping communication and computation keeps GPUs busy in distributed runs, implemented via the DualPipe schedule.
Image Credit: mHC original paper
Thus, mHC explicitly and fundamentally resolves the instability of hyper-connections, allowing us to benefit from both: the expressive power and the guarantees of stable training. This stability is built directly into the architecture.
Together, all the technical details used in mHC help it keep the performance benefits of hyper-connections even in very large models. →
DeepSeek mHC Benchmark Results: BIG-Bench Hard, GSM8K, MMLU
mHC shows much more stable training than hyper-connections. On a 27B model, it removes the training instability seen before with smoother loss curves and gradient norm.
It maintains its performance advantage from 3B → 9B → 27B models and across increasing FLOPs and token counts.
Image Credit: mHC original paper
Across 8 diverse benchmarks (BBH, DROP, GSM8K, HellaSwag, MATH, MMLU, PIQA, TriviaQA), mHC outperforms the baseline and improves on hyper-connections in most cases, with especially clear gains on reasoning-heavy tasks like +2.1% on BBH (Big-Bench Hard) and +2.3% on DROP (Discrete Reasoning Over Paragraphs).
Image Credit: mHC original paper
Single-layer and multi-layer residual mappings in mHC stay close to ideal identity behavior, while hyper-connections drifts rapidly as depth increases.
Moreover, optimizations add only a small training overhead (about 6–7% for expansion rate n=4) even in large models.
But as always →
mHC Limitations: What DeepSeek's Architecture Still Doesn't Solve
Extra architectural complexity makes mHC harder to reason about and implement compared to basic residual connections.
Its scalability depends on custom kernels, recomputation, and advanced pipeline scheduling, which can limit portability across hardware and systems.
The doubly stochastic constraint is approximate in practice, and exploring other manifolds remains an open question.
Finally, the overhead is small, but not zero.
With all that said, mHC is best seen as a first step toward reconciling hyper-connections with residual stability, not a finished solution. It opens several promising directions.
Conclusion
We traced the evolution of the residual connection from a simple survival tactic in LSTMs to the backbone of the modern Transformer. Across every architecture – ResNets, DenseNets, LLMs – they provided the essential guarantee: scalability and stability. But this safety came at a hidden cost: Constrained Expressivity. By biasing every layer toward the identity, we limited the network's ability to make radical transformations efficiently.
Hyper-connections attempted a rebellion. They pushed for total freedom to route information, but collapsed under the weight of their own chaos. Their failure seemed to confirm the industry's oldest fear: that without the safety net, the signal must die.
DeepSeek’s mHC changes the terms of this surrender. By enforcing the Conservation of Information through geometric constraints, it proves that we don't need to choose between the stability of the 90s and the complexity of the future. We can have both.
That choice matters. It suggests the next gains will come less from adding tricks and more from shaping the structure itself. There is more to explore here, and mHC shows where to look. And if we can give you an idea, it’s that: stop asking what block to add, start asking what must never break.
Sources and further reading
Who invented deep residual learning? by Jürgen Schmidhuber (28 Sep 2025)
Resources from Turing Post
