Manifold-Constrained Hyper Connections (mHC) from DeepSeek

🚀 Frontier AI Seminar Series

This morning, I gave the first Frontier seminar of 2026 for the members of the AI by Hand Academy. Thank you to the hundreds of people who showed up live and made the classroom feel full and energized!

This Frontier Seminar kicked off my commitment to unpacking one frontier paper at a time, focusing on how the algorithm works, not just benchmark results.

This week’s topic was DeepSeek’s mHC (Manifold-Constrained Hyper-Connections)—a paper published last week. This is as “Frontier” as we can get.

The paper is full of intimidating jargon that actually builds on a very familiar idea: residual connections. I started by revisiting why ResNets work so well—layers learn small additive updates instead of full transformations—and how that insight made deep networks possible.

From there, we moved to hyper-connections: extending a single skip connection into multiple interacting streams. Once you do that, the real challenge becomes mapping shapes correctly—merging streams, expanding them back, and mixing them together. I showed how all of this reduces to carefully designed matrix multiplications, with both static and input-dependent (dynamic) mixing.

Finally, we tackled the “manifold-constrained” part: restricting these mixing matrices so they stay stable and well-behaved, rather than arbitrary and noisy. The result is a powerful generalization of residual connections that still ends in the simplest operation of all—addition.

The big message: this paper isn’t magic. Once you break it down, it’s a clean extension of ideas many of us already know.

Outline

1. Residual Connection

   • Layer

   • Residual Connection

2. Hyper Connections

   • Multiple Streams

   • Static Mapping (n → 1)

   • Dynamic Mapping (n → 1)

   • Static Mapping (1 → n)

   • Dynamic Mapping (1 → n)

   • Static Mapping (n → n)

   • Dynamic Mapping (n → n)

3. Manifold Constrained

   • Sinkhorn–Knopp Algorithm

4. Complete Architecture

Recording

The full recording and the associated Excel workbook are available to AI by Hand Academy members. You can become a member via a paid Substack subscription.