Manifold-Constrained Hyper Connections (mHC) from DeepSeek (Jan 9, 2026)

AI by Hand ✍️ Seminars

Library › Seminar Series 2026

1. Manifold-Constrained Hyper Connections (mHC) from DeepSeek (Jan 9, 2026)

2. How Small Models Learn Tool Use (Jan 11, 2026)

3. Generative AI (Jan 11, 2026)

4. Attention (Jan 15, 2026)

5. Google Ironwood TPU: From Bits to HBM (Jan 19, 2026)

6. 9 AI Eval Formulas You Must Know (Jan 24, 2026)

7. Meta Superintelligence Labs vs Facebook AI Research (Jan 30, 2026)

8. Transformer: Six Levels of Understanding (Feb 13, 2026)

9. OpenClaw Seminar (Feb 23, 2026)

10. PPO → DPO → GRPO → Rubrics (Mar 2, 2026)

11. Gemma 4 (May 7, 2026)

12. Qwen 3.6 (May 21, 2026)

Hundreds of people showed up live for this Frontier seminar and made the classroom feel full and energized!

This Frontier Seminar kicked off my commitment to unpacking one frontier paper at a time, focusing on how the algorithm works, not just benchmark results.

The topic was DeepSeek’s mHC (Manifold-Constrained Hyper-Connections), a recent paper. This is as “Frontier” as we can get.

The paper is full of intimidating jargon that actually builds on a very familiar idea: residual connections. I started by revisiting why ResNets work so well, layers learn small additive updates instead of full transformations, and how that insight made deep networks possible.

From there, we moved to hyper-connections: extending a single skip connection into multiple interacting streams. Once you do that, the real challenge becomes mapping shapes correctly, merging streams, expanding them back, and mixing them together. I showed how all of this reduces to carefully designed matrix multiplications, with both static and input-dependent (dynamic) mixing.

Finally, we tackled the “manifold-constrained” part: restricting these mixing matrices so they stay stable and well-behaved, rather than arbitrary and noisy. The result is a powerful generalization of residual connections that still ends in the simplest operation of all, addition.

The big message: this paper isn’t magic. Once you break it down, it’s a clean extension of ideas many of us already know.

Outline

1. Residual Connection

   • Layer

   • Residual Connection

2. Hyper Connections

   • Multiple Streams

   • Static Mapping (n → 1)

   • Dynamic Mapping (n → 1)

   • Static Mapping (1 → n)

   • Dynamic Mapping (1 → n)

   • Static Mapping (n → n)

   • Dynamic Mapping (n → n)

3. Manifold Constrained

   • Sinkhorn–Knopp Algorithm

4. Complete Architecture

Recording & Workbook

Become a paid subscriber to access the full recording and the associated Excel workbook.