RoPE vs PE in QKV Self-Attention

Frontier AI Drawings: 11 of 13

Library › Frontier AI Drawings

1. "Expert Choice" Mixture of Experts (MoE)

2. MHA, MQA, GQA, MoE-A: More Attention!

3. New GPT-OSS Trick to Ignore Tokens

4. MXFP4, FP4, FP8

5. LoRA, Fine-Tune, Pre-Train

6. QLoRA, DoRA, BitFit, NF4 vs INT4

7. KV Cache, Prefill, Decode

8. EmbeddingGemma, MRL, InfoNCE, Embed vs. Decode

9. Inference Batching, Request-vs-Token Level

10. MLP Parallelism: Data, Context, Row, Column, Pipeline

11. RoPE vs PE in QKV Self-Attention

12. RMS, Group, Layer, Batch Norm, Tensor Parallelism

13. Qwen 3

Yesterday, DeepSeek released version 3.2, formally introducing DSA (DeepSeek Sparse Attention) and releasing both the Python reference implementation and open pre-trained weights. I’ve been busy building an Excel reference implementation to help a partner company grasp the math behind DSA and begin constructing its own version.

For this week’s issue, I intend to focus on RoPE (Rotary Positional Embedding). The timing of DeepSeek’s 3.2 is perfect. It reinforces why understanding RoPE is no longer optional. RoPE has quickly become a de facto standard in modern transformer architectures since it was published in April 2021, and its influence is only growing.

Drawings

I created four new sets of drawings to illustrate the following:

1. Sine & Cosine Positional Encoding Patterns

2. Positional Encoding (PE) vs. Rotary Positional Embedding (RoPE)

3. QKV Self-Attention + Positional Encoding (PE)

4. QKV Self-Attention + Rotary Positional Embedding (RoPE)

Once I finish explaining RoPE, only a few more building blocks remain, like RMSNorm and SwiGLU, before I’ll have unpacked all the key pieces needed to do a full architecture breakdown. My plan is to work toward a complete Qwen3 breakdown in the coming weeks.

---

Page 1 of 10

Become a member to access the rest of the drawings.