ViT to Llama 1, 2, 3, 4

LLM Paper Club (5/14)

Live Version:

https://o365coloradoedu-my.sharepoint.com/:x:/g/personal/peye9704_colorado_edu/EaPGoLubxvVAhWpd8vr1b8MBIiN6jZ-5AX9fUKTLLac3MA?e=W06vTc&activeCell=%27ViT%27!A1

Baseline Version

https://o365coloradoedu-my.sharepoint.com/:x:/g/personal/peye9704_colorado_edu/EV1eVo6xEJNGlSXnZ8LLcX4BCrYSko5lalP-f5rGjVlblw?e=z7sNAw&activeCell=%27ViT%27!A1

I will use Vision Transformer (ViT) as the baseline and extend it to Llama 1, 2, 3, 4 live.

Baseline: ViT

+ RMSNorm

+ model dimensions

+ layers

+ RoPE

+ Group Query Attention

+ Sparse Attention

+ Flash Attention

+ context length
​

Or download

ViT Excel Preview

207KB ∙ XLSX file

Download

Download