Full Fine-Tuning vs LoRA

Fine-Tuning series: 9 of 9

Library › Fine-Tuning

1. Weight Update

2. Pretrain vs Fine-Tune

3. Full Fine-Tuning

4. Freezing Layers

5. Linear Probe

6. Feature Extraction + Head

7. Adapter Layers

8. LoRA

9. Full Fine-Tuning vs LoRA

Finally, I drew this diagram to compare full fine-tuning and LoRA side by side, sharing an input and producing the same output, so that the only thing left to look at is the difference between them.

Open the interactive diagram ↗

In my experience students rarely struggle with what LoRA is. They struggle with why it is cheap, and that stays abstract until you can drag the rank yourself and watch one count fall while the other does not move at all. That is where this series has been heading since the first weight update: freezing layers, a linear probe, a new head, adapters and now LoRA are all the same move, letting fewer numbers change and being deliberate about which ones. Full fine-tuning touches all of them. LoRA touches a rank you choose.