Now the mirror image: narrow in the middle, full width at both ends. The first weight squeezes 64 features down to 8, the second opens them back out to 64.
The narrow middle is a budget. Whatever this block produces, all of it had to pass through 8 numbers, so it can only ever express a small set of the possible outputs. That sounds like a weakness, and it is exactly the point: the two small weights together hold a fraction of the parameters we would need for a single 64 by 64 weight.
This is the shape behind LoRA, the standard way to fine-tune a large model cheaply, where a pair of small weights like these learns a correction while the original model is left alone. The fine-tuning series covers it properly.
I put this next to the autoencoder deliberately: the same silhouette, narrow in the middle, doing a different job. The autoencoder squeezes because it wants a compact representation. This squeezes because it wants a cheap one.
Next:
10. GAN