Two layers, and I have made the middle wider than either end. The first weight opens 64 features out to 256, the second brings them straight back to 64. Nothing is compressed and nothing is kept; the width is borrowed and then returned.
The point is the room in the middle. A wider layer can hold combinations of features that will not fit in the narrow input, and the second weight decides which of those combinations are worth carrying forward. Because the output is the same width as the input, we can stack this block, or add it back to whatever fed it.
This is the feed-forward layer inside a transformer. Every block has one, sitting right after attention, and the expansion is usually four times: 512 in, 2048 in the middle, 512 out. Attention decides which tokens to mix; the feed-forward layer is where each token gets thought about on its own.


