I drew the same stack three times, one layer tall, then three, then six, and I kept every dimension identical so there is nothing to look at but the depth. Same input, same width all the way down, same output. Reading left to right, each network is thinking deeper: every extra layer builds on what the layer above it found, so it can learn higher order concepts.
The mechanics never change. Each layer takes the activation output H from the layer above as its input, multiplies by a weight, and applies ReLU. What changes with depth is not the arithmetic but what the numbers stand for.
One layer can only describe the input in terms of the input. Three layers describe what the layer above them found, and then describe that. By the sixth layer the network is working with concepts several removes from the raw features, and none of those concepts had to be specified by us. That is the trade depth actually offers: the deeper stack reaches for more abstract ideas, using the same width and the same operation.
The cost is real. Six layers means six times the weights of one and twice the weights of three, with the multiplications to match, and the gradient has that much further to travel back. It also has to survive that trip, and past a certain depth it stops arriving at the layers nearest the input. Getting deeper than that takes more than stacking, which is where skip connections come in.
The ReLU between each layer is what makes the depth worth having. Take the activations out and the stack collapses: six weight matrices multiply together into a single one, and the six-layer network on the right would be an expensive way to draw the one-layer network on the left.
Next:
4. Binary Classifier