Why a stack of straight lines needs one bend to think. · Source

Stack a hundred layers. Without one bend, you still have one.

A deep network is layers stacked on layers — surely more layers mean more power. They don't. Without one tiny trick wedged between them, a hundred layers do exactly what one does: draw a single straight line. The whole reason depth works — the reason a model can learn anything curved, tangled, or real — is one humble move called the activation function. The bend. Let's see why a stack of straight lines is still just a straight line.

Two straight steps in a row are secretly one.

W_2(W_1 x + b_1) + b_2 = (W_2 W_1)\,x + (W_2 b_1 + b_2) = W' x + b'

Like clear panes of glass: look through ten flat panes and the view is just as straight as through one — no stack of flat glass ever bends the light. A layer without a bend is the same: it only stretches and rotates space, a straight-line map. Chain two and the math folds them into a single straight-line map — same for a hundred. Depth, on its own, buys you nothing.

One straight cut can't split this.

f_{\text{lin}}(x)=\operatorname{sign}(w^\top x + b)\quad(\text{one straight cut});\qquad \text{XOR}:\ (0,0)\!\to\!0,\ (0,1)\!\to\!1,\ (1,0)\!\to\!1,\ (1,1)\!\to\!0

Like a rope on a checkerboard: lay one taut rope across the squares and try to put every dark tile on one side, every light tile on the other — you can't, the pattern interlocks. The simplest real task, XOR, is exactly this: two classes you cannot separate with a single straight line. A straight-line model is blind to it. The world is full of these — and flat maps can't bend around them.

The fix is almost nothing: a single bend.

\operatorname{ReLU}(x)=\max(0,\,x)

Like a hinge in a steel rod: a straight bar can't change direction — until you add one hinge, and now it folds. After each straight-line layer, apply one simple bend: keep what's positive, flatten the rest to zero. That kink is the ReLU, and it's the cheapest nonlinearity there is. One hinge does little. But a hinge after every layer turns a rigid ruler into something that can fold around anything.

Why this bend, and not the old smooth squashers?

\sigma'(x)=\sigma(x)\bigl(1-\sigma(x)\bigr)\le \tfrac{1}{4}\ \Rightarrow\ \prod_{l=1}^{L}\sigma' \le \left(\tfrac{1}{4}\right)^{L}\!\to 0;\qquad \operatorname{ReLU}'(x)=1\ \ (x>0)

Early nets bent with an S-curve that flattens at both ends. Like a ball dropped into deep foam: each soft, flat layer swallows the push, and across many layers the learning signal dies before it returns — the dreaded vanishing gradient. ReLU's positive side is a hard floor with slope exactly one: the signal bounces straight back, undimmed. That's why deep stacks finally trained — the bend let the signal breathe.

But the hinge has a dead elbow. So round it.

\operatorname{GELU}(x)=x\,\Phi(x),\qquad \Phi(x)=\tfrac{1}{2}\left[\,1+\operatorname{erf}\!\left(\tfrac{x}{\sqrt{2}}\right)\right]

ReLU's corner is sharp: for every negative input the slope is flat zero, and a neuron stuck there goes dead — no slope to follow, no way to learn back. Like a ramp instead of a curb: a wheel jams against a hard edge but rolls smoothly up a graded slope. Smooth bends like GELU round the corner — a soft gate that opens almost fully for big inputs and eases shut for small ones, leaving a usable slope everywhere.

Straight pieces plus bends can trace any shape.

\forall\,\varepsilon>0\ \ \exists\,N:\quad \left|\,g(x)-\sum_{i=1}^{N} c_i\,\sigma\!\left(w_i^\top x + b_i\right)\right|<\varepsilon

Here's the payoff. Like bending wire into a silhouette: a stiff straight wire draws only a line, but with enough small bends it traces any outline you like — a bird, a coastline, a signature. A linear layer lays the straight pieces; the activation adds the bends. Stack enough and the network can approximate any continuous shape, to any precision. That's the famous result — and it dies the instant the bend is removed.

🌱 Every curve it draws is really straight lines.

Look closely at what the bend builds. A network of these simple kinks doesn't draw a true curve at all — it draws countless tiny straight pieces, fitted so finely they only look smooth. The whole grace of a deep model — every elegant arc of thought — is a staircase, sanded down until the steps disappear. 🌱 So is the curve real? Or is smoothness just straightness, cut fine enough to fool us?