Why activation functions matter

A network trains for two hours, the loss drops for the first few hundred steps, and then it sits at 0.68 and refuses to move. The data is fine. The learning rate has been halved twice with no effect. Somebody suggests more layers, which makes it worse.

That plateau usually has a specific cause, and the cause is arithmetic in the activation function. Nothing about it requires a theory of representation learning to see. It requires multiplying four or five numbers together and noticing how small the answer is.

The number that governs everything

Backpropagation worked by hand established that every gradient is a product with one term per stage the derivative passed through. For an activation function, that term is its slope at the value it received. So the only thing that matters about an activation function, from the point of view of training, is how large its slope is where the network actually operates.

The logistic sigmoid squashes any input into the range 0 to 1. Its slope at output s is s times 1 - s. That expression is largest when s is 0.5, where it equals 0.25, and it falls away sharply on both sides.

Compute it at a few inputs.

• At an input of 0, the output is 0.5 and the slope is 0.25.
• At an input of 4, the output is 0.982 and the slope is 0.982 times 0.018, which is 0.0177.
• At an input of 6, the output is 0.9975 and the slope is 0.9975 times 0.0025, which is 0.00247.

A unit sitting at an input of 6 passes back a hundredth of the gradient a unit sitting at zero would pass back. It is not broken and it is not stuck at a bad answer. It is simply confident, and confidence in a sigmoid costs learning rate. Anything that pushes a unit's weighted sum away from zero, including large weights, large inputs, or an unscaled column, buys that penalty.

Ten layers of it

Now stack the penalty. Suppose a network is well behaved, so every sigmoid sits at its best possible operating point with a slope of 0.25, and suppose the weights between layers are around one so they neither shrink nor grow the signal. The gradient reaching the first layer is multiplied by 0.25 once per layer.

Ten layers gives 0.25 to the tenth power, which is 1 divided by 1,048,576, or 0.00000095.

The first layer receives roughly a millionth of the gradient the last layer receives. With a single learning rate for the whole network, the last layers train and the first layers effectively do not. This is the best case: the calculation assumed every unit was at its optimum. In practice they are not.

Glorot and Bengio measured this on real networks rather than assuming it. Their finding is that the logistic sigmoid activation is unsuited to deep networks with random initialisation because of its mean value, which can drive especially the top hidden layer into saturation, and that saturated units can move out of saturation by themselves, albeit slowly, which explains the plateaus sometimes seen when training neural networks. The plateau at 0.68 in the opening has a name and a measurement behind it.

The hyperbolic tangent is the usual first fix. It is a rescaled sigmoid centred on zero, with a maximum slope of 1 rather than 0.25, so a well-conditioned stack neither shrinks nor grows the gradient. It still saturates. At an input of 3 its slope is already about 0.01, so it delays the problem rather than removing it.

The rectifier, and what it costs

A rectifier passes positive numbers through unchanged and returns zero otherwise. Its slope is exactly 1 on one side and exactly 0 on the other. There is no saturation on the positive side at all, which is why a hundred-layer rectifier network can be trained and a hundred-layer sigmoid network cannot.

Glorot, Bordes and Bengio made the case for it with an argument that was not only about gradients. Rectifying neurons, they found, outperform hyperbolic tangent networks, and they create sparse representations with true zeros which seem remarkably suitable for naturally sparse data. Their networks reached their best performance from supervised training alone, without the unsupervised pre-training that was standard practice at the time.

The exact zero is the whole trick and the whole bill. A unit whose weighted sum is negative outputs zero, and its slope there is zero, so no gradient reaches its weights from that example. If every example in the data pushes a unit negative, that unit is permanently silent: it produces nothing, it learns nothing, and it never comes back, because coming back would require a gradient it can no longer receive.

The last article watched this happen in three lines of arithmetic. One gradient step with a learning rate of 0.5 moved the second hidden unit's weighted sum from 0.10 to -0.243, at which point it stopped contributing and stopped learning. A network with a thousand such units and an aggressive learning rate can lose a large fraction of them in the first epoch, and the loss curve reports this as a plateau rather than as a fault.

Lu, Shin, Su and Karniadakis define the dying ReLU problem as the case where ReLU neurons become inactive and only output zero for any input, and prove that a deep rectifier network will eventually die in probability as the depth goes to infinity. Their remedy is an initialisation, not an architecture change: a randomised asymmetric initialisation which they show can effectively prevent the dying ReLU.

The variants, and what each one is buying

Everything after the rectifier is an attempt to keep the unsaturated positive side while putting a non-zero slope back on the negative side.

• Leaky rectifier. Multiply negatives by a small constant such as 0.01 rather than zeroing them. A unit at -2 now outputs -0.02 and has a slope of 0.01, which is small but not zero, so the unit can climb back. It costs the exact sparsity that Glorot, Bordes and Bengio argued for.
• GELU. Hendrycks and Gimpel define it as x times the standard Gaussian cumulative distribution function of x, and describe the difference from a rectifier precisely: GELU weights inputs by their value, rather than gating inputs by their sign as in ReLUs. At an input of -1, a rectifier outputs 0 with a slope of 0. GELU outputs about -0.159, and its slope there is about -0.083. Small, negative, and not zero.
• Swish. Ramachandran, Zoph and Le searched automatically for activation functions and reported x times the sigmoid of beta x as their best find, claiming a 0.9 percent improvement in ImageNet top-1 accuracy for Mobile NASNet-A and 0.6 percent for Inception-ResNet-v2 when substituted for ReLU. They note that the simplicity of Swish and its similarity to ReLU is what makes it easy to adopt.

Read those two percentage figures carefully, because they are the honest scale of this decision. Under half a point to nine tenths of a point of top-1 accuracy, on a benchmark, from replacing every activation in the network. That is a real improvement and it is not the difference between a working system and a broken one. The difference between a saturating sigmoid stack and any of these is.

The output layer is a separate decision

The activation inside a network and the activation on its last layer are chosen for different reasons, and confusing them causes real errors.

Inside, the question is gradient flow, which is everything above. On the output, the question is what the number is supposed to mean. A regression output is usually left alone, so it can take any value. A binary classifier ends in a sigmoid because the output is a probability and probabilities live between 0 and 1. A multi-class classifier ends in a softmax so the outputs sum to 1.

Saturation on the output layer is not the same problem. A sigmoid output paired with a cross-entropy loss has a derivative that simplifies to the prediction minus the target, exactly the clean form worked through in the last article for squared error, and the saturating term cancels. Pair the same sigmoid output with a squared-error loss instead and it does not cancel: a confidently wrong prediction then produces a tiny gradient, which is the worst possible arrangement. The pairing of loss and output activation matters more than either choice alone, and it connects directly to the calibration argument in What probability buys you.

What to do when the loss will not move

• Print the fraction of units in each layer that output zero on a batch. If a layer is above about 80 percent dead, the learning rate or the initialisation is the fault, not the architecture.
• Print the mean absolute gradient per layer. A first layer whose gradient is three or four orders of magnitude below the last layer's is the vanishing product, visible directly.
• Check the input scaling before touching the activation. A column running to the thousands drives every unit it touches into saturation immediately, which is one of the concrete reasons for the scaling work in How a dataset becomes features.
• Use a rectifier or GELU inside and stop deliberating. The gap between the sensible options is under a point; the gap between a sensible option and a saturating one is the whole model.
• Lower the learning rate before switching to a leaky variant. Dead units are usually a step-size symptom, and the step size is the thing that is actually wrong.

Everything so far has assumed a layer where every unit sees every input. The next article gives up that assumption on purpose, and the parameter count falls by eight orders of magnitude.

References

1. Understanding the difficulty of training deep feedforward neural networks. Xavier Glorot and Yoshua Bengio, Proceedings of Machine Learning Research, volume 9, 2010.
2. Deep Sparse Rectifier Neural Networks. Xavier Glorot, Antoine Bordes and Yoshua Bengio, Proceedings of Machine Learning Research, volume 15, 2011.
3. Dying ReLU and Initialization, Theory and Numerical Examples. Lu Lu, Yeonjong Shin, Yanhui Su and George Em Karniadakis, arXiv, 2019.
4. Gaussian Error Linear Units (GELUs). Dan Hendrycks and Kevin Gimpel, arXiv, 2016.
5. Searching for Activation Functions. Prajit Ramachandran, Barret Zoph and Quoc V. Le, arXiv, 2017.