Backpropagation worked by hand

Training a network is one line of code, and that is the problem. When the loss stops moving, or moves the wrong way, or turns into a NaN in the fourth epoch, the person holding the keyboard needs a mental model of what loss.backward() did, and "it computes the gradients" is not a mental model.

So compute them by hand. The network below has five stages and nine weights, which is small enough to do on paper and large enough that nothing important has been left out. Every number here can be checked with a calculator, and one of them is checked twice, against a method that does not use calculus at all.

The network

Two inputs, two hidden units with rectifiers, one linear output, and a squared-error loss with a half in front of it so the derivative comes out clean.

• Inputs: x1 = 1.0, x2 = 0.5
• Hidden unit one: weights 0.4 and -0.2, bias 0.1
• Hidden unit two: weights -0.3 and 0.8, bias 0.0
• Output unit: weights 0.6 and -0.5 on the two hidden units, bias 0.2
• Target: 1.0

Forward, once

Hidden unit one computes 0.4 times 1.0, plus -0.2 times 0.5, plus 0.1. That is 0.4 minus 0.1 plus 0.1, which is 0.40. Positive, so the rectifier passes it through and its activation is 0.40.

Hidden unit two computes -0.3 times 1.0, plus 0.8 times 0.5, plus 0.0. That is -0.3 plus 0.4, which is 0.10. Also positive, so its activation is 0.10.

The output unit computes 0.6 times 0.40, plus -0.5 times 0.10, plus 0.2. That is 0.24 minus 0.05 plus 0.2, which is 0.39.

The loss is half of the squared difference between 0.39 and the target 1.0. The difference is -0.61, its square is 0.3721, and half of that is 0.186.

Every one of those intermediate numbers has to be kept. That is not an accident of doing it by hand; it is the memory cost of training, and it is the reason a batch size that fits in memory for prediction may not fit for training.

Backward, once

The backward pass asks a single question at every stage, in reverse order: if this number had been slightly larger, how much larger would the loss have been. The PyTorch documentation describes it as traversing backwards from the output, collecting the derivatives of the error with respect to the parameters of the functions.

At the output. The loss is half of (y - 1.0) squared, so its derivative with respect to the prediction is simply y - 1.0, which is 0.39 minus 1.0, or -0.61. Negative, correctly: the prediction is too low, so raising it lowers the loss.

The output weights. The prediction was 0.6 times 0.40 plus -0.5 times 0.10 plus 0.2. The derivative of the prediction with respect to the first output weight is whatever that weight was multiplied by, which is 0.40. Chain them:

• gradient of the first output weight: -0.61 times 0.40, which is -0.244
• gradient of the second output weight: -0.61 times 0.10, which is -0.061
• gradient of the output bias: -0.61 times 1, which is -0.61

Back into the hidden units. The derivative of the prediction with respect to the first hidden activation is the weight sitting on it, 0.6. So the derivative of the loss with respect to that activation is -0.61 times 0.6, which is -0.366. For the second, the weight is -0.5, so the derivative is -0.61 times -0.5, which is +0.305.

Read those two numbers next to each other. The same mistake arrives at the two hidden units with opposite signs, because one of them feeds the output through a positive weight and the other through a negative one. Both units are being told to help, and helping means different things to each.

Through the rectifiers. A rectifier's derivative is 1 where its input was positive and 0 where it was not. Both hidden sums were positive, so both derivatives pass through unchanged: -0.366 and +0.305.

The input weights. Same rule as before. Each weight's gradient is the derivative arriving at its unit times the value that entered through that weight.

• first unit, weight on x1: -0.366 times 1.0, which is -0.366
• first unit, weight on x2: -0.366 times 0.5, which is -0.183
• first unit, bias: -0.366
• second unit, weight on x1: 0.305 times 1.0, which is +0.305
• second unit, weight on x2: 0.305 times 0.5, which is +0.1525
• second unit, bias: +0.305

Nine weights, nine numbers, one pass. That is the property that makes training feasible: the backward pass costs about what the forward pass costs, no matter how many parameters there are.

Checking one of them without calculus

A gradient of -0.366 on the first input weight predicts that raising that weight by 0.01 lowers the loss by about 0.00366. Test it.

Set the weight to 0.41. The first hidden sum becomes 0.41 minus 0.1 plus 0.1, which is 0.41, and its activation is 0.41. The prediction becomes 0.6 times 0.41, minus 0.05, plus 0.2, which is 0.246 plus 0.15, or 0.396. The loss is half of 0.604 squared, which is 0.182408.

The loss fell from 0.186050 to 0.182408, a change of -0.003642 over a step of 0.01. Divide: -0.3642, against the calculated -0.366. The small gap is the curvature of the loss over that step, and it shrinks as the step shrinks.

This is worth doing once by hand and then knowing about permanently. A finite-difference check like this is the standard way to find a bug in a hand-written derivative. Baydin, Pearlmutter, Radul and Siskind describe automatic differentiation as a family of techniques similar to but more general than backpropagation, for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs, and part of the argument for it is precisely that it removes the class of error the check above exists to catch. Numerical differencing is slow and approximate; symbolic differentiation produces expressions that grow explosively; automatic differentiation gives exact derivatives at the cost of one extra pass.

One step, and what it breaks

Apply plain gradient descent with a learning rate of 0.5. Each weight moves against its gradient by half the gradient's size.

The output weights become 0.722 and -0.4695, and the output bias becomes 0.505. The first hidden unit's weights become 0.583 and -0.1085 with a bias of 0.283. The second hidden unit's weights become -0.4525 and 0.72375 with a bias of -0.1525.

Run the same input through again. The first hidden sum is 0.583 minus 0.054 plus 0.283, which is 0.812. The second hidden sum is -0.4525 plus 0.362 minus 0.1525, which is -0.243.

Negative. The rectifier returns zero, so the second hidden unit contributes nothing at all to this prediction, and because a rectifier's derivative is zero there, it will receive no gradient from this input either. One step moved a unit from active to silent.

The prediction is 0.722 times 0.812 plus 0.505, which is 1.091, and the loss is half of 0.091 squared, or 0.0041. The loss fell by a factor of forty five, which looks excellent, and the prediction overshot the target of 1.0, which is the warning that the step was too big. Both facts came from the same learning rate.

That interaction, between a step size chosen for speed and a unit that switches off permanently, is the subject of the next article. Here it is enough to notice that the arithmetic gave no warning: nothing in the gradient calculation knew that 0.5 was an aggressive step.

Why the products matter more than the individual terms

Every gradient computed above is a product of terms, one per stage the derivative passed through. With five stages the products are short. With fifty they are not, and the behaviour of a long product is governed by whether its typical term is above or below one.

Pascanu, Mikolov and Bengio studied exactly this for recurrent networks, where the same weights are applied at every step so the product is a power. They set out the vanishing and exploding gradient problems from analytical, geometric and dynamical systems perspectives, and propose a gradient norm clipping strategy for the exploding case and a soft constraint for the vanishing one. Clipping is now standard practice in a great deal of training code, and it exists because of the shape of the products written above, not because of anything specific to language or sequences.

The arithmetic is unforgiving. A term of 0.9 repeated fifty times is 0.005. A term of 1.1 repeated fifty times is 117. Neither is a bug; both are what a chain of multiplications does.

What to take to the keyboard

• The backward pass is one multiplication per edge, run in reverse. Nothing else is happening.
• Every activation computed on the way forward is held until the backward pass consumes it, which is where training memory goes.
• A gradient is the derivative arriving at a unit times the value that entered the weight, which is why an input that is always zero produces a weight that never moves. The scaling advice in How a dataset becomes features is a gradient argument, not an aesthetic one.
• If a hand-written derivative disagrees with a finite-difference check by more than a fraction of a percent on a small step, the derivative is wrong, not the check.
• A loss that falls dramatically in one step is not automatically good news. Check whether the prediction crossed the target, and check whether any units went quiet.

The unit that went silent above did so because of the rectifier, which was introduced in A neuron is a weighted sum and a decision as the cheapest useful non-linearity and is now revealing its price. The next article works out what the alternatives cost, and how many units a real network loses this way.

References

1. A Gentle Introduction to torch.autograd. PyTorch documentation, version 2.13, 2026.
2. Automatic differentiation in machine learning, a survey. Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul and Jeffrey Mark Siskind, arXiv, 2015.
3. On the difficulty of training Recurrent Neural Networks. Razvan Pascanu, Tomas Mikolov and Yoshua Bengio, arXiv, 2012.