A neuron is a weighted sum and a decision

The request arrives in almost the same words every time. The logistic regression is doing fine, the team is comfortable with it, and someone senior would like to know whether a neural network would do better. Nobody in the room can say what the network would be doing differently, which makes the question impossible to answer and impossible to refuse.

It is answerable, and the answer sits in a table with four rows. A network is not a different kind of object from the logistic regression already in production. It is that object, repeated, with one small piece added between the repetitions. The piece is what earns the whole thing its keep, and it is easier to see by watching what happens when it is missing.

One unit, in full

Take a single unit with two inputs. It holds a weight for each input and one number of its own, the bias. Give it inputs of 1.0 and 0.5, weights of 0.4 and 0.2, and a bias of 0.1. It computes

0.4 times 1.0, plus 0.2 times 0.5, plus 0.1, which is 0.6.

That is the entire arithmetic of a neuron. The PyTorch documentation states the operation with no more ceremony than that: nn.Linear applies an affine linear transformation, holding a weight of shape out-features by in-features and a bias of shape out-features, both initialised from a uniform distribution whose width is set by the number of inputs. A layer of 64 such units over 30 inputs is 64 copies of the sum above, computed at once, holding 64 times 30 weights and 64 biases.

Anyone who has read Linear regression is still the baseline has already met this sum, and anyone who has read Logistic regression and decision boundaries has already met what happens when you squash it into a probability. A unit is one of those, and a layer is a rack of them. The interesting question is what you get by putting one rack in front of another.

Two layers of pure arithmetic collapse into one

Suppose a first layer computes h = A x + a and a second computes y = B h + b, with no switch in between. Substitute:

y = B (A x + a) + b, which is (B A) x + (B a + b).

B A is a single matrix. B a + b is a single vector. Whatever those two layers computed, one layer with the weights B A and the bias B a + b computes exactly the same function, using fewer parameters and less arithmetic. Adding a third and a fourth layer changes nothing: the product of any number of matrices is still a matrix.

A deep stack of purely linear layers is a very expensive way to write a linear model. The depth is real in the code, real in the memory, real in the training time, and completely absent from the function.

The four rows that force the issue

Here is a problem a linear model cannot solve, small enough to check by hand. Two binary inputs, and an output that is 1 when exactly one of them is 1.

Inputs 0 and 0, output 0
Inputs 1 and 0, output 1
Inputs 0 and 1, output 1
Inputs 1 and 1, output 0

A single unit followed by a threshold decides by asking whether w1 x1 + w2 x2 + b is above zero. Write down what the four rows demand.

Row one demands b be at or below zero. Row two demands w1 + b be above zero, so w1 is greater than minus b, which is at least zero. Row three demands the same of w2. Row four demands w1 + w2 + b be at or below zero.

Add rows two and three: w1 + w2 is greater than minus 2 times b. Since b is at or below zero, minus 2 times b is at least minus b, so w1 + w2 is greater than minus b, which puts w1 + w2 + b above zero. Row four demanded the opposite. There is no assignment of three numbers that satisfies all four rows, and no amount of training will find one, because none exists.

That is the whole case for depth, stated without a single figure of speech. Some functions are not a weighted sum of their inputs, and no weighted sum of weighted sums will reach them either.

Adding the switch, with numbers

Put a rectifier between the layers. A rectifier passes a positive number through unchanged and returns zero for anything at or below zero. It is one comparison per unit, which is the cheapest non-linear thing anyone has found that works.

Two hidden units are enough. Let the first compute x1 + x2 - 0.5 and the second compute x1 + x2 - 1.5, both passed through the rectifier, and let the output be the first minus three times the second.

Inputs 0 and 0: the first unit sees minus 0.5 and outputs 0, the second sees minus 1.5 and outputs 0. Output 0.
Inputs 1 and 0: the first sees 0.5 and outputs 0.5, the second sees minus 0.5 and outputs 0. Output 0.5.
Inputs 0 and 1: identical to the row above. Output 0.5.
Inputs 1 and 1: the first sees 1.5 and outputs 1.5, the second sees 0.5 and outputs 0.5. Output 1.5 minus 1.5, which is 0.

Threshold anywhere between 0 and 0.5 and the four rows come out right. Three units, seven weights, two biases. The impossible problem became a small one because two units are now allowed to disagree about which region of the input they care about, and the rectifier is what lets a unit stop caring.

Notice what the second hidden unit is doing. It is silent for three of the four rows and speaks only when both inputs are on, at which point it cancels the first unit's answer. That is a condition, expressed in arithmetic. A linear model has no way to say "except when", and this is how a network says it.

What depth buys that width does not

If one hidden layer with rectifiers can express a condition, why build twenty. The honest answer is that one wide layer is enough in principle and hopeless in practice, and the theory is more specific than that summary suggests.

Lu, Pu, Wang, Hu and Wang proved a universal approximation result from the width side rather than the depth side: rectifier networks of width equal to the input dimension plus four are universal approximators. They also proved the boundary is sharp, in that except for a set of measure zero, functions cannot be approximated by rectifier networks of width equal to the input dimension. Their third result is the one that matters for architecture choices: there exist classes of wide networks that no narrow network can realise unless its depth exceeds a polynomial bound, and they conclude that depth is more effective than width for the expressiveness of rectifier networks.

Read that as a budget statement. For a fixed number of parameters, arranging them deep tends to express more than arranging them wide. It is not a guarantee that your deeper model will be better on your data, and it says nothing at all about whether it will train.

Where a fresh network goes wrong before it learns anything

A network that cannot be trained is not a model. Two failures happen before the first gradient step, and both come from the numbers the weights are set to.

If the initial weights are too large, the weighted sums in the first layer are large, and every layer after it multiplies the problem. If they are too small, the sums shrink towards zero layer by layer, and the last layer sees almost no variation to work with. Glorot and Bengio measured this on real networks and found that the logistic sigmoid activation is unsuited to deep networks with random initialisation because of its mean value, which can drive especially the top hidden layer into saturation, and that saturated units move out of saturation by themselves only slowly, which explains the plateaus sometimes seen when training. Their diagnosis is that training is harder when the singular values of the Jacobian associated with each layer are far from one, and their remedy is an initialisation scheme scaled to the layer's fan-in and fan-out.

The practical residue is short. Do not initialise weights to zero, because every unit in a layer would then compute the same thing and receive the same gradient forever. Do not initialise them by hand at all. The default in any current framework is fan-in scaled, which is why the PyTorch documentation specifies a uniform distribution whose width depends on the number of inputs rather than a fixed range.

What this costs

A network gives up things the linear model was quietly providing.

A readable coefficient. In a linear model, a weight is the effect of one input on the output, holding the rest fixed. In a network, the same input reaches the output through every unit in every layer, and no single number reports its effect.
Convexity. The loss surface of a linear regression has one bottom. A network's does not, and the discussion of gradient descent in What a model actually learns applies with the extra warning that where you start now changes where you end.
Data. Twenty parameters can be fitted on a few hundred rows. Two hundred thousand cannot, and the overfitting described in that same article arrives faster and less visibly.
A baseline you can defend. If the network beats the linear model by one point of accuracy, the difference has to be worth the operational cost of the thing that produced it.

What to check before reaching for depth

The question that started this article has a usable form. Instead of asking whether a neural network would do better, ask whether the problem contains conditions the current model cannot express, and then look for them.

Fit the linear model, then plot its errors against each input. A residual that changes sign as an input rises is an interaction the model cannot represent.
Try adding the interaction by hand, as a product of two columns, using the feature work in How a dataset becomes features. If two or three hand-built terms close the gap, build those instead of a network.
Count the rows. A rule of thumb worth arguing about rather than obeying: if the model has more parameters than you have examples, you are relying entirely on regularisation to keep it honest.
If the answer is still a network, start with one hidden layer and a rectifier, and make it the thing to beat rather than the thing to ship.

Everything in the rest of this series is built out of the unit above. The next article follows a single mistake backwards through a network of five of them, one multiplication at a time, so that the word "training" stops being a black box and becomes arithmetic you can check on paper.

References

torch.nn.Linear. PyTorch documentation, version 2.13, 2026.
The Expressive Power of Neural Networks, A View from the Width. Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu and Liwei Wang, arXiv, 2017.
Understanding the difficulty of training deep feedforward neural networks. Xavier Glorot and Yoshua Bengio, Proceedings of Machine Learning Research, volume 9, 2010.