Convolutions and what a filter sees

Count the parameters in the obvious architecture and the idea dies on the spot. A colour image at 224 by 224 is 150,528 numbers. A fully connected layer producing 64 feature maps at the same resolution would produce 3,211,264 outputs, and connecting every input to every output takes 150,528 times 3,211,264 weights. That is about 483 billion parameters, for one layer, before anything has been learned.

The convolution that replaces it holds 1,728 weights and 64 biases. Same input, same output shape, roughly one part in 270 million of the storage. Two assumptions buy the difference, and both of them are wrong about some kinds of data, which is why it is worth knowing exactly what they are.

One kernel, one patch, by hand

Take a five by five patch of an image, with a bright vertical bar running down the middle of its top three rows. Every pixel is 10 except the bar, which is 90.

Now take a three by three kernel holding the numbers

• plus 1, 0, minus 1
• plus 1, 0, minus 1
• plus 1, 0, minus 1

Lay the kernel over the top right three by three region of the patch, so it covers columns 3, 4 and 5 of the first three rows. Multiply each of the nine covered pixels by the weight sitting on it and add.

The kernel's left column has weight plus 1 and sits on the bar: 90 plus 90 plus 90, which is 270. Its middle column has weight 0 and contributes nothing. Its right column has weight minus 1 and sits on background: minus 10, minus 10, minus 10, which is minus 30.

The output for that position is 270 minus 30, which is plus 240.

Slide the kernel one step left, so it covers columns 2, 3 and 4. Now the bar is in the middle column, under the weights of zero. The left and right columns both sit on background: 30 minus 30, which is 0.

Slide once more, to columns 1, 2 and 3. Now the bar is under the weight of minus 1: 30 minus 270, which is minus 240.

Do that at all nine valid positions and the output map is

• minus 240, 0, plus 240
• minus 160, 0, plus 160
• minus 80, 0, plus 80

The magnitudes fall down the map because the bar only occupies the top three rows, so the window catches less of it each time it moves down. Read the signs: negative where the bar enters from the right, positive where it leaves to the left, zero where it is centred. Those nine weights are an edge detector, and nobody wrote "detect edges" anywhere. The behaviour is entirely in the numbers.

The two assumptions

Locality. Each output looks at nine inputs, not 150,528. The kernel asserts that whatever matters about a pixel can be decided from its immediate neighbours. For a photograph that is close to true, since an edge, a corner, or a texture is a local arrangement. For a table of customer attributes it is nonsense, because there is no meaning to the two columns adjacent to "monthly turnover".

Weight sharing. The same nine numbers computed every one of the nine outputs above. The kernel asserts that a question worth asking in one part of the image is worth asking, in exactly the same form, everywhere. For a photograph that is close to true, since an edge is an edge wherever it appears. For a document scan where the top strip is always a header and the bottom is always a signature block, it is throwing away information you have.

Those two assumptions are the entire saving. Locality cuts each output's inputs from 150,528 to 9. Sharing cuts the number of distinct weights from one set per output position to one set for the whole image.

The shapes, exactly

Getting the arithmetic of shapes right is most of the work of assembling one of these, and the PyTorch documentation states the rule directly. For each spatial dimension,

output = floor of ((input + 2 times padding, minus dilation times (kernel size minus 1), minus 1) divided by stride, plus 1).

Check it against the worked example. Input 5, kernel 3, no padding, stride 1, dilation 1: 5 plus 0 minus 2 minus 1 is 2, divided by 1 is 2, plus 1 is 3. A three by three output, which is what came out.

Check it against a first layer of a typical image network. Input 224, kernel 7, padding 3, stride 2: 224 plus 6 minus 6 minus 1 is 223, divided by 2 is 111.5, floored to 111, plus 1 is 112. The resolution halves, which is what stride 2 is for.

The same documentation gives the weight shape as out-channels by in-channels divided by groups, by kernel height, by kernel width. So a layer taking 3 colour channels to 64 feature maps with a 3 by 3 kernel holds 64 times 3 times 3 times 3 weights, which is 1,728, plus 64 biases: 1,792 parameters in total. That is the number promised at the top.

The compute does not fall nearly as far as the parameter count does, and this catches people out. Each of the 224 times 224 times 64 outputs costs 27 multiply-adds, which is about 86.7 million multiply-adds for that one layer on one image. Convolutions are cheap in memory and expensive in arithmetic, which is precisely the shape of workload a GPU is built for.

Dumoulin and Visin wrote the reference on this. Their guide sets out to clarify the relationship between input shape, kernel shape, zero padding, strides and output shape for convolutional, pooling and transposed convolutional layers, and it is the thing to open when a transposed convolution produces an output two pixels smaller than expected.

Stacking, and the size of the window

One 3 by 3 kernel sees a 3 by 3 region. Feed its output into a second 3 by 3 kernel and each output of the second layer depends on a 5 by 5 region of the original image. A third layer reaches 7 by 7.

That growth is why small kernels won. Three stacked 3 by 3 layers cover the same 7 by 7 window as one 7 by 7 kernel, using 27 weights per channel pair instead of 49, with two extra non-linearities in between. Fewer parameters, more expressive, and the non-linearity argument from Why activation functions matter applies at each step: without the rectifiers between them, the three layers would collapse back into one linear operation, exactly as shown in A neuron is a weighted sum and a decision.

Krizhevsky, Sutskever and Hinton's network is the one that made the argument to the rest of the field. They report training on 1.3 million high resolution images across 1,000 classes, with 60 million parameters and 500,000 neurons in five convolutional layers, some followed by max-pooling, and two globally connected layers ending in a 1,000-way softmax, reaching top-1 and top-5 error rates of 39.7 percent and 18.9 percent. They credit non-saturating neurons and an efficient GPU implementation for the training speed, and a new regularisation method for controlling overfitting in the fully connected layers.

What the filters actually learn

The edge detector above was written by hand. In a trained network nobody writes them, and the interesting question is what appears instead.

Zeiler and Fergus built a way to look. They introduce a visualisation technique that gives insight into the function of intermediate feature layers and the operation of the classifier, and use ablation studies to measure the contribution of each layer, which led them to architectures that beat the previous results on ImageNet, Caltech-101 and Caltech-256.

The finding that survives is the ordering. Early layers hold things very much like the kernel above: edges at various orientations, colour blobs, simple gradients. Middle layers hold combinations of those, corners and repeated textures. Later layers respond to arrangements that a person would name. Nothing in the loss function asked for this hierarchy; it is what falls out of stacking local, shared filters and training end to end.

Where the assumptions fail

• Position matters after all. A convolution is deliberately blind to where in the image a pattern occurred. If the position carries meaning, as in a form with fixed fields, that information has to be supplied some other way or the architecture will discard it.
• Rotation and scale are not free. Weight sharing gives you invariance to translation, and nothing else. A filter trained on upright faces does not recognise a face rotated by 90 degrees. That is what augmentation is for, and augmentation is a way of buying invariance with data rather than with architecture.
• The receptive field is a hard limit. A network whose deepest layer sees a 60 pixel window cannot use evidence 200 pixels away, no matter how much you train it. Long-range dependence is an architectural decision, not a training outcome.
• Tabular data has no neighbours. Columns have no spatial order, so locality means nothing and sharing weights across adjacent columns is arbitrary. The tree-based argument in Decision trees and how they split is not sentiment about simpler models; it is the observation that the assumptions above do not hold there.

The next architecture in this series gives up the fixed window entirely, because the data it handles has no fixed size at all: a sentence, a session, a series of readings that might be ten steps long or ten thousand.

References

1. A guide to convolution arithmetic for deep learning. Vincent Dumoulin and Francesco Visin, arXiv, 2016.
2. torch.nn.Conv2d. PyTorch documentation, version 2.13, 2026.
3. Visualizing and Understanding Convolutional Networks. Matthew D. Zeiler and Rob Fergus, arXiv, 2013.
4. ImageNet Classification with Deep Convolutional Neural Networks. Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, Advances in Neural Information Processing Systems 25, 2012.