What a model actually learns

A delivery business wanted to know why its arrival estimates were reliable in the city centre and useless past the ring road. Someone on the call said the model had learned that the suburbs behave differently. It had not learned anything of the sort. It had done something far narrower, and saying precisely what it had done was the difference between fixing the estimates and arguing about them for another month.

That gap between what a model does and what people describe it as doing is the most expensive misunderstanding in applied machine learning. It is worth closing early, because almost every later question, about evaluation, about data, about whether a system can be trusted with a decision, depends on getting this one right.

The mechanism, stated plainly

A model is a function with adjustable numbers inside it. You choose the shape of the function in advance. You choose a loss, which is a single number measuring how wrong the function's outputs are on data you already have. Then an optimiser changes the adjustable numbers, over and over, in whatever direction makes the loss smaller.

That is training. All of it. The parameters that come out are the ones that made a particular arithmetic quantity small on a particular set of rows.

Nothing in that procedure contains a concept. The optimiser does not know what a kilometre is, whether Tuesday comes before Wednesday, or that a delivery is a physical event involving a van. It has numbers going in, numbers coming out, and a rule for adjusting itself. Goodfellow, Bengio and Courville set this out carefully in chapter five of their textbook, and it is worth reading in full precisely because it refuses to make the process sound cleverer than it is.

A line through nine orders

Here are nine deliveries, distance in kilometres against the time each one took in minutes.

• 1.2 km, 14 minutes
• 2.0 km, 17 minutes
• 2.8 km, 19 minutes
• 3.5 km, 26 minutes
• 4.4 km, 24 minutes
• 5.1 km, 33 minutes
• 6.0 km, 31 minutes
• 7.2 km, 42 minutes
• 8.5 km, 45 minutes

Pick the simplest useful shape: a straight line, time = a + b * distance. Two adjustable numbers. Choose the loss most people reach for first, the mean of the squared differences between what the line says and what actually happened. Then find the pair of numbers that makes that mean as small as possible.

For these nine orders the answer is time = 8.16 + 4.36 * distance. Its typical miss is 2.2 minutes, measured as the square root of the mean squared error. Predicting the average of the nine times for every order, which is the laziest model that still counts as one, misses by 10.2 minutes on the same measure. The line is worth having.

Notice what the model now consists of: two numbers, 8.16 and 4.36. That is the entire content of what was learned. You can say something sensible about them, that the first looks like fixed overhead per delivery and the second like time per kilometre, but the fitting procedure did not know either of those things and did not check whether they were true. It made a sum small. The interpretation is yours, and you are responsible for it.

Capacity, and the bill it brings

Two numbers is a tight budget. Give the model more and it can bend.

A polynomial of degree eight has nine adjustable numbers, one for each order in the table, and it can pass through all nine points exactly. Training error: zero. Perfect, by the only measure the optimiser was ever given.

Ask it about deliveries it has not seen and it falls apart. That curve predicts 157 minutes for a 7.8 kilometre delivery, sitting between two real orders that took 42 and 45 minutes. At 6.6 kilometres it predicts 14.9 minutes, quicker than the 1.2 kilometre order it was fitted on. The curve is not wrong about the training data. It is perfect about the training data, and it is worthless.

Capacity is the room a model has to express different functions. Too little and it cannot represent the pattern that is there, which is underfitting. Too much and it starts reproducing the accidents of the sample it was given, which is overfitting. The scikit-learn documentation has a small worked version of exactly this: thirty noisy points drawn from a cosine, fitted with polynomials of degree 1, 4 and 15, then scored by ten-fold cross-validation. Degree 1 scores a mean squared error of 0.41 because it cannot bend at all. Degree 4 scores 0.043. Degree 15 scores 0.22, worse than degree 4 despite being able to fit the training points far more closely.

The important part of that example is that the training error keeps falling as the degree rises. Only the held-out score turns around. If you never hold anything out, you never see the turn, and every added parameter looks like an improvement right up until the model reaches production.

Why a good fit is not understanding

A model that predicts well is not thereby a model that has understood anything, and there is direct evidence for the distinction rather than just a philosophical objection.

Zhang and colleagues took standard image classification networks and trained them on datasets where the labels had been replaced with random ones. There was no pattern to find. The networks fitted the training data anyway, driving training error to essentially nothing, and they did the same when the images themselves were replaced with unstructured noise. Their conclusion was that the usual explanations for why large networks generalise, appeals to the model family or to explicit regularisation, do not account for what these systems can do. A network that can memorise arbitrary labels is a network whose low training error tells you very little on its own.

The second piece of evidence concerns what models settle on when there is a real pattern available. Geirhos and colleagues describe shortcut learning: decision rules that perform well on the standard benchmark and then fail when the testing conditions get harder or more realistic. A shortcut is a legitimate solution to the problem as posed by the loss and the data. It only looks like cheating from the outside, where someone can see that the model has attached itself to something incidental rather than to the thing anyone cared about.

Both results point at the same structural fact. The loss is a proxy for what you want. The model optimises the proxy exactly, including the parts where the proxy and your intention come apart. A model that appears to understand a task is a model whose proxy happened to line up well with the task across the situations you have checked.

What generalisation costs

Generalisation is performance on data the fitting procedure never saw. It cannot be measured on the training set, by definition, so it has to be bought with data you deliberately refuse to train on.

That refusal has a price. Every row in the test set is a row not used for fitting, which matters most when data is scarce, and it is exactly when data is scarce that people are most tempted to skip the split. Cross-validation reduces the cost by rotating which part is held out, at the price of fitting the model several times over.

The subtler cost is that a held-out set is a budget, and looking at it spends the budget. Each time a threshold is nudged, a feature added, or an architecture swapped because the validation score improved, information from that set has entered the model through your hands. Do it fifty times and the score is no longer an estimate of performance on new data; it is an estimate of performance on that particular set, which you have now partly fitted to. This is why serious evaluation keeps a final set sealed until the decision is made, and why the number of times a test set has been consulted is worth writing down.

There is a further wrinkle worth knowing about, because it complicates the tidy story of a single sweet spot. Belkin, Hsu, Ma and Mandal documented what they called a double descent curve: as capacity grows past the point where a model can interpolate the training data exactly, test performance can start improving again rather than continuing to degrade. The classical U-shaped curve is one part of a longer curve. That does not rescue the degree-eight polynomial above, and it does not mean capacity is free. It means the relationship between size and generalisation is not the simple trade-off it is usually taught as, and that empirical measurement outranks the rule of thumb.

Questions worth asking in a model review

The framing here changes what is useful to ask when someone brings a model to a review.

• What exactly is the loss rewarding, and where does that come apart from the decision this model informs?
• What does the simplest baseline score? A single average, a rule someone wrote by hand, last year's numbers. If the model beats it by a slim margin, the model is not the interesting part of the work.
• How many times has the held-out set been looked at, and by whom?
• What would this model have to see to be wrong, and does anything in the evaluation contain such a case?
• Which incidental feature of the training data could a shortcut attach to, and how would anyone notice if it had?

None of those questions require knowing which algorithm was used. They follow from the mechanism: parameters adjusted to make a number small on a sample of the world.

The delivery estimates, in the end, were fine in the centre and poor in the suburbs because distance and travel time have a different relationship once you leave a dense road network, and the training data was mostly city centre. The model had not learned that the suburbs are different. It had learned one relationship, from the places that supplied most of its rows, and it applied that relationship everywhere it was asked. Which brings up the thing that decided the outcome long before any model was chosen: the data.

References

1. Deep Learning, chapter 5, Machine Learning Basics. Ian Goodfellow, Yoshua Bengio and Aaron Courville, MIT Press, 2016.
2. Underfitting vs. Overfitting. scikit-learn documentation, version 1.9.0, 2026.
3. Understanding deep learning requires rethinking generalization. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht and Oriol Vinyals, ICLR, 2017.
4. Shortcut Learning in Deep Neural Networks. Robert Geirhos and colleagues, Nature Machine Intelligence, 2020.
5. Reconciling modern machine learning practice and the bias-variance trade-off. Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal, arXiv, later published in PNAS, 2018.