Bias and variance in plain terms

A pricing model was rebuilt three times in six weeks. Version one was a linear fit and everyone agreed it was too crude. Version two was gradient boosting with default settings and scored beautifully in the notebook. Version three was version two with more trees, because if boosting helped then more boosting should help more.

Version three was worse in production than version one. By that point nobody could say why, because the team had been changing the model without a name for what was wrong with it.

There is a name. Two, in fact, and having them turns a guessing game into a short diagnostic that takes an afternoon.

The three parts of an error

Suppose you could rerun the whole project many times: collect a fresh training sample from the same source, fit the model again, and predict the same new case. You would get a different prediction each time, because the training sample differs each time.

Across those runs, the expected squared error on that case breaks into three pieces.

Bias is how far the average of all those predictions sits from the truth. It is the error the model makes because of what it can represent, not because of which sample it saw. A straight line fitted to a curved relationship has bias no matter how much data you give it.

Variance is how much the individual predictions scatter around their own average. It is the part of the error caused by the model being sensitive to which rows it happened to receive. A deep tree grown to purity has high variance: change a handful of training rows and its predictions move.

Noise is the part nobody can remove. Two identical orders take different times. Two identical applicants default differently. This is a property of the world and the measurement, not of the model, and it sets a floor beneath every score anyone will ever report on that problem.

The floor is the part teams forget. When someone asks why the model cannot reach 99 percent, the honest first answer is often that nothing could, because the outcome is not fully determined by anything you are allowed to observe.

Measured, not asserted

The decomposition is not only a diagram. The scikit-learn documentation carries a worked version that fits the same estimator fifty times on fifty different training samples drawn from one generator, then splits the mean squared error into its parts.

A single decision tree comes out at 0.0255 total error: 0.0003 of squared bias, 0.0152 of variance, and 0.0098 of noise. Almost everything the tree gets wrong beyond the floor is variance. It is not that the tree cannot represent the function. It represents a slightly different function every time it is trained.

Bagging the same trees gives 0.0196 total: 0.0004 of squared bias, 0.0092 of variance, and the same 0.0098 of noise. Variance fell by about 40 percent, bias rose by a hair, and the noise did not move, because nothing about averaging changes the world.

That is the whole logic of an ensemble in three numbers. It also shows the limit: total error fell from 0.0255 to 0.0196, and it cannot fall below 0.0098 no matter what is fitted, because that part was never the model's to fix.

Telling the two apart in your own project

You do not need to run fifty resamples to diagnose a real model. The signature is visible in two numbers you already have.

Training error high, validation error about the same. The model cannot fit the data it was shown. That is bias. More rows will not help, because the problem is not information, it is the shape of the function. Regularising harder will make it worse.

Training error low, validation error much higher. The model fits its own sample and does not transfer. That is variance. A different model family is not the first thing to reach for. More data, fewer parameters, stronger regularisation, or averaging over several fits all attack it directly.

Both errors near the floor and neither improving. You are close to the noise, and the remaining gains are in the features or the data rather than in the model. How a dataset becomes features is where that work lives.

The pricing team's three versions map onto this cleanly in hindsight. Version one was bias: a linear fit on a relationship with a genuine kink in it. Version two swapped bias for variance and looked better because the notebook was reporting a score on rows the model had effectively memorised parts of. Version three added variance to a model that already had too much.

What each fix costs

Neither knob is free, and the cost is usually the reason a team picks the wrong one.

Reducing bias means giving the model more room: more capacity, more interaction terms, a richer family. It costs the extra variance that comes with the room, and it costs interpretability, which is a real operational expense when somebody has to explain a decision to a customer.

Reducing variance means constraining the model or feeding it more evidence. More training data is the clean fix and is often the one that cannot be bought. Regularisation is cheap and buys stability at the price of a little bias. Averaging over many fits, as in bagging, is the version the scikit-learn numbers above quantify: it costs compute at training and at serving, and it costs the ability to read the model.

Reducing noise is not a modelling activity at all. It means better measurement, a cleaner label definition, or a feature that captures something currently unobserved. It is usually the highest-value work available and it never appears in a modelling ticket.

There is one asymmetry worth carrying into a review. Variance problems announce themselves in the gap between training and validation scores, which is why they get caught. Bias problems produce two disappointing numbers that agree with each other, which reads as "the problem is just hard" and ends the investigation. A model that is comfortably beaten by a hand-written rule on some segment is a bias problem hiding behind an average.

Where the tidy story stops being true

The U-shaped curve is a good working model and it is not the last word, which is worth saying plainly rather than discovering later.

Belkin, Hsu, Ma and Mandal set out what they call a double descent curve. Past the point where a model has enough capacity to interpolate the training data exactly, test error can begin falling again rather than continuing to rise, and they report evidence for this across a range of models and datasets. Their framing is that the classical U-shaped curve is one region of a longer curve rather than the whole picture.

Neal and colleagues looked at the same puzzle from the decomposition side. Their observation is that as neural networks get wider, both bias and variance can fall together, which is not what the classical trade-off predicts and which explains why the expected U often fails to appear in deep learning experiments.

Two things follow, and they point in opposite directions. The trade-off is not a law you can lean on to argue that a bigger model must overfit, so measure before asserting. And nothing in either result rescues the pricing team's third version, because a modestly sized boosted ensemble on a few thousand rows is nowhere near the interpolation regime these papers describe. The classical picture remains the right default for tabular work at ordinary scale.

The diagnostic, as a sequence

Run this before changing the model again.

• Score the trivial baseline on the same split. It puts a ceiling on how much of your error is worth attributing to the model at all.
• Record training error and validation error together, from a split that mirrors deployment, as argued in Train, test, and the lie of a single score.
• Read the gap. Small gap with two poor numbers is bias. Large gap is variance.
• Fit the same model on half the training data and on all of it. If validation error improves substantially between the two, more data will keep helping and you have a variance problem worth spending on. If the curve has flattened, more data is not the answer.
• Estimate the floor. Duplicate or near-duplicate inputs with different outcomes tell you roughly how much of the target is not determined by the features at all.

Only after those five does it make sense to argue about model families.

The pricing model ended up as a boosted ensemble with a fraction of the trees of version three, monotonic constraints on two features where the business genuinely knew the direction of the relationship, and a training set built by weekly split rather than at random. Its validation score was lower than version two's had been. It was also, for the first time, roughly the score it went on to get.

That still leaves a question the error decomposition cannot answer. A model can be well fitted, correctly diagnosed, and confidently wrong about a particular case, and knowing how much to trust an individual prediction is a different property altogether.

References

1. Single estimator versus bagging, bias-variance decomposition. scikit-learn documentation, version 1.9.0, 2026.
2. Reconciling modern machine learning practice and the bias-variance trade-off. Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal, arXiv, later published in PNAS, 2018.
3. A Modern Take on the Bias-Variance Tradeoff in Neural Networks. Brady Neal and colleagues, arXiv, 2018.