Train, test, and the lie of a single score

Two models went into a review. One scored 88.3 percent on the held-out set, the other 87.7 percent. The meeting spent forty minutes on why the first architecture suited the problem better, agreed to take it forward, and closed.

The held-out set had two hundred rows. Six tenths of a percentage point across two hundred rows is 1.2 rows. The meeting had spent forty minutes explaining a single disagreement about a single ticket, and the explanation was fluent, plausible, and about nothing.

This is the most common way an evaluation misleads a competent team. Not fraud, not leakage, not a broken pipeline. A number reported without the width of the number.

What a score actually is

A test score is a measurement taken on a sample. Draw a different sample and you get a different number. The question is how different, and that is answerable with arithmetic you can do in your head.

Accuracy on a test set of n independent rows behaves like a proportion. Its standard error is the square root of p * (1 - p) / n. For 88 percent accuracy on two hundred rows: 0.88 times 0.12 is 0.1056, divided by 200 is 0.000528, and the square root of that is 0.023. So the standard error is 2.3 percentage points, and a rough 95 percent interval runs from about 83.5 percent to about 92.5 percent.

Nine points wide. Both models in that review sit comfortably inside each other's interval, and so would a third model scoring 85 or 91.

The same arithmetic tells you what it would take to see a real difference. At two thousand test rows the standard error falls to 0.7 points and the interval narrows to roughly plus or minus 1.4 points. Precision in this measurement is bought with test rows, and it is bought slowly, because the error shrinks with the square root of the count. Ten times the data halves the interval twice over, not ten times.

Varoquaux made the same point against practice rather than against theory, looking at predictive modelling in brain imaging. His summary is that error bars on cross-validation are routinely underestimated, that sample sizes typical of the field inherently produce large ones, around plus or minus 10 percent at one hundred samples, and that the standard error computed across folds strongly underestimates the true figure. That last clause matters. Reporting the spread across folds is better than reporting nothing, and it is still optimistic.

Why there has to be a split at all

The scikit-learn documentation states the base case in one sentence: "Learning the parameters of a prediction function and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data."

What a model actually learns works through why a model with enough capacity can do exactly that. Holding data back is the only way to find out whether the fitted parameters describe the pattern or the sample.

A single split has one weakness. The estimate depends on which rows happened to land in the test set, and with a small dataset that dependence is severe. Cross-validation trades computation for stability. The training data is divided into k parts; the model is fitted k times, each time on k - 1 parts and scored on the part left out. The reported figure is the average across the folds.

The documentation's own worked example is instructive for what it shows accidentally. A linear support vector classifier scored on five folds of the iris data returns 0.96, 1.00, 0.96, 0.96, 1.00, which the example summarises as 0.98 accuracy with a standard deviation of 0.02. One model, one dataset, one procedure, and the individual fold scores span four percentage points. If you had run a single 80/20 split and happened to get the second fold, you would have reported a perfect classifier.

The split has to mirror the gap you actually face

Random splitting assumes the rows are independent and interchangeable. Real datasets frequently violate that, and the violations have names.

Grouped data. The scikit-learn guide is explicit: the independence assumption "is broken if the underlying generative process yields groups of dependent samples", and GroupKFold exists so "that the same group is not represented in both testing and training sets". Their example is a study with several samples per subject, where a flexible model can learn person-specific features and then fail on new people. Substitute customer, device, hospital or document and it is the same problem. If forty rows belong to one customer and a random split scatters them, the test set is measuring how well the model remembers that customer.

Time. Observations near each other in time are correlated, so the documentation notes it is "very important to evaluate our model for time series data on the future observations least like those that are used to train the model". TimeSeriesSplit does this by always training on a prefix and testing on what comes next. A random split on a time series trains on next month and tests on last month, which is not a task anyone will ever ask the model to perform.

The general rule behind both cases is the one argued in The data comes first: split along whatever axis production differs from training. By time, by site, by customer, by device. The number will be lower than a random split gives, and it will be the number you actually get.

The held-out set is a budget

Here is the failure that survives everything above. You split correctly, you cross-validate, and you still end up with an estimate that is too high, because of what happens between the split and the decision.

The documentation names it: when hyperparameters are tuned against a test set, "there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can leak into the model and evaluation metrics no longer report on generalization performance."

Cawley and Talbot took that observation and measured it. Their argument is that the variance of a model selection criterion matters as much as its bias, because a criterion with non-negligible variance can be over-fitted during selection, and their finding is that "the degradation in performance due to over-fitting the model selection criterion can be surprisingly large", often on the same scale as the differences between the algorithms being compared. They also describe the second-order problem: once a model has been chosen by optimising a cross-validation score, that same score is a biased estimate of the chosen model's performance, so the reported figure inherits a selection bias.

The consequence is not that tuning is wrong. It is that the score you tuned against is no longer an evaluation. If a hundred configurations were tried and the best was kept, the best score is partly a measure of how many configurations were tried. The repair is structural: tune inside a loop that is itself wrapped in an outer evaluation loop, so no configuration is ever chosen using the data that produces the final number, and keep one set sealed until the decision has been made.

Write down how many times the sealed set has been opened. That count belongs in the model review alongside the score.

What to report instead of one number

A single figure is not wrong, it is incomplete. A report that supports a decision carries five things, and none of them is expensive.

The score with its spread. Every fold's number, or an interval, not just the mean. If the folds disagree by four points, the reader needs to know before comparing two means that differ by one.
The size of the evaluation set. Everything above follows from it, and it is usually omitted.
The baseline on the same split. The majority class, last week's value, the rule the operations team already uses. A model that beats the baseline by less than the error bar has not been shown to beat it.
The score for the segments that matter. An aggregate hides a subgroup that is much worse, and the thin segments are where the first production failure will come from.
The number of configurations tried and the number of times the final set was consulted. This is the honesty column, and it is the one that will be missing.

There is one more question the numbers cannot answer, and it decides whether any of this matters. A score compresses every kind of mistake into one quantity, as though a missed fraud and a wrongly blocked customer were the same event. They are not, and treating them as equal is a choice being made silently every time accuracy is quoted.

Before that, though, comes the diagnostic question every disappointing score raises: whether the model is too rigid to see the pattern, or too free and following the sample. The vocabulary for that is bias and variance, and it is more useful in a review than it looks in a textbook.

The two models in that meeting were rerun properly. Grouped by customer, five folds, both scored with intervals. The first model came out at 84.1 percent with folds spanning six points, the second at 84.4 with a similar spread. They were the same model, in every sense that mattered to the business, and the forty minutes spent choosing between them had been the least useful part of the project so far.

References

Cross-validation, evaluating estimator performance. scikit-learn documentation, version 1.9.0, 2026.
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Gavin C. Cawley and Nicola L. C. Talbot, Journal of Machine Learning Research, volume 11, 2010.
Cross-validation failure, small sample sizes lead to large error bars. Gael Varoquaux, arXiv, 2017.