Linear regression is still the baseline

A lending team wanted to size working capital loans for small retailers, and the modelling proposal that arrived was gradient boosting with a hyperparameter search attached. The first question in the review was not about the search. It was what a straight line scored on the same data, and nobody in the room knew, because nobody had fitted one.

That is a common gap and an expensive one. Without the line, the boosted model has no scale. It might be worth three months of engineering or it might be worth four percent, and those two situations look identical on a slide that reports only the winner.

Fitting a line on six shops, by hand

Six retailers, monthly footfall in thousands of visitors against monthly revenue in lakh.

2 thousand visitors, 4.9 lakh
3 thousand, 4.4 lakh
5 thousand, 7.9 lakh
6 thousand, 6.9 lakh
8 thousand, 10.6 lakh
12 thousand, 13.3 lakh

Ordinary least squares picks the intercept and slope that make the sum of squared vertical misses as small as possible. For one input the answer has a closed form you can compute on paper.

The mean footfall is 6 and the mean revenue is 8.0. Write each observation as a deviation from its mean. Footfall deviations: minus 4, minus 3, minus 1, 0, 2, 6. Revenue deviations: minus 3.1, minus 3.6, minus 0.1, minus 1.1, 2.6, 5.3.

Multiply them pairwise and add: 12.4 plus 10.8 plus 0.1 plus 0 plus 5.2 plus 31.8 gives 60.3. Square the footfall deviations and add: 16 plus 9 plus 1 plus 0 plus 4 plus 36 gives 66.

The slope is 60.3 divided by 66, which is 0.914. The intercept is the mean revenue minus the slope times the mean footfall: 8.0 minus 0.914 times 6, which is 2.52.

revenue = 2.52 + 0.914 * footfall

Two numbers. That is the whole model, and the same sentence applies to it that applies to any other: it is parameters chosen to make an arithmetic quantity small on a sample, as set out in What a model actually learns.

Now score it. The shop with 6 thousand visitors is predicted at 8.0 and took 6.9, a miss of 1.1 lakh, the worst of the six. The 12 thousand shop is predicted at 13.48 against 13.3, a miss of 0.18. Squaring all six misses and adding gives 3.55, so the mean squared error is 0.591 and the typical miss, its square root, is 0.77 lakh.

Compare that to predicting the average of 8.0 for every shop. Its squared misses add to 58.64, a typical miss of 3.13 lakh. The line cuts the error by three quarters, and one divided by the other gives an R squared of 0.94.

Those two numbers, 0.77 and 3.13, are the ones to write on the board before anyone proposes an architecture. Every later model gets measured against 0.77, and the question stops being whether the new model is good and becomes how much of the remaining 0.77 it removes.

What the two coefficients license you to say

The slope carries units, and reading them out loud is the cheapest sanity check available. Revenue is in lakh, footfall in thousands, so 0.914 means roughly 91,000 rupees of monthly revenue per additional thousand visitors, or about 91 rupees per visitor. A retail operator can tell you in one second whether that is plausible for their category. A boosted model will not hand you a sentence anyone can argue with.

The intercept is where care is needed. It reads as 2.52 lakh of revenue at zero footfall, which sounds like fixed income and is not supported by anything in the data. The smallest shop in the sample had two thousand visitors. Nothing was observed anywhere near zero, so the intercept is where the line happens to cross an axis it never visited. Treat it as a fitting artefact unless the data actually reaches that far.

The second caution matters more in real datasets, where inputs arrive in bundles. The scikit-learn documentation states the mechanism plainly: the coefficient estimates rely on the independence of the features, and when features are correlated so that the design matrix is close to singular, the least squares estimate becomes highly sensitive to random errors in the observed target, producing a large variance. Add average basket size alongside footfall and the two coefficients start trading magnitude between themselves. The fitted line predicts as well as it ever did. The individual numbers stop meaning what you were about to say they mean, and the model does not warn you.

That failure has a signature. Refit on a different sample, or on the same sample with one shop removed, and watch the coefficients. Stable predictions with unstable coefficients is exactly the collinearity picture, and it is the reason a coefficient table should never be read as a table of causes.

Why the baseline is a measurement, not a modesty exercise

Two pieces of published evidence are worth having on hand the next time a simple model is skipped.

Fernandez-Delgado and colleagues evaluated 179 classifiers from 17 families over 121 datasets. Random forests came out best, and the strongest of them reached 94.1 percent of the maximum accuracy achieved on each dataset, with Gaussian kernel support vector machines second at 92.3 percent. Read the other way round, that is the ceiling: across a broad sweep of ordinary problems, the best of 179 methods left only a few percent of headroom over the next family down. Most of the difference between a good result and a bad one sits somewhere other than the choice of algorithm.

Ferrari Dacrema, Cremonesi and Jannach went at the same question from the reproduction side. They took 18 neural recommendation methods from top conferences. Seven could be reproduced with reasonable effort, and of those seven, six could often be outperformed by comparably simple heuristic methods such as nearest neighbour approaches. The remaining one did not consistently beat a well tuned linear ranking method. The published gains had been measured against baselines nobody had bothered to tune.

Breiman's essay on the two cultures of statistical modelling is worth reading alongside these, because it argues the opposite case competently. His position is that algorithmic methods are a more accurate and informative alternative to data modelling, on small datasets as well as large ones, and that a field which insists on tidy assumed models produces questionable conclusions. Both things are true at once. Flexible methods often win. The size of the win is the number that decides whether you should pay for it, and you only have that number if you fitted the line.

Where a line genuinely fails

Being the baseline does not make linear regression the answer. It fails in specific, recognisable ways, and each one is visible in a residual plot long before it is visible in a score.

Curvature. Plot the residuals against the fitted values. If they arc, the relationship is not linear in the inputs you gave it. Sometimes that is fixed inside the linear model itself, by adding a squared term or a log, which is a feature question rather than a model question and is covered in How a dataset becomes features.

Interactions. Linear regression adds contributions. If footfall is worth more in a shop that stays open late, and nothing in the feature set encodes that pairing, the line cannot express it. Trees find such combinations without being told, which is the subject of Decision trees and how they split.

Spreading error. If the residuals fan out as the predictions grow, the model is more wrong about large shops than small ones while reporting a single error figure that averages the two. Modelling the log of revenue often flattens the fan.

Single influential observations. Squared error punishes large misses quadratically, so one extreme point can pull the whole line towards itself. A shop with 40 thousand visitors in a sample of six would set the slope almost single handedly. Fit with and without each suspect row and see whether the answer moves.

Extrapolation. A line will answer confidently about a shop with 60 thousand visitors, because arithmetic does not know where the data stopped. Record the observed range of every input and refuse predictions outside it, or at least flag them.

None of these is a reason to skip the fit. They are the diagnostics the fit gives you for free, and they usually point at the data rather than at the model.

What to put in the review

The baseline error and the trivial error, in the units of the decision. For the shops that is 0.77 lakh against 3.13, not an R squared.
The slope of every input with its units spelled out, and a named person who has said whether the sign and size are plausible.
The observed range of every input, so extrapolation is a visible act rather than an accident.
Whether the coefficients move when the sample changes. If they do, report predictions and stop interpreting the individual numbers.
The margin the more complex model wins by, measured the same way on the same held-out data, following the discipline in Train, test, and the lie of a single score.
What that margin is worth in money, and what the complex model costs to serve, retrain, and explain when it goes wrong.

The lending team fitted the line. It reached a typical miss of about 1.1 lakh on their real data, the boosted model reached 0.95, and the gap was worth a little under two percent on the loan book. They shipped the line first, because it could be reviewed by the credit team in an afternoon, and kept the boosted model as the second release once the pipeline was stable enough to retrain it safely.

The line answers a question with a number for an answer. The next question in that project was different in kind: not how much a shop will make, but whether a given applicant will repay at all, which is a yes or no, and yes or no turns out to be geometry.

References

Linear Models, Ordinary Least Squares. scikit-learn documentation, version 1.9.0, 2026.
Statistical Modeling, The Two Cultures. Leo Breiman, Statistical Science, volume 16, issue 3, pages 199 to 231, 2001.
Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. Maurizio Ferrari Dacrema, Paolo Cremonesi and Dietmar Jannach, arXiv, 2019.
Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?. Manuel Fernandez-Delgado, Eva Cernadas, Senen Barro and Dinani Amorim, Journal of Machine Learning Research, volume 15, pages 3133 to 3181, 2014.