Gradient boosting explained by its mistakes

A forest grows every tree on the same problem and averages the answers. Boosting grows each tree on a different problem: the problem of what is still wrong after all the previous trees have spoken. That one change in the setup produces a method with a different failure mode, a different set of knobs, and a habit of winning tabular competitions.

The mechanism is easiest to believe after doing it. Six houses, three rounds, arithmetic small enough to check on paper.

Six houses and a first guess

Six sale prices in lakh, against floor area in square metres.

• 60 square metres, 32 lakh
• 72, 38
• 85, 44
• 95, 51
• 110, 57
• 140, 68

Start with the least informative model available: predict the mean of 48.3 for every house. The residuals, meaning the actual price minus the prediction, are minus 16.3, minus 10.3, minus 4.3, plus 2.7, plus 8.7 and plus 19.7. Their average magnitude is 62.0 divided by 6, which is 10.3 lakh.

Now fit a tree to those residuals. Not to the prices. To the residuals. Keep it to a single split, which is the smallest tree that does anything, and let it choose the threshold that best separates the residuals into two groups.

It splits at 95 square metres. Below the threshold the four residuals average minus 7.05; above it the two residuals average plus 14.2.

Here is the part that decides everything downstream. The tree is not added at full strength. It is multiplied by a learning rate, and at a rate of 0.5 the corrections applied are minus 3.5 and plus 7.1. The model becomes 44.8 for small houses and 55.4 for large ones.

New residuals: minus 12.8, minus 6.8, minus 0.8, plus 6.2, plus 1.6, plus 12.6. Average magnitude 40.8 divided by 6, which is 6.8 lakh.

Two more rounds

Round two fits a stump to those six residuals and finds the best threshold at 72 square metres, separating the two most negative residuals from the rest. Their averages are minus 9.8 and plus 4.9, and half of each is added: minus 4.9 and plus 2.45.

Residuals after round two: minus 7.9, minus 1.9, minus 3.3, plus 3.8, minus 0.9, plus 10.2. Average magnitude 4.6 lakh.

Round three splits at 110 square metres, isolating the largest house, whose residual of 10.2 is now the loudest thing in the data. The two group averages are minus 2.03 and plus 10.15, and half of each is added.

Residuals after round three: minus 6.9, minus 0.9, minus 2.2, plus 4.8, plus 0.2, plus 5.1. Average magnitude 3.3 lakh.

The sequence of average misses runs 10.3, 6.8, 4.6, 3.3. Each round removed a chunk of what the previous rounds left, and each round used a model too weak to have done anything on its own: one question, two answers.

That is the whole idea, and Friedman's paper states it in the general form. Function estimation is treated as numerical optimisation in function space, and boosting is gradient descent carried out there: at each step, fit a base learner to the direction of steepest descent of the loss, then take a step along it. With squared error, that direction is exactly the residual, which is why the arithmetic above looks like plain subtraction. With a different loss the quantity being fitted changes, and the shape of the procedure does not.

What the learning rate is actually buying

Run round one again at a learning rate of 1.0, taking the stump at full strength. The predictions become 41.25 and 62.5, and the residuals become minus 9.25, minus 3.25, plus 2.75, plus 9.75, minus 5.5 and plus 5.5, for an average magnitude of 6.0 lakh.

That is better than the 6.8 the half-strength step reached. Full steps descend faster, and they descend faster on the training data specifically, which is the only place the residuals are defined.

The scikit-learn documentation gives the trade in its own terms. The learning rate scales the step length of the gradient descent procedure, it interacts strongly with the number of estimators, and smaller values require more weak learners to reach the same training error. Then the operational recommendation: empirical evidence suggests that small values favour better test error, so set the rate to a small constant such as 0.1 or below and choose a number of estimators large enough that early stopping applies.

Read that as a division of labour. The learning rate is a regularisation setting. The number of rounds is not a setting at all: it is a stopping decision, made by watching a validation curve and halting when it turns. Fixing the number of rounds in advance and tuning everything else around it is the most common way to arrive at an overfitted booster, and it looks like diligence while it is happening. The discipline that keeps this honest is in Train, test, and the lie of a single score: the validation set is what stops the training, so the test set has to be somewhere else entirely.

Overfitting, with the complication intact

The obvious story is that adding rounds forever must eventually overfit, and it usually does. The literature complicates it in a way worth knowing, because it changes what you look at.

Bartlett, Freund, Lee and Schapire set out to explain an observation that did not fit the simple story: the test error of a boosted classifier usually does not increase as the ensemble grows very large, and is often observed to decrease even after the training error reaches zero. Their explanation is the distribution of margins, the gap between the votes for the correct answer and the votes for the best wrong one. Boosting keeps widening those margins after it has stopped making mistakes, and wider margins go with lower test error.

So a training error of zero is not the signal to stop, and neither is round count. The signal is the validation curve, which is the only thing that measures the quantity you care about. A boosted model can keep improving long after it has memorised the training set, and it can also start degrading at round 300 having looked fine at round 200. Watch it rather than reasoning about it.

Where boosting fails, and why the mechanism says so

A mislabelled row gets more attention every round. This follows directly from the procedure. A row whose recorded price is wrong produces a large residual, the next tree is fitted to residuals, so the largest residuals dominate the split it chooses. Correcting it leaves a residual that is still large, so the round after that chases it too. Boosting hunts down the loudest disagreement, and it cannot tell a genuine hard case from a data entry error. Watch the third round above: it spent a split isolating a single house. If that house had a typo in its price, three of the model's rounds would be devoted to fitting the typo.

It is sequential. Every round depends on the residuals from the last, so rounds cannot be trained in parallel the way a forest's trees can. Chen and Guestrin's XGBoost paper is largely a response to this, and its contributions are engineering ones: a sparsity-aware algorithm, a weighted quantile sketch for approximate split finding, and attention to cache access patterns, data compression and sharding, which together let the system scale beyond billions of examples using far fewer resources than existing systems. The mathematics did not change. The constant factors did, and they were what made the method practical at size.

Its settings are coupled. The learning rate, the number of rounds, the tree depth, and the subsampling rate all move the same underlying quantity. Tune them one at a time and the result depends on the order. Search them jointly, or fix the rate low, cap the depth at something small like 3 to 6, and let early stopping choose the count.

It still cannot leave the training range. Every leaf holds an average of training targets, and adding averages together produces something bounded by what was seen. A booster trained on houses up to 68 lakh has no mechanism for saying 90, exactly as in Random forests and why averaging works.

It gives up the audit trail entirely. A forest at least votes on comparable objects. A booster's output is a sum of corrections, most of them small, none of them meaningful alone. Anything you say about why it made a prediction comes from an attribution method applied afterwards, and the caution in Reading a model you did not build applies with full force.

A working order for the settings

• Fix the learning rate low, 0.05 or 0.1, and stop treating it as the thing to search.
• Cap the tree depth between 3 and 6. Deeper trees inside a booster express interactions you probably did not intend and overfit faster.
• Choose the number of rounds by early stopping on a validation set, never by a round number carried over from a previous project.
• Subsample rows per round if the data is large. It lowers the correlation between rounds for the same reason it does in a forest, and it costs nothing.
• Before blaming the model for a stubborn error, look at the row. Boosting will point at your worst data, insistently, several rounds in a row, and that is a feature if you read it as a data report rather than a modelling result.

Boosting and forests both build their answers out of axis-aligned rectangles, because that is what a tree split is. A different family of methods asks a different question altogether: not where to cut, but how much clear space to leave between the classes.

References

1. Greedy function approximation, a gradient boosting machine. Jerome H. Friedman, The Annals of Statistics, volume 29, issue 5, pages 1189 to 1232, 2001.
2. Boosting the margin, a new explanation for the effectiveness of voting methods. Peter Bartlett, Yoav Freund, Wee Sun Lee and Robert E. Schapire, The Annals of Statistics, volume 26, issue 5, pages 1651 to 1686, 1998.
3. XGBoost, A Scalable Tree Boosting System. Tianqi Chen and Carlos Guestrin, arXiv, 2016.
4. Ensembles, Gradient boosting, random forests, bagging, voting, stacking. scikit-learn documentation, version 1.9.0, 2026.