Random forests and why averaging works

Refit a decision tree on a resampled copy of the same training data and the top question often changes. That is unsettling the first time you see it, because the tree presented itself as a set of rules and rules are supposed to be stable. It is also the property the whole method is built on.

A tree is a low bias, high variance estimator. It can express almost any shape given depth, and it commits hard to whichever shape the particular rows in front of it suggest. Feed it a slightly different sample and it commits somewhere else. If the commitments are wrong in different directions, their average is right in a way none of them was, and the arithmetic that describes how much right is short enough to work through.

What averaging actually buys

Suppose each tree's prediction for a given case carries an error with variance sigma squared. Average n of them and ask what the variance of the average is.

If the errors were independent, the variance of the mean of n such quantities is sigma squared divided by n. A hundred trees would cut the error variance to one percent of a single tree's. That is the version people remember, and it is not the version that happens, because the trees are grown on overlapping data and share most of their features. Their errors move together.

Write the pairwise correlation between two trees' errors as rho. The variance of the sum is the sum of the variances plus all the covariances: n terms of sigma squared, plus n times n minus 1 terms of rho times sigma squared. Divide by n squared, because averaging divides by n and variance scales by the square:

variance of the average = rho * sigma squared + (1 - rho) * sigma squared / n

Two terms, and only the second one shrinks. Put numbers in it, with sigma squared set to 1 so the answers read as fractions of a single tree's error variance.

Correlation 0, one hundred trees: 0 plus 1 divided by 100, which is 0.010.
Correlation 0.2, one hundred trees: 0.2 plus 0.8 divided by 100, which is 0.208.
Correlation 0.6, one hundred trees: 0.6 plus 0.4 divided by 100, which is 0.604.

Now hold the correlation at 0.2 and go from one hundred trees to one thousand. The second term falls from 0.008 to 0.0008, so the total moves from 0.208 to 0.2008. Ten times the compute, ten times the memory, ten times the prediction latency, and four tenths of one percent of the variance removed.

That single line of arithmetic explains most of what a forest does and most of what it does not. The number of trees buys almost nothing past the point where the second term is small. The correlation between the trees is the whole game.

How the correlation is driven down

Two devices, and the second one is what separates a forest from plain bagging.

The first is the bootstrap: each tree is grown on a sample of the same size drawn with replacement, so it sees a different set of rows. How different is checkable. The chance that a specific row is missed by a specific draw is one minus one over n, and it has to be missed n times running, so the chance it is left out of a tree altogether is that quantity raised to the power n. For twelve rows, eleven twelfths to the twelfth power is 0.352. For a thousand rows it is 0.368, and it settles there for anything larger.

So roughly a third of the training rows are absent from any given tree, and every row is absent from roughly a third of the forest. Those are the out-of-bag rows, and scoring each row using only the trees that never saw it produces an estimate of held-out error without holding anything out. It is convenient rather than authoritative. Raschka's review of evaluation practice is worth reading before treating it as a substitute for a proper split, particularly his point that the holdout method is not recommended on small datasets, which is exactly the situation where the out-of-bag figure is most tempting.

The second device is feature subsampling. At every split, the tree is only allowed to consider a random subset of the columns. The scikit-learn documentation gives the defaults: the square root of the number of features for classification, and all features for regression. On a table with 36 columns, a classification tree considers 6 of them at each node. A column that is strongly predictive is unavailable at roughly five splits in six, so the other columns get used, and the trees stop agreeing with each other.

That is a deliberate handicap. Each individual tree is worse than it would have been with full access. The documentation states the trade in one sentence: random forests achieve a reduced variance by combining diverse trees, sometimes at the cost of a slight increase in bias, and in practice the variance reduction is often significant enough that the overall model is better. The whole design accepts weaker members in exchange for a lower rho.

The same page notes why bagging is applied to fully grown trees rather than to stumps: bagging methods work best with strong and complex models, in contrast with boosting, which works best with weak ones. Do not prune the trees in a forest. The pruning is the averaging.

The evidence that this works ordinarily well

Fernandez-Delgado and colleagues ran 179 classifiers from 17 families over 121 datasets. The random forest versions came out as the family most likely to be best, and the strongest of them reached 94.1 percent of the maximum accuracy attained on each dataset, ahead of Gaussian kernel support vector machines at 92.3 percent.

Grinsztajn, Oyallon and Varoquaux found the same ranking holding against modern alternatives on tabular data specifically, across 45 datasets with a 20,000 compute hour hyperparameter search per learner. Their explanation matters more than the ranking: tree ensembles cope with uninformative columns, keep the identity of individual features rather than mixing them, and fit irregular target functions that smooth models flatten out.

A forest is therefore a reasonable thing to reach for early, and an unreasonable thing to treat as a finish line, because everything in the next section survives it.

What a forest cannot fix

Return to the variance formula. Nothing in it touches bias. Averaging estimators that are all wrong in the same direction produces an estimate that is wrong in that direction with more confidence. The full accounting is in Bias and variance in plain terms, and the practical consequences are these.

A missing input stays missing. If the feature that drives the outcome was never collected, five hundred trees agree with each other about a world they cannot see.

Leakage is amplified, not diluted. A column that quietly encodes the answer is available to every tree, so every tree uses it, the correlation between them rises, and the out-of-bag score rises with it. The forest reports excellent numbers right up to deployment, for the reasons set out in The data comes first.

Predictions cannot leave the training range. Every leaf holds an average of training targets, and an average of a set of numbers lies between the smallest and the largest of them. A forest trained on shops billing between 4 and 14 lakh will never predict 20, however far the inputs are pushed. For a trend that continues past the observed range, a fitted line does something a forest structurally cannot, which is one of the arguments in Linear regression is still the baseline.

The audit trail is gone. A single tree explains a decision as a path. Five hundred trees explain it as a vote, and the sentence a reviewer could read out loud in Decision trees and how they split does not survive.

Feature importances are not a ranking of causes. The impurity-based importances a forest reports inherit the bias described in that same article: a column with many distinct values gets more chances to look useful, because splitting on it produces small, pure groups. Correlated columns also split their credit between themselves, so a genuinely important feature can appear halfway down the list because a near copy of it took the other half. Read importances as a description of what the model used, never as a claim about the world.

The cost is real and it is per prediction. Serving a forest means walking every tree. Memory grows with the number of trees times their depth, and inference latency grows with the number of trees.

Settings that matter, in order

Number of trees. Enough that the second variance term is small, and no more. Plot the out-of-bag or validation error against the count and stop where the curve flattens. It flattens sooner than most defaults assume.
Features per split. The one setting that moves rho, and therefore the one worth searching. Try values above and below the default rather than accepting it.
Minimum leaf size. Still worth setting, not to prune but to stop leaves formed from one or two rows dominating a vote.
Class weights or resampling when the classes are imbalanced, decided from the cost of each mistake rather than to make a metric look tidy.

The lending team's forest scored better than the single tree and worse than they expected, by about the margin the variance arithmetic predicts once you look at how correlated the trees were. What it did not do was fix the cases the model got badly wrong, because those were wrong in the same direction in every tree.

Fixing errors that persist across an ensemble needs a different idea: instead of growing many models independently and averaging them, grow each one specifically to correct what the previous ones got wrong.

References

Ensembles, Gradient boosting, random forests, bagging, voting, stacking. scikit-learn documentation, version 1.9.0, 2026.
Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?. Manuel Fernandez-Delgado, Eva Cernadas, Senen Barro and Dinani Amorim, Journal of Machine Learning Research, volume 15, pages 3133 to 3181, 2014.
Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning. Sebastian Raschka, arXiv, 2018.
Why do tree-based models still outperform deep learning on tabular data?. Leo Grinsztajn, Edouard Oyallon and Gael Varoquaux, arXiv, 2022.