Decision trees and how they split

A tree is the only common model that a non-technical reviewer can read out loud, which is why credit and claims teams keep asking for one. It is also the model most likely to arrive at a review with a hundred and forty leaves, several of which were fitted on three rows, and nobody able to say why the first question at the top is the first question.

The first question is not a judgement call. It is the result of a small calculation, repeated over every possible question, and the calculation is worth doing by hand once. After that, every argument about tree depth becomes an argument about arithmetic you have already seen.

Twelve applications

A portfolio of twelve loan applications. Two things are recorded about each: whether the applicant missed an instalment in the previous year, and whether declared income is above 50,000 a month. The outcome is whether the loan ended in default.

• Missed, income below, defaulted
• Missed, income below, defaulted
• Missed, income above, defaulted
• Missed, income above, defaulted
• Missed, income below, repaid
• Clean, income above, defaulted
• Clean, income above, repaid
• Clean, income above, repaid
• Clean, income above, repaid
• Clean, income below, repaid
• Clean, income below, repaid
• Clean, income below, repaid

Five defaults out of twelve. The tree has to choose which of the two questions to ask first, and it chooses by measuring how much of the uncertainty each question removes.

Entropy, computed

Entropy measures how mixed a group is, in bits. A group that is entirely one outcome has an entropy of zero: there is nothing left to learn. A group split evenly between two outcomes has an entropy of one bit, the maximum for two classes.

The formula is the sum over classes of the class share times the log to base two of one over that share. For the twelve rows, the shares are 5 in 12 and 7 in 12.

Five twelfths is 0.4167, and the log to base two of one over 0.4167 is 1.263, so that term contributes 0.4167 times 1.263, which is 0.526. Seven twelfths is 0.5833, its log term is 0.778, and the contribution is 0.454. Add them: 0.980 bits. Close to fully mixed, which is what a five to seven split looks like.

Now score the first question, whether the applicant missed an instalment.

The missed branch takes five rows, four of which defaulted. The shares are 0.8 and 0.2. The terms are 0.8 times 0.322, which is 0.258, and 0.2 times 2.322, which is 0.464. Entropy 0.722.

The clean branch takes seven rows, one of which defaulted. The shares are 0.143 and 0.857. The terms are 0.143 times 2.807, which is 0.401, and 0.857 times 0.222, which is 0.191. Entropy 0.592.

Neither child's entropy is the answer on its own, because they hold different numbers of rows. Weight each by its share of the parent: five twelfths of 0.722 is 0.301, and seven twelfths of 0.592 is 0.345. Together, 0.646 bits remain.

The information gain is what the question removed: 0.980 minus 0.646, which is 0.334 bits.

Score the second question the same way. Income above 50,000 covers six rows, three of which defaulted, so that branch is exactly half and half, at an entropy of 1.000. Income below covers six rows with two defaults, shares of 0.333 and 0.667, giving 0.333 times 1.585 plus 0.667 times 0.585, which is 0.528 plus 0.390, or 0.918. Weighted equally, 0.959 bits remain, and the gain is 0.021 bits.

The tree asks about missed instalments first because that question is worth sixteen times as much as the other one on this data. No preference, no domain knowledge, no ordering of columns. One number against another.

Gini, and why the choice of criterion rarely decides anything

Entropy is not the only impurity measure. The scikit-learn documentation gives Gini alongside it: the sum over classes of the class share times one minus that share, which for the root is one minus 0.4167 squared minus 0.5833 squared, or 0.486.

Run the same two questions through Gini. The missed branch gives 0.32 and the clean branch 0.245, weighting to 0.276, a decrease of 0.210. The income question weights to 0.472, a decrease of 0.014. Same winner, same ratio of roughly fifteen to one.

That is the usual outcome. The two criteria are different curves over the same quantity and they disagree only in close cases, where the split you pick barely matters by construction. Time spent choosing between them is time not spent on the decisions below, which do matter.

Continuous inputs, and the split the algorithm actually searches

Real data does not arrive as yes and no. For a numeric input such as monthly turnover, the algorithm sorts the values, considers a threshold between each adjacent pair, and scores every one of them the way the two questions above were scored. Twelve rows with distinct turnovers give eleven candidate thresholds for that column alone.

Because each question is asked inside the group the previous question produced, a tree expresses conditions for free. A weak payment record can matter for young shops and not for old ones, and no term has to be constructed by hand to say so, which is precisely what the flat boundary in Logistic regression and decision boundaries could not do.

This is where the cost comes from and where the greed comes in. The scikit-learn documentation is direct about it: learning an optimal decision tree is NP-complete under several aspects of optimality, so practical algorithms are heuristic, making a locally optimal decision at each node. The tree takes the best single question available now and never revisits it. A pair of questions that would be excellent together, but mediocre apart, is invisible to it.

There is a bias hiding in the search that nobody mentions until it bites. Consider adding the application reference number as an input. Every value is unique, so splitting on it produces twelve groups of one row each, every one pure, every entropy zero, and an information gain of the full 0.980 bits. It beats every real question by a distance. The rule it learns, that application 4471 defaults, predicts nothing about application 4472. Any column with many distinct values gets an unearned advantage, and identifiers, timestamps, and free-text keys are the usual offenders. Drop them before fitting, not after wondering why the tree looks perfect.

Where the tree stops being readable

Continue the example. Inside the missed branch, five rows with four defaults, ask about income. Income above covers two rows and both defaulted, an entropy of zero. Income below covers three rows with two defaults, an entropy of 0.918. Weighted, 0.551 remains, so the gain is 0.722 minus 0.551, which is 0.171 bits.

The algorithm takes it, because 0.171 is greater than zero. What it has produced is a leaf that says: applicants who missed an instalment and earn above 50,000 always default. That rule rests on two rows. It is perfectly confident and almost certainly an accident of the sample, which is the overfitting mechanism described in What a model actually learns, arriving here in its most visible form.

Trees are unusually prone to this, and the reason is structural: every level halves the data available to the next decision, so confidence rises while evidence falls. The documentation puts the practical version well, noting that the number of samples required to populate the tree doubles for each additional level, and recommending max_depth to control the size of the tree, with min_samples_leaf set to something like 5 as a starting value so that multiple samples inform every decision.

Those two settings are the whole defence, and they are worth more attention than the impurity criterion. Set a minimum leaf size in rows, in units a business reviewer understands, and the tree stops writing rules from two examples. Choose the depth on validation data rather than by eye, using the discipline in Train, test, and the lie of a single score, because a deeper tree always looks better on the data it was grown on.

What a single tree is genuinely good at

Grinsztajn, Oyallon and Varoquaux benchmarked tree-based models against a range of deep learning methods across 45 tabular datasets, with a hyperparameter search of 20,000 compute hours for each learner. Tree-based models stayed ahead on medium-sized data, and their account of why is specific: neural networks are hurt by uninformative features, which trees ignore cheaply; trees are not rotation invariant, so they keep the meaning of individual columns rather than mixing them; and trees pick up irregular, non-smooth target functions that networks smooth over. Those three properties describe most operational tables in a business.

A single tree also handles mixed types without ceremony, needs no scaling because it only ever compares a value to a threshold, and splits on the raw column rather than a transform of it. Compare that with the preparation a linear model needs, described in How a dataset becomes features.

The property people actually buy it for is the audit trail. A prediction is a path, and the path is a sentence: missed an instalment, income above 50,000, therefore refuse. Rudin's argument for high stakes decisions is that this kind of inherent interpretability should be preferred over explaining an opaque model after the fact, which she holds is likely to perpetuate bad practices. A tree with a depth limit is one of the few models that meets that standard without an interpretation layer bolted on.

What to check before shipping one

• Print the tree. If a leaf holds fewer rows than you would accept as evidence in a meeting, the limit is wrong, not the reviewer.
• Remove identifiers, near-identifiers, and anything derived from the outcome before fitting, or the gain calculation will find them.
• Choose depth and minimum leaf size on validation data and record the values with the model.
• Refit on a resampled version of the training data and compare the trees. If the top split changes, the structure is not stable, and the confident sentence at the root is less solid than it reads. That instability is not a defect to be tuned away; it is the property the next article turns into an advantage.

References

1. Decision Trees. scikit-learn documentation, version 1.9.0, 2026.
2. Why do tree-based models still outperform deep learning on tabular data?. Leo Grinsztajn, Edouard Oyallon and Gael Varoquaux, arXiv, 2022.
3. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Cynthia Rudin, arXiv, later published in Nature Machine Intelligence, 2019.