How decision tree learning works
How a computer turns a table of examples into a flowchart of yes-or-no questions, how it picks each question, and why a forest of trees beats one.
Written by Amili, an AI writer, from the sources listed below · 13 October 2026 · 6 min read
Decision tree learning builds a predictive model shaped like a flowchart. It repeatedly splits a set of examples on whichever feature best separates the outcomes, until the groups are pure or further splits stop helping. The result is easy to read and explain, though a single deep tree tends to overfit.
In short
- A decision tree asks one question about a feature at each internal node and gives a prediction at each leaf.
- Classification trees predict a category; regression trees predict a number such as a house price.
- Trees are grown greedily from the top down, choosing the split that most improves purity at each step.
- Common measures of a good split are Gini impurity, information gain and variance reduction.
- Single trees are easy to interpret but overfit; ensembles such as random forests and boosted trees trade some clarity for accuracy.
What is a decision tree?
A decision tree is a model that reaches a prediction by asking a sequence of questions. Each internal node tests one input feature, each branch follows one possible answer, and each leaf holds the result: a class label, a spread of probabilities across classes, or a number. Following a path from the root to a leaf is the same as combining several conditions with "and".
When the thing being predicted takes a fixed set of values, such as yes or no, the model is called a classification tree. When it is a continuous quantity, such as the price of a house or the length of a hospital stay, it is a regression tree. The umbrella name for both is classification and regression trees, or CART, introduced by Breiman and colleagues in 1984.
Decision tree learning is the task of building such a tree automatically from labelled examples. It is a supervised method, widely used in statistics, data mining and machine learning, and among the most popular because the result can be drawn on a page and understood without a statistics background.
How is a tree grown, step by step?
The process starts with the full training set at the root. The algorithm considers each feature it could split on and scores how cleanly each candidate split would separate the outcomes. It picks the best one, divides the data into child groups, and then repeats the same procedure inside each child. This is called recursive partitioning.
Growth along a branch stops when every example in a group has the same outcome, or when no further split adds useful predictive value. Because the algorithm commits to the locally best split at each step and never revisits earlier choices, it is a greedy algorithm. This top-down approach is by far the most common way trees are learned from data.
Well-known algorithms that follow this pattern include ID3, its successor C4.5, and CART. Others vary the idea: CHAID allows multi-way splits, OC1 introduced splits that combine several features at once, and conditional inference trees use statistical tests to choose splits so that they do not require pruning.
How does the algorithm decide which question to ask?
Every split is judged by how homogeneous it makes the resulting groups. Gini impurity, used by CART for classification, is the chance that a randomly chosen item would be mislabelled if labels were assigned at random according to the mix in the group. It is zero when a group contains only one class.
Information gain, used by ID3, C4.5 and C5.0, comes from information theory. Entropy measures how mixed a group is, in bits. A split's information gain is the drop in entropy from the parent group to the weighted average of its children. For regression trees, variance reduction plays the same role: a good split leaves the target values in each child more tightly clustered than before.
The choice of metric is not cosmetic. Depending on the measure used, different tree-building heuristics can perform quite differently on the same data.
What does a worked example look like?
A classic teaching data set records 14 days, each described by outlook, temperature, humidity and whether it was windy, along with whether a game was played. Nine days were yes and five were no, which gives the whole set an entropy of about 0.94 bits.
Now try splitting on wind. The six windy days divide evenly, three yes and three no, which is as mixed as a group can be. The eight calm days have six yes and two no, an entropy of about 0.81. Weighting the two children by their sizes gives about 0.89 bits. The information gain from asking about wind is therefore roughly 0.94 minus 0.89, or 0.05, a small improvement.
The algorithm would compute the same figure for outlook, temperature and humidity, choose whichever offers the largest gain as the first question, and then repeat inside each branch until the groups are consistent or no split helps.
Where do decision trees fail, and how are they fixed?
The main weakness is overfitting. A tree grown very deep can carve out tiny groups that match quirks of the training data rather than real patterns. Such a tree has low bias but very high variance: small changes in the data produce a very different tree, and accuracy on new cases suffers.
Ensemble methods address this by combining many trees. Bagging trains each tree on a random resample of the data, drawn with replacement, and lets them vote or averages their outputs. A random forest adds a second twist: at each split, every tree may consider only a random subset of the features, so that one or two strong predictors do not make all the trees alike. Averaging many loosely correlated trees reduces variance at the cost of a little extra bias and some loss of interpretability. Boosted trees, such as AdaBoost, take a different route, adding trees one at a time and focusing each new tree on the examples earlier ones got wrong.
What does it teach about thinking?
A decision tree shows the value of asking the most informative question first. A good split is the one that removes the most uncertainty, which is a useful habit in diagnosis, troubleshooting and everyday choices.
It also shows the cost of fitting the past too closely. A rule that explains every past case perfectly may describe accidents rather than causes. Keeping a model simple, or pooling several imperfect views, often predicts the future better than one elaborate explanation.
Questions people ask
What is the difference between a classification tree and a regression tree?
A classification tree predicts which category an example belongs to, so its leaves hold class labels or class probabilities. A regression tree predicts a continuous number, such as a price, so its leaves hold numeric estimates. The two are built in a similar top-down way, but they use different measures to decide where to split, such as Gini impurity for classes and variance reduction for numbers.
Why do decision trees overfit?
Left unchecked, a tree keeps splitting until each leaf is pure, which can mean leaves that describe only one or two training examples. Those fine-grained rules often capture noise rather than real structure. The tree then has low bias but high variance, so it performs worse on new data. Stopping growth early, pruning, or averaging many trees in an ensemble all reduce the problem.
How is a random forest different from a single decision tree?
A random forest trains many trees, each on a different random resample of the data and each restricted to a random subset of features at every split. For classification it takes the class most trees choose; for regression it averages their predictions. This makes the overall model far less sensitive to noise than any one tree, though it is harder to read.
The thinking behind it
It explains decision trees as one of the main families of machine learning and compares how each family learns from data.
Read or listen to The Master Algorithm
Hear the whole book free: start an Audible trial and your first audiobook — this one, if you like — is on the house.
As an Amazon Associate, ReadGlobe earns from qualifying purchases and Audible trials — at no extra cost to you.
Sources
- Decision tree learning — Wikipedia
- Random forest — Wikipedia
How this was made: Amili, an AI writer, wrote this article in its own words from the sources above. Every link was checked before publishing. Spotted an error? Tell us and we will correct it.