What overfitting is, and how models avoid it
Why a model that matches its training data perfectly often predicts badly, and the techniques used to keep models honest.
Written by Amili, an AI writer, from the sources listed below · 18 October 2026 · 6 min read
Overfitting happens when a model fits one particular dataset so closely that it captures random noise as if it were real structure, and then predicts new data poorly. It is the central risk in statistics and machine learning, countered by simpler models, held-out testing and regularization.
In short
- An overfitted model has more flexibility than the data can justify, so it learns quirks instead of patterns.
- The warning sign is a gap: excellent results on training data, worse results on data the model has not seen.
- Underfitting is the opposite failure, a model too simple to capture the real structure.
- Cross-validation, regularization, early stopping, pruning and dropout are common defenses.
- Overfitting also has practical costs, from fragile models to leaking details of the training data.
What does it mean for a model to overfit?
Any dataset mixes two things: the underlying pattern and random variation, often called noise. A good model captures the pattern and ignores the noise. An overfitted model fails to tell the two apart and treats some of the noise as if it were part of the structure. It ends up describing the particular sample it was built on, rather than the process that produced it.
In statistical terms, an overfitted model contains more parameters than the data can support. The model is selected, or trained, by how well it fits the data in hand, but it is ultimately judged by how well it handles data it has never seen. Whenever those two yardsticks differ, overfitting becomes possible. In the words often used to describe it, the model memorizes rather than learns.
How does overfitting happen, step by step?
It usually starts with flexibility. A model with many adjustable parts can bend to follow almost any set of points. As it is trained, its error on the training data keeps falling. At first that reflects genuine learning, because the model is picking up the real trend.
Past a certain point, though, the remaining gains come from chasing the noise: the specific, accidental features of this sample. Training error continues to drop while error on fresh data starts to rise. Training for too long, or learning from too few examples, makes this more likely. In the extreme, when a model has as many parameters as there are observations, it can reproduce the training data perfectly simply by memorizing it, and it will usually fail badly on anything new.
What does a simple example look like?
Imagine five measurements that rise roughly in a straight line but wobble a little above and below it. A straight line, with just two parameters, misses each point by a small amount but captures the trend. A fourth-degree polynomial, with five parameters, can pass exactly through all five points, with zero error.
On paper the polynomial wins. In practice it has to twist sharply between points to hit every wobble, and those twists are artifacts of the noise. Add a sixth measurement from the same process and the straight line will likely land close to it, while the polynomial may swing far away. The model that looked worse on the training data is the better guide to the next observation.
A real-world version of the same trap appears with a database of retail purchases that records the time of each sale. A model could fit the past perfectly by keying everything to those timestamps, yet it would be useless for prediction, because those exact moments will never occur again.
Where does overfitting show up?
It is common in regression. With as many variables as data points, a fitted line can pass through every point exactly. When many candidate variables have no real relationship to the outcome, some will still appear statistically significant by chance; keeping them in the model is a form of overfitting known as Freedman's paradox. Rules of thumb such as the one-in-ten rule, about ten observations per explanatory variable, exist to limit this.
In machine learning the term has been used since at least the 1980s for models that break Occam's razor by using more adjustable parameters, or a more elaborate method, than the problem really needs. The risk grows when there is little theory to guide the choice of model, because then there are many candidate models to pick from and one of them is bound to fit the sample well by luck.
How do people prevent it, and what does it cost if they don't?
Two broad strategies dominate. One is to penalize complexity directly. Regularization adds a cost for large or numerous parameters, so the model only becomes complicated when the data strongly justify it; ridge and lasso regression are well-known examples. Pruning trims unnecessary parts of a model, and dropout randomly removes inputs during training so that no single feature can be leaned on too heavily.
The other is to test on data the model has not seen. Cross-validation repeatedly holds out part of the data for evaluation, and early stopping halts training once performance on held-out data stops improving. Both assume the held-out data resemble what the model will meet later.
Ignoring the problem has costs beyond bad predictions. Overfitted models often demand more input data than necessary, are harder to reproduce or move to new settings, and can sometimes leak details of individual training examples, which is a concern when training data include personal information or copyrighted material. Going too far the other way creates underfitting: a model too simple to capture real structure, such as a straight line forced onto curved data. The goal is a balance between the two errors.
What does overfitting teach about thinking?
Overfitting is the mathematical form of a familiar mistake: building an elaborate explanation for something that was mostly chance. A theory that accounts for every past detail can feel convincing precisely because it was shaped to fit those details.
The useful habit is to judge an explanation by how well it predicts what has not happened yet, and to prefer the simpler account unless the evidence clearly earns the extra complexity. Fitting the past is easy; foresight is the real test.
Questions people ask
What is the difference between overfitting and underfitting?
Overfitting means a model is too flexible: it fits the training data very closely, including its noise, and then predicts new data poorly. Underfitting means the model is too simple to capture the real pattern, so it performs poorly even on the training data. In bias-variance terms, overfitting pairs low bias with high variance, and underfitting pairs high bias with low variance.
How can you tell if a model is overfitting?
The clearest sign is a gap between performance on the training data and performance on data the model did not see during training. If accuracy keeps improving on training examples while it stalls or worsens on a held-out validation set, the model is likely learning noise. Cross-validation is a standard way to measure this gap reliably.
What is regularization?
Regularization is a family of techniques that discourage a model from becoming more complex than the data justify, typically by adding a penalty for large or numerous parameters to the quantity being minimized. Ridge regression, which uses an L2 penalty, and lasso, which uses an L1 penalty, are common examples. Early stopping and dropout are often grouped with them.
The thinking behind it
It has a chapter on overfitting that carries the idea from statistics into everyday decisions about when to think less.
Read or listen to Algorithms to Live By
Hear the whole book free: start an Audible trial and your first audiobook — this one, if you like — is on the house.
As an Amazon Associate, ReadGlobe earns from qualifying purchases and Audible trials — at no extra cost to you.
Sources
- Overfitting — Wikipedia
- Regularization (mathematics) — Wikipedia
How this was made: Amili, an AI writer, wrote this article in its own words from the sources above. Every link was checked before publishing. Spotted an error? Tell us and we will correct it.