How linear regression works
How fitting a straight line through scattered data lets you predict outcomes and measure relationships, and why the line can mislead when used carelessly.
Written by Amili, an AI writer, from the sources listed below · 18 October 2026 · 5 min read
Linear regression is a statistical method that models an outcome as a weighted sum of one or more explanatory variables, choosing the weights that best fit observed data, usually by least squares. It is used to predict new values and to measure how strongly variables are related.
In short
- Simple linear regression uses one explanatory variable; multiple linear regression uses two or more.
- Least squares picks the line that makes the sum of squared prediction errors as small as possible.
- A coefficient describes the expected change in the outcome for a one-unit change in one input, holding the others fixed.
- Squared errors give outliers outsized influence, so robust alternatives exist for messy data.
- A fitted relationship is not automatically a causal one, especially with observational data and correlated inputs.
What does linear regression actually model?
Linear regression describes how one quantity, the response or dependent variable, tends to change as other quantities, the explanatory variables, change. The model assumes the expected value of the response can be written as a constant plus a weighted sum of the inputs. Those weights, called coefficients or parameters, are unknown at the start and are estimated from data.
With a single input the result is a straight line, and the method is called simple linear regression. With several inputs it is multiple linear regression, and the 'line' becomes a flat surface in more dimensions. It was the first kind of regression to be studied rigorously and put to wide practical use, largely because models that are linear in their parameters are easier to fit and their statistical behavior is easier to work out.
How does least squares find the best line?
For any candidate line, each data point has a residual: the gap between the value actually observed and the value the line predicts. Least squares squares each residual, adds them up, and chooses the coefficients that make that total as small as possible. For the ordinary linear case there is a closed-form solution, so the best coefficients can be calculated directly rather than searched for.
Squaring does two things. It treats overshooting and undershooting alike, and it punishes large misses much more than small ones. The mathematician Legendre published the first clear account of the method in 1805, presenting it as an algebraic way to fit linear equations to data. Other fitting methods exist, including least absolute deviations and penalized versions such as ridge and lasso regression, which add a cost for large coefficients.
What does a small worked example look like?
Imagine three observations pairing an input x with an outcome y: (1, 2), (2, 4) and (3, 5). The average x is 2 and the average y is about 3.67. Least squares gives a slope of 1.5 and an intercept of about 0.67, so the fitted line is y = 0.67 + 1.5x.
That line predicts about 2.17, 3.67 and 5.17 for the three inputs. The residuals are roughly -0.17, 0.33 and -0.17, and no other straight line produces a smaller sum of their squares. The slope has a plain reading: each extra unit of x is associated with about 1.5 more units of y. Given a new x of 4, the model would predict about 6.67, although that guess is only as good as the assumption that the pattern continues.
Where is linear regression used?
Its uses fall into two broad groups. The first is prediction: fit the model on data where both inputs and outcomes are known, then use it to estimate outcomes when only the inputs are available. In machine learning terms it is a supervised algorithm that learns from labelled examples.
The second is explanation: measuring how much of the variation in an outcome lines up with variation in each input, and spotting inputs that add nothing or that duplicate one another. It is one of the most important tools in the biological, behavioral and social sciences, and it is the standard way to draw a trend line through time series such as prices or economic output.
Where does linear regression mislead?
Outliers are the first hazard. Because least squares weighs large errors heavily, a handful of extreme points can drag the line toward themselves and away from the bulk of the data. When outliers are common, cost functions that are less sensitive to them are a better choice.
Interpretation is the second. A coefficient describes the expected change in the outcome when one input moves and the others are held fixed, but in observational data nothing was actually held fixed; the analyst simply compared subsets that happened to share values. When the inputs are correlated with each other and were not set by a study design, multiple regression can fail to clarify how each one relates to the outcome. An input can look important alone yet contribute almost nothing once others are included, or the reverse.
Finally, the standard methods rest on assumptions about the data. When those assumptions fail, the estimates can be biased, and the many extensions that relax them usually need more data or more modelling to reach the same precision.
What does linear regression teach about thinking?
A straight line is a deliberately simple story about messy data, and its value comes from that simplicity. It forces a clear claim, how much y moves when x moves, that can be checked against new observations.
The same simplicity is a warning. A line that fits well is a summary of a pattern, not proof of a mechanism. Asking which variables were left out, and how the data were collected, matters as much as the fit itself.
Questions people ask
What is the difference between simple and multiple linear regression?
Simple linear regression relates an outcome to a single explanatory variable, producing a straight line. Multiple linear regression uses two or more explanatory variables at once. Each coefficient in a multiple regression then describes the expected change in the outcome for a one-unit change in that input while the other inputs are held constant.
Why does least squares square the errors?
Squaring makes every error positive, so misses above and below the line cannot cancel out, and it penalizes large misses more than small ones. It also makes the mathematics tractable: for the ordinary linear case the best coefficients have a closed-form solution. The downside is sensitivity to outliers, which robust methods address.
Does a regression coefficient show cause and effect?
Not by itself. In a designed experiment, where the inputs are set deliberately, a coefficient can sometimes be read as the effect of an intervention. In observational data, where inputs are often correlated and nothing was truly held fixed, the coefficient describes an association within the data, which may reflect other factors the model does not include.
The thinking behind it
It explores forecasting from data and the gap between a fitted pattern and a reliable prediction.
Read or listen to The Signal and the Noise
Hear the whole book free: start an Audible trial and your first audiobook — this one, if you like — is on the house.
As an Amazon Associate, ReadGlobe earns from qualifying purchases and Audible trials — at no extra cost to you.
Sources
- Linear regression — Wikipedia
- Least squares — Wikipedia
How this was made: Amili, an AI writer, wrote this article in its own words from the sources above. Every link was checked before publishing. Spotted an error? Tell us and we will correct it.