How backpropagation works

What problem does backpropagation solve?

A neural network is a stack of layers of simple units. Each unit takes a weighted sum of the signals arriving from the layer before, passes it through an activation function, and sends the result onward. The weights are the knobs. Training a network means turning those knobs until its outputs match the correct answers in a set of examples.

To turn the knobs sensibly, the network needs a measure of how wrong it is, called the loss or cost function. Squared error is a common choice for predicting numbers, and cross-entropy for sorting things into classes. The question backpropagation answers is narrow but crucial: for one example, in which direction should each weight move to make the loss smaller, and how strongly does the loss respond to it? That collection of sensitivities is the gradient.

How does it work, step by step?

First comes the forward pass. An example enters at the input layer and flows through each hidden layer to the output. Along the way, the network stores each layer's weighted input, its activation, and the slope of its activation function at that point, because the return journey will need them.

Next the loss is computed by comparing the output with the target. Then the backward pass begins. At the output layer, it is easy to say how the loss changes when each output unit changes. That quantity is often described as the error at that layer, or the share of the cost attributable to each node.

The key observation is that a weight in an early layer can only affect the loss through the layers that follow it. So once the error at one layer is known, the error at the layer before it can be found by multiplying by the connecting weights and by the slopes of the activation functions. The gradient for any single weight is then that downstream error multiplied by the activation feeding into the weight. Repeating this layer by layer, from back to front, yields every gradient in the network.

Finally, an optimiser uses the gradient. In plain gradient descent each weight is nudged a small step against its gradient, with the step size set by a learning rate. Repeating forward pass, backward pass and update over many examples gradually lowers the loss.

Why go backwards instead of forwards?

Each piece of the gradient could in principle be worked out on its own with the chain rule. Doing that separately for every weight would recompute the same downstream derivatives over and over. Going backwards lets each layer's result be reused by the layer before it, so nothing is calculated twice.

There is a second saving. Working backwards means each step multiplies a single vector of errors by a matrix. Working forwards from an early layer would mean multiplying whole matrices together, effectively tracing every possible path by which a small change could ripple through the network. That is far more expensive and computes intermediate quantities nobody needs. Backpropagation keeps only what leads to the final loss.

What does a tiny worked example look like?

Picture the smallest possible network: two inputs feeding straight into one output unit, with no hidden layer and a simple linear output. Suppose a training example gives both inputs the value 1 and says the correct output is 0. The weights start at random values, so the network will probably produce something other than 0.

If the loss is squared error, plotting the error against the output gives a parabola whose lowest point sits exactly at the target. Because the output is a weighted sum of the inputs, the error is really a function of the two weights. The gradient points uphill on that surface, so stepping the weights the opposite way moves the output towards 0.

This toy case also shows a subtlety. Any pair of weights that adds up to zero gives a perfect answer here, so the single example does not pin down a unique solution. More training data, or extra constraints on the weights, is needed to choose among them. In a real network with hidden layers the same reasoning applies, except that the chain rule has to carry the error back through every intermediate layer.

Where is it used, and where does it struggle?

Backpropagation is the standard way to compute gradients when training neural networks, from small feedforward models to the convolutional, recurrent and transformer architectures behind computer vision, speech and large language models. It also serves as a step inside more elaborate optimisers such as Adaptive Moment Estimation. Training at scale remains computationally heavy and relies on graphics processors and large datasets.

It has limits. It only works when the loss and the activation functions can be differentiated efficiently. The popular rectifier, or ReLU, is not differentiable at zero, though in practice that single point has not stopped it from being widely used. Backpropagation also says nothing about how far to step; the learning rate and optimiser make that choice, and a poorly chosen step can make training slow or unstable. Methods that use second-order derivatives, such as the Levenberg–Marquardt algorithm, can converge faster on awkward error surfaces, at a higher cost per step.

What does it teach about thinking?

Backpropagation is a disciplined way of assigning blame in a long chain of cause and effect. Instead of asking every early decision how it affected the final outcome directly, it asks each stage only about its immediate successor, then passes the answer back. Large problems become tractable when credit and blame are traced one link at a time.

It is also a reminder that a fix depends on the measure of success. The gradient only points towards a lower loss, so whatever the loss function rewards is what the network will learn, intended or not.

Questions people ask

The thinking behind it

• systems thinking
• goodharts law
• local vs global optimum

More algorithms, explained

• How decision tree learning works
• How hash functions work
• How k-means clustering works
• How tf–idf works: weighing words by how much they tell you