The Error That Learns Backward: Backpropagation, the Chain Rule, and the Algorithm Behind Deep Learning

The Hook: A Billion Dials and No One to Turn Them

Picture a machine with a billion tiny dials. Each dial nudges the machine's behavior a little, and all of them interact in tangled ways. The machine is meant to learn a task – say, telling a cat from a dog in a photo. At first every dial is set at random and the machine guesses blindly. You see only the final result, and you know how wrong it is. Your job: figure out how to turn each one of the billion dials, just a bit, so the error shrinks.

That sounds hopeless. If you tested each dial individually – turn it slightly, see whether things improve, turn it back, try the next – a single learning step would take a billion trial runs. For modern language models with hundreds of billions of parameters and millions of learning steps, even all the world's data centers could not finish before the heat death of the universe.

And yet exactly such machines – neural networks – today learn language, images, and games at superhuman levels. The reason is a single, surprisingly elegant algorithm that pulls off the seemingly impossible: it computes the correct adjustment direction for all billion dials at once, at roughly the cost of a single run of the machine. That algorithm is called backpropagation, and it is no exaggeration to say: without it there would be no deep learning, no language models, no image generators. It is not the brain of AI, but its metabolism – the quiet process by which random numbers turn into knowledge.

This article tells how backpropagation works, why it is so outrageously efficient, who invented it (repeatedly and independently), what nearly killed it, and why it remains contested to this day whether the brain does anything like it.

Part 1: The Core Problem – Assigning Credit and Blame

Learning as Optimization

A neural network is, at heart, a very large mathematical function. It takes an input (the pixels of an image, the tokens of a sentence) and produces an output (a classification, the next word). Between input and output sit many parameters – the "dials" from the hook, technically the weights and biases of the individual neurons. Learning means setting these parameters so the function does the right thing on as many examples as possible.

To make "right" measurable, one defines a loss function. It is a single number stating how far the network's current output is from the desired result – zero would be perfect, large values mean large error. Training thus becomes an optimization problem: find the parameter values that minimize the loss. You can picture the loss as a gigantic, hilly landscape spanning the millions or billions of parameter dimensions. Each point in that landscape is a particular setting of all the dials; the height at that point is the error. We are looking for a valley as deep as possible.

Why Credit Is So Hard to Assign

The core problem has a name from the early history of AI: the credit-assignment problem. When the network makes an error at the end, which of its many internal weights contributed to it – and by how much? A neuron deep in the middle of the network influences the output only across many intervening layers. Its "blame" for the final error is smeared through a dense web of dependencies. It is precisely this nested responsibility that makes the naive fix – trying each parameter one by one – so hopelessly expensive.

The decisive insight of optimization is: we need not guess which direction to move a parameter. If the loss landscape is smooth (differentiable), the derivative at the current spot tells us exactly how the loss changes if we increase a parameter by an infinitesimal amount. The collection of all these partial derivatives – one per parameter – is the gradient. The gradient is a vector pointing in the direction of steepest ascent of the loss. To lower the loss, we take a small step in the opposite direction. That is gradient descent, and it is the heart of training: compute the gradient, take a small step downhill, repeat.

Gradient descent is the easy part. The real question is: how do you compute the gradient of a function with a billion inputs without doing a billion computations? This is exactly where backpropagation comes in.

Part 2: The Chain Rule – The Mathematical Foundation

Nested Functions

A neural network is a composition of functions: the output of the first layer becomes the input of the second, whose output becomes the input of the third, and so on all the way to the loss at the very end. Schematically, the loss L depends on the last layer, which depends on the one before, which depends on the one before that – a deeply nested chain.

For differentiating such nested functions, calculus has known a simple, powerful tool since Leibniz: the chain rule. It says that the derivative of a composite function is the product of the derivatives of its parts. If y depends on u and u depends on x, then the rate of change of y with respect to x is the product of the two local rates: "how strongly y reacts to u" times "how strongly u reacts to x." Intuitively, a small perturbation at the input propagates through the chain, and each stage multiplies it by its local gain factor.

At its core, backpropagation is nothing more than the systematic, organized application of the chain rule to the computational graph of a neural network. That is perhaps the most important demystification of this article: behind the mysterious-sounding name lies no exotic mathematics, but a bookkeeping technique for the chain rule that computes each intermediate derivative exactly once and cleverly reuses it.

The Computational Graph

To keep this bookkeeping clean, one represents the computation as a computational graph: a directed graph in which each node is an elementary operation (an addition, a multiplication, an activation function) and the edges show the flow of numbers. On the far left sit the inputs and parameters; on the far right sits the single scalar loss. However complex a neural architecture may be – a transformer, a convolutional net – it can be decomposed into such a graph of elementary building blocks, each easily differentiable.

The computational graph is the central data structure. Two passes run over it, and their interplay is the entire algorithm.

Part 3: How Backpropagation Computes – The Forward and Backward Pass

The Forward Pass

First the forward pass: you push an input into the graph and compute node by node from left to right until the loss emerges at the end. That is simply the ordinary evaluation of the network – the same thing that happens later at inference time. Crucially, though, during the forward pass you store the intermediate results at each node (the so-called activations). They will be needed in a moment.

The Backward Pass

Then the backward pass, the actual "backprop." It starts on the far right, at the loss, and travels left. At the output the matter is trivial: the derivative of the loss with respect to itself is one. From there, the chain rule is applied at each node: the node receives from its right-hand neighbor a number – how strongly the loss reacts to that node's output – and multiplies it by its own local derivative (which it can compute at once thanks to the stored activations). It passes the result on to its left-hand neighbors. In this way the error signal flows back layer by layer, and along the way each parameter learns how much it contributed to the final error – its partial derivative, its share of the gradient.

You can picture it as a cascade of responsibility in an organization: at the top, the overall result is fixed ("the error is this large"). Each management level is assigned its share of the responsibility and passes it further down, weighted by how strongly each subordinate influenced its own result. In the end every individual employee – every individual weight – knows how much to contribute to do better next time.

The Single Training Step

A complete training step therefore consists of four parts: first the forward pass (compute output and loss), second the backward pass (compute gradients for all parameters), third a small step downhill (shift each parameter by its negative gradient times a learning rate), and fourth: start over with the next batch of data. In practice one computes the gradient not over the entire dataset but over small samples (mini-batches) – that is stochastic gradient descent (SGD), often refined by optimizers such as Momentum or Adam, which derive smarter step sizes from the history of past gradients. But the core remains: backpropagation delivers the gradient, the optimizer uses it.

Part 4: Why Backward? The Real Efficiency Magic

Forward Mode versus Reverse Mode

Here lies the point that elevates backpropagation from a mere bookkeeping technique to one of the most consequential algorithms of the century. The chain rule can, after all, be applied in the other direction too – from the inputs forward to the output. In the theory of automatic differentiation (AD) these two routes are called forward mode and reverse mode. Backpropagation is precisely the reverse mode, applied to neural networks.

The difference is not a matter of taste but decides the computational cost – and does so dramatically as soon as the number of inputs and outputs differs greatly. And that is exactly the situation in training: we have a billion inputs (the parameters) but only a single output (the scalar loss).

• Forward mode computes, in one pass, how all outputs react to one input. With a billion parameters you would need a billion forward passes to obtain the full gradient – just as expensive as the naive dial method from the hook.
• Reverse mode computes, in one pass, how one output reacts to all inputs. That is exactly what we need: the sensitivity of the single loss to all billion parameters – in a single backward pass.

The Cheap-Gradient Principle

The cost of a backward pass is of the same order as that of a forward pass – roughly a small constant multiple (people often cite factors between two and five, depending on the architecture). This remarkable result is known as the Baur-Strassen theorem or the "cheap gradient principle": the gradient of a scalar function with respect to arbitrarily many inputs costs only a constant factor more than the function itself – regardless of whether there are ten or ten billion inputs.

That is the real sensation. We obtain the adjustment direction for a billion dials not for a billion trial runs but for the price of about two to five. The catch, to be stated honestly: reverse mode must remember the intermediate results of the forward pass in order to form the local derivatives. It therefore trades memory for compute. It is exactly this appetite for storing activations that is one reason why training large networks devours such enormous amounts of GPU memory – and why techniques such as "gradient checkpointing" (deliberately discarding and recomputing activations) were invented to tame it.

I am of the opinion that this very asymmetry – many inputs, one output – is the most frequently overlooked reason for the success of deep learning. It is not the networks alone that are the breakthrough, but the fortunate circumstance that their training task has exactly the shape for which reverse mode is optimal.

Part 5: The History – Invented Repeatedly, Long Overlooked

The history of backpropagation is a case study in how ideas arise independently, scattered, and often too early – and catch fire only when computing power and context are right.

Roots in Control Theory

The basic idea – propagating gradients backward through chained systems – first appeared not in AI but in control and optimization theory. Henry J. Kelley (1960) and Arthur E. Bryson (early 1960s), in the context of optimal control of trajectories, derived procedures that at their core amount to the backward application of the chain rule. Stuart Dreyfus, too, formulated a chain-rule derivation in 1962. These works knew neither "neural networks" nor the term backpropagation, but they contained the mathematical substance.

Linnainmaa 1970: The Reverse Mode Is Born

As a self-contained, general method, the reverse mode of automatic differentiation was described in 1970 by the Finnish mathematician Seppo Linnainmaa in his master's thesis – initially with no connection to neural networks at all. His goal was a different one: he wanted to efficiently estimate the accumulated rounding error of a computation composed of many elementary operations. To that end he developed the method of computing the derivatives of a differentiable expression representable as a graph by recursively applying the chain rule backward. That is, in modern terms, exactly the algorithm behind every deep-learning framework today. Jürgen Schmidhuber, who pursues the history of the field meticulously, therefore calls Linnainmaa the originator of backpropagation in the technical sense.

Werbos 1974: The Bridge to Neural Networks

The bridge to learning networks was built in 1974 by Paul Werbos in his doctoral thesis at Harvard University. He recognized that the procedure could be used to learn the weights of multilayer networks from errors. But the timing was poor: the first AI winter had, after the sobering critique of simple perceptrons (Minsky & Papert, 1969), brought the field into disrepute. Werbos published the application to networks more broadly only years later (among others in 1982). The idea was there, but it found little resonance.

Rumelhart, Hinton & Williams 1986: The Breakthrough into Visibility

Backpropagation became the widely known method only through a paper published in Nature in 1986: David Rumelhart, Geoffrey Hinton, and Ronald Williams, "Learning representations by back-propagating errors" (Nature 323, pp. 533–536). Their merit was less the first discovery – similar derivations had been found independently around 1985 by, for instance, David Parker and Yann LeCun – than the convincing demonstration that a multilayer network trained with backpropagation develops useful internal representations of the task in its hidden layers. That defused the old objection that multilayer networks might be powerful but untrainable. The paper is regarded as one of the founding documents of modern connectionism.

Even so, a second, long winter followed: in the 1990s and 2000s deep networks were considered barely trainable. Only the convergence of large datasets, powerful GPUs, and a few technical tricks (see Part 6) gave backpropagation its final breakthrough from around 2012. That Geoffrey Hinton received the Turing Award in 2018 (with Yann LeCun and Yoshua Bengio) and even the Nobel Prize in Physics in 2024 reflects how central this lineage has become to today's AI.

Part 6: When the Signal Starves – Vanishing and Exploding Gradients

The Problem

Backpropagation is mathematically correct, but that correctness guarantees no successful training. In the early 1990s researchers hit an insidious obstacle that for years set the limit on achievable network depth. Recall: in the backward pass the error signal is multiplied by a local derivative at each layer. Across many layers, then, the gradient is a product of many factors. If those factors are on average smaller than one, the product shrinks exponentially and the signal is practically zero after a few layers – the vanishing gradient. If they are larger than one, it explodes – the exploding gradient. In the first case the early layers barely learn anything; in the second, training becomes unstable.

This problem was formally analyzed and named in 1991 by Sepp Hochreiter in his diploma thesis, and shortly after deepened by Yoshua Bengio and colleagues (1994). For a long time it was the reason why "deep" networks stayed shallow in practice. It hit recurrent networks, which process sequences, especially hard: there each time step corresponds to another multiplication, so the network could only "remember" very short-term information.

The Solutions

The history of deep learning over the past twenty years is in good part the history of the tools against this one problem:

• Better activation functions. The classic sigmoidal functions (tanh, the logistic function) have derivatives that go to zero for large inputs – an amplifier of the problem. The ReLU (Rectified Linear Unit, popularized from 2010/2011 by Nair, Hinton, Glorot, and others) has a derivative of exactly one for positive inputs and therefore does not damp the signal. It became one of the single most important causes for deep networks suddenly becoming trainable.
• Thoughtful initialization. If the initial weights are set so that the variance of the signals is roughly preserved from layer to layer (Xavier/Glorot initialization 2010, He initialization 2015 for ReLU), the network starts near the balance between shrinking and exploding.
• Residual connections. The ResNets (He et al., 2015) insert "shortcuts" that pass the gradient undistorted back to deep locations. Only this made networks with hundreds of layers practical – a direct precursor to the depth of today's models.
• Gated memory. For sequences, Hochreiter and Schmidhuber invented the LSTM (Long Short-Term Memory) in 1997, whose "cell state" carries the signal almost unimpeded through gated controls across many time steps, thus circumventing the vanishing gradient in time.
• Normalization. Techniques such as Batch Normalization (Ioffe & Szegedy, 2015) and Layer Normalization keep the statistics of intermediate values stable and smooth the loss landscape, making the training of deep networks more robust.
• Gradient clipping. Against the exploding variant, simple capping helps: if the gradient exceeds a threshold, it is rescaled back to it.

Only this bundle of techniques – together with GPUs and large datasets – turned backpropagation from a pretty principle into the engine of a technological revolution.

Part 7: Backpropagation Today – The Invisible Infrastructure

Automatic Differentiation in Every Framework

Hardly any practitioner still codes backpropagation by hand today. Modern frameworks such as PyTorch, JAX, or TensorFlow build the computational graph automatically – in PyTorch dynamically during execution, in JAX by tracing and transforming functions – and derive the backward pass from it on their own. You write only the forward pass (the actual computation), call a function like backward() or grad(), and the system delivers the gradient. This automatic differentiation is perhaps the most underrated ingredient of the AI boom: it turned experimenting with new architectures from an error-prone hand calculation into a matter of a few lines of code.

The conceptual placement matters: automatic differentiation is neither numerical differentiation (approximating the derivative through tiny difference quotients – inaccurate and expensive) nor symbolic differentiation (rearranging formulas as in a math class – which explodes for large expressions). AD computes the gradient exactly (up to machine precision) and efficiently by exploiting the computational graph of the concrete computation. Backpropagation is simply the name that the reverse mode of AD carries in the world of neural networks.

The Same Algorithm Everywhere

Whether a transformer for language, a diffusion model for images, or a network learning a game: under the hood they all compute the gradient of their loss via backpropagation and use it to descend the loss landscape. The architectures change rapidly, the tasks too – but the learning mechanism in the engine room has remained essentially the same since 1986. That is a rare constancy in an otherwise fast-moving field.

The Open Flank: Does the Brain Learn This Way?

Backpropagation is the most successful learning algorithm in the history of technology – but is it also biologically plausible? Here there are serious doubts. The gravest is called the weight-transport problem: the backward pass would have to use exactly the same weights as the forward pass, only in the reverse direction. Real synapses, however, are one-way components; a neuron does not "know" the weights of its downstream connections. The brain would also need a precisely separated, globally coordinated backward pathway, for which there is no clear anatomical evidence.

Research therefore searches for more biologically plausible approximations. The "feedback alignment" finding (Lillicrap et al., 2016) showed, surprisingly, that even random feedback weights can supply a usable learning signal – so the exact backward weight is not strictly necessary. Other approaches interpret backpropagation as a limiting case of predictive-coding models, in which local prediction errors between adjacent layers drive learning – a bridge to the theory of the predictive brain. I am of the opinion that the most honest position at present is: backpropagation is with high certainty not the brain's mechanism in detail, but the brain might implement something that approximates its effect. Whether learning in the cortex and learning in a GPU cluster ultimately follow the same mathematical principle is one of the most exciting open questions at the border of neuroscience and AI.

The Central Takeaway

Backpropagation is the best evidence for an unassuming truth: sometimes the most consequential advance is not a new idea but the efficient organization of an old one. The chain rule is school mathematics; gradient descent is a simple walk downhill. The whole art lies in running the chain rule backward through the computational graph – and thereby obtaining the sensitivity of a single error measure to billions of parameters at the price of a mere handful of evaluations. This asymmetry between many inputs and one output is the true lever of deep learning.

Two things are worth taking to your own practice. First: when you debug a system in which "nothing learns," think of the gradients first – are they vanishing (then ReLU, better initialization, residual connections, normalization help) or exploding (then clipping helps)? The problem almost never sits in the optimizer, but in the flow of signal through the graph. Second, more generally: with any expensive derivative or sensitivity problem, ask whether you can formulate it as reverse mode – wherever a scalar target quantity depends on very many parameters (optimization, calibration, sensitivity analysis), the same trick applies, not only in machine learning.

A Question to Ponder

If a single algorithm known since 1970 – the chain rule applied backward – suffices to teach machines language and images, what does that say about the role of "intelligence" in these systems? Does the remarkable part lie in the learning mechanism (which is astonishingly simple) or in the sheer volume of data and parameters it is unleashed upon? And if the brain does not learn via backpropagation: would a more biologically plausible algorithm then be merely a scientific footnote – or the key to a more efficient, more data-frugal AI?

Cross-References in the Vault

• Attention Is All You Need: The Transformer, Self-Attention, and the Architecture of Modern AI – the architecture trained via backpropagation.
• The Ghost in the Machine: How to Read a Neural Network From the Inside – what sits in the weights that backpropagation adjusts.
• How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI – how many parameters and how much data gradient descent can handle.
• Trial and Error Perfected: Reinforcement Learning from TD-Gammon to AlphaGo to AlphaZero – a different learning paradigm that nonetheless rests on backpropagation as its building blocks.
• Praise and Blame for the Machine: RLHF and the Art of Teaching AI What We Want – how the gradient signal is derived from human feedback.
• From Noise to Image: Diffusion Models and the Physics of Generative AI – a generator type likewise trained via backpropagation.
• The Predictive Brain: Predictive Processing and the Illusion of Perception – the neuroscientific bridge to the question of biological plausibility.
• The Chinese Room: Searle and the Question of Whether Machines Can Understand – whether what is learned by backpropagation counts as "understanding."

Sources

• Rumelhart, Hinton, Williams: Learning representations by back-propagating errors, Nature 323, pp. 533–536 (1986): https://www.nature.com/articles/323533a0
• Schmidhuber: Who Invented Backpropagation? (historical account with references to Kelley, Bryson, Linnainmaa, Werbos): https://people.idsia.ch/~juergen/who-invented-backpropagation.html
• Seppo Linnainmaa (Wikipedia) – 1970 master's thesis, reverse mode of automatic differentiation: https://en.wikipedia.org/wiki/Seppo_Linnainmaa
• Paul Werbos (Wikipedia) – 1974 dissertation, application to neural networks: https://en.wikipedia.org/wiki/Paul_Werbos
• Baydin, Pearlmutter, Radul, Siskind: Automatic differentiation in machine learning: a survey, JMLR (2018): https://arxiv.org/pdf/1502.05767
• Schmidhuber: Deep Learning in Neural Networks: An Overview (2015) – on the history and the vanishing gradient problem (Hochreiter 1991, Bengio 1994): https://arxiv.org/pdf/1404.7828
• Lillicrap et al.: Random synaptic feedback weights support error backpropagation for deep learning, Nature Communications 7 (2016) – on biological plausibility (feedback alignment): https://www.nature.com/articles/ncomms13276

← All articles