Backpropagation
Backpropagation is the fundamental algorithm for training artificial neural networks. It computes gradients of a loss function with respect to each weight and bias in the network by efficiently applying the chain rule of calculus. This enables neural networks to learn from data by adjusting parameters in the direction that reduces prediction error.
Forward and Backward Passes
Training with backpropagation consists of two distinct phases. During the forward pass, input data flows through the network layer by layer, with each neuron computing its output based on weighted inputs and an activation function. The network produces a final prediction, which is compared to the target output to calculate a loss value. During the backward pass, the error signal is propagated backward through the network. The algorithm computes how much each parameter contributed to the final loss by applying the chain rule recursively, determining the gradient of the loss with respect to every weight and bias.
Gradient Descent Integration
Once gradients are computed, an optimization algorithm—typically gradient descent or a variant like Adam—uses these gradients to update network parameters. Weights and biases are adjusted by a small amount proportional to their gradients, moving toward lower loss values. This process repeats over many iterations through the training dataset until the network converges to a solution that generalizes well to unseen data.
Backpropagation’s efficiency relative to computing gradients by finite differences makes it practical for networks with millions of parameters. Its discovery and refinement in the 1980s was instrumental in enabling the deep learning revolution that followed.