Loss Functions
Loss functions are mathematical functions that quantify the difference between a model’s predicted outputs and actual target values. During training, a loss function evaluates model performance on a batch of data and produces a single numerical value representing prediction error. This measurement serves as a critical feedback signal that guides optimization algorithms in adjusting the model’s parameters toward improved performance. By repeatedly computing loss across training batches and backpropagating the error signal, machine learning systems iteratively refine their internal representations.
Common Loss Functions
Different tasks require different loss functions. Regression problems typically use mean squared error (MSE) or mean absolute error (MAE), which penalize deviations between continuous predictions and targets. Classification tasks often employ cross-entropy loss, which measures the divergence between predicted probability distributions and true class labels. More specialized domains have domain-specific variants: reinforcement learning agents may use policy gradient losses, while contrastive learning approaches use similarity-based losses that encourage similar samples to have nearby representations.
Role in Model Training
The choice of loss function fundamentally shapes what a model learns to optimize for. A well-chosen loss function aligns with the actual objective—for instance, using classification accuracy as a loss function is problematic because accuracy is non-differentiable, whereas cross-entropy loss provides smooth gradients suitable for gradient descent. The loss function also encodes priorities: weighted losses can emphasize certain data points or classes over others, allowing practitioners to reflect domain knowledge or business requirements into the training process. The cumulative loss across an entire dataset or validation set provides a scalar metric for comparing different model architectures or hyperparameter choices.