Activation Functions
Activation functions are mathematical gates in neural-network layers that introduce non-linearity, enabling models to learn complex patterns and represent high-dimensional data. Without them, multi-layer networks would collapse into a single linear transformation, regardless of depth.
Core Purpose & Mechanics
- Non-Linearity: Essential for approximating arbitrary functions (Universal Approximation Theorem). Linear activations only allow hyperplane decision boundaries.
- Gradient Flow: Determines how errors propagate during backpropagation. Vanishing or exploding gradients often stem from poor activation choices.
- Output Transformation: Maps pre-activation values (logits) to specific ranges (e.g., [0,1] for probabilities, [-inf, inf] for dense outputs).
Common Functions
- ReLU (Rectified Linear Unit): . Standard for hidden layers; sparse activation prevents vanishing gradients but suffers from “dying ReLU” issues.
- Sigmoid: . Outputs [0,1]; used in binary classification outputs. Prone to saturation and vanishing gradients.
- Tanh: Hyperbolic tangent; outputs [-1,1]. Zero-centered output often leads to faster convergence than Sigmoid.
- Softmax: Multi-class generalization of Sigmoid; converts logits into a probability distribution summing to 1.
Interpretability & Internal Mechanisms
Recent research into Mechanistic Interpretability suggests that activation patterns may encode logical operations or token relationships in specific ways. See Anthropic’s NLA Research: Decoding Claude AI’s Internal Workings for details on how modern LLMs internally represent concepts through activation space geometry.