Transformer Architectures
Core family of deep learning models based on the attention-mechanism, utilizing self-attention to process sequential data in parallel.
Key Components
- Self-Attention: Mechanism allowing the model to weigh the importance of different parts of the input sequence relative to each other.
- Multi-Head Attention: Parallel attention layers capturing diverse relationships within the data.
- Positional Encoding: Adds sequence order information since transformers lack inherent recurrence or convolution.
- Feed-Forward Networks: Non-linear transformations applied to each position independently.
- Encoder-Decoder Structure: Standard architecture for sequence-to-sequence tasks, though Decoder-Only variants dominate modern LLMs.
Evolution & Notable Implementations
GPT-3
- Scaling Laws: Demonstrated that performance improves predictably with increased model size, dataset size, and compute budget GPT-3: Unprecedented Scaling and Emergent Language Model Capabilities.
- Emergent Capabilities: Abilities not explicitly present in smaller predecessors (e.g., GPT-2) appeared at scale, such as few-shot learning and basic reasoning.
- Architecture: Pure Decoder-Only transformer, optimized for autoregressive language modeling.
- Context Window: Utilized a 4096-token context window.
- Training: Trained on a massive, diverse dataset including books, web pages, and code.
Related Concepts
- Attention Is All You Need
- large-language-models
- Few-Shot Learning
- Neural Network Scaling Laws